Update to Firefox 24 ESR

[SilvanScherrer]

latest for GA we have to update the source to latest 17 ESR which is 17.0.10

[SilvanScherrer]

as 17 ESR is already very old we decided to go direct to 24 ESR

[dmik]

I'm trying to fetch from hg::http://hg.mozilla.org/releases/mozilla-esr24 using git fetch and the script is being run for quite some time already. Several hours. Waiting.

[StevenLevine]

Did this work for ESR17?

[dmik]

Yes, sure. It just takes too long because it's a relatively slow script. See https://github.com/bitwiseworks/mozilla-os2/issues/1.

[StevenLevine]

OK. Perhaps I misunderstood. Are you giving a status report or saying that there is a problem with the fetch?

[dmik]

I had to abort and restart it though since I recalled that I fetched only one specific tag to (greatly) save time. This time it must be FIREFOX_24_3_0esr_RELEASE. So for this update the proper command is

git fetch upstream-24 FIREFOX_24_3_0esr_RELEASE

(where upstream-24 is defined by git add remote hg::http://hg.mozilla.org/releases/mozilla-esr24).

Note for myself and everybody else who needs to know how to update.

[dmik]

Sorry, it was just a status report.

[dmik]

Sh*t, we're in trouble. It crashed with this near the very end:

Traceback (most recent call last):
  File "/Users/dmik/bin/git-remote-hg", line 805, in <module>
    sys.exit(main(sys.argv))
  File "/Users/dmik/bin/git-remote-hg", line 793, in main
    do_import(parser)
  File "/Users/dmik/bin/git-remote-hg", line 517, in do_import
    export_tag(repo, tag)
  File "/Users/dmik/bin/git-remote-hg", line 390, in export_tag
    export_ref(repo, tag, 'tags', repo[tag])
  File "/Users/dmik/bin/git-remote-hg", line 317, in export_ref
    committer = "%s %s %s" % (user, time, gittz(int(tz)))
ValueError: invalid literal for int() with base 10: '<bsmith@mozilla.com>'
fatal: stream ends early
fast-import: dumping crash report to .git/fast_import_crash_11310
fatal: Error while running fast-import

[dryeo]

On 02/26/14 11:02 PM, Dmitriy Kuminov wrote:

Sh*t, we're in trouble. It crashed with this near the very end:

Why not just fetch from https://github.com/mozilla/gecko-dev/tree/esr24

[dmik]

Because the original merge is done from hg.mozilla.org (using the git-remote-hg script) as there was no up-to-date git repo back then. We can merge from git now but I would have to match the key git commits to hg commits to find the right diff (gecko-dev lacks firefox release tags at all, so it's a bit puzzling). I will first give it another try with the latest version of git-remote-hg (https://github.com/felipec/git/blob/fc/master/git-remote-hg.py).

[dmik]

Unfortunately, the new script fails just as the previous one. It seems not to expect this specific input. Sorry, but I'm not going to debug it. I will merge a single cumulative diff to our tree. Probably, on a dedicated branch, to have a clean state w/o our changes.

[dmik]

Okay, 24esr is not directly derived from on 17esr at all. Nice (but expected more or less). Their histories must split at some pre-17esr point. (And they could have merged many-many times again since then). Since our repo doesn't have full mozilla history (which was a not very good decision from this point of view), the best we can do is just to import the new 24esr source tree as a single commit on a separate branch, then port our 17esr work to it and then make it master (saving the current master on the 17esr branch). Ugh.

[StevenLevine]

Are you familiar with git's notion of vendor branches? This may be less work than what your are planning and you may even a able to save your 17esr commit history.

[dryeo]

They branch of at 10.0.2, 17.0.x (which ever the last release was, there's often a problem with releases forcing another quick point release) and 24 should have branched at 24.0.x. The 24ESR numbering has changed with the second number = a release so 24.3.0 is released along with 27 and if a 27.0.1 it would equal 24.3.1. Perhaps you should start a new history with 24 when it branched from trunk, then work on a branch and also pull the ESR fixes into the branch. Judging from 10ESR the changes were very small between the point releases

[dmik]

Steve, of course I know what vendor branches are. The problem is that we don't have a git repo that would contain the history we need. We only have gecko-dev ATM but it lacks Firefox releases, at least you won't font ESR releases in there. And, besides that, vendor branches are not what you typically do in git. You use remotes instead (which in this particular case we don't have, as said above). My attempt to make git-remote-hg work was to fulfill this gap and we have to admit it failed.

Dave, okay, I guessed something like that. I think we will go a bit different way. Since we are stuck with git now, we will have to create our own 'vendor branch' right in our own repository here. I will update 17esr to 24esr (instead of creating a new history starting form 24esr) and will then merge the diff to master. I.e. I will simulate the SVN approach (since it turned out that we lack the real remotes).

[StevenLevine]

Dmik, this is the type of vendor branch I was recommending. Remotes automate the process, but manually maintained vendor branches just take a bit more work.

For those that are unfamiliar with doing vendor branches this way, there's:

http://happygiraffe.net/blog/2008/02/07/vendor-branches-in-git/

[dmik]

Fun but this is what I was planning to do in my last commend and actually already did. Now we have a branch called vendor-esr. where all ESR code updates from upstream currently live and new ones will land to. The 24esr code is already on that branch. I will now merge the changes to our master.

I must say that there is quite a lot of changes. And they seem to move to python for parts of the build system (I see a lot of new moz.build files). I expect some bit of work in order to make this new tree build on OS/2.

BTW, someone has been updating the OS/2 code as well (e.g. in widgets/os2). Most changes look like automatic replacements but there are also some new methods etc.

PS. The instructions on the page you reference to are not very accurate: it's better to run git add -A after deleting all files of the previous release and unzipping the new release when you update the upstream branch. The -A option will nicely play with removal of missing (deleted) files. So the generic algorithm for further upstream updates is as follows:

1. Go to a dir for the mercurial repo's working copy.
2. hg clone http://hg.mozilla.org/releases/mozilla-esrXX/ (if not already done).
3. hg update FIREFOX_XX_Y_Zesr_RELEASE
4. Go to a dir of this repo's working copy.
5. git checkout vendor-esr
6. rm -r *
7. Copy everything from the mercurial working copy (excluding .hg) to this dir.
8. git add -A
9. git commit
10. git tag FIREFOX_XX_Y_Zesr_RELEASE
11. git push --all -u --tags

[StevenLevine]

Best I can tell from my usage is that git commit --all has the same effect. That said, I like the idea of flagging the deletes at git add time better.

[dmik]

Well, to be precise commit --all will process deletes but it won't add new files (you will have to git add them first). While add -A does it all: removal, addition and simple modifications of the existing files.

[dmik]

I successfully merged the code, had to resolve dozens of conflicts. The next task is to build it. Expect surprises here.

[dmik]

First of all, they require a new Python (2.7 or above which I took from here: http://os2ports.smedley.id.au/index.php?page=python). However, this doesn't want to work out-of-the-box complaining about the missing_virtualenv thingy. The previous version of Firefox worked just fine w/o it but now it seems to fail. I have to understand what it is and why it fails.

[dryeo]

virtualenv is now a strict requirement. Lots more Python scripts as well. I saw an interview with the Python guy, he seems to think that everything should be done in Python. As an aside, I've been using 2.7.3 for quite a while unluckily here it locks the system hard (no numlock) during linking of xul.dll, apparently while generating the list file

[dmik]

In short, virtualenv is a way to build a localized python environment where the packages do not interfere with the system installation of python. It is used in mozilla to create a python environment for maintaining the build process by installing a set of necessary packages.

The current version of mozilla relies quite a lot on this environment. The previous one had only a single python package requirement, in the current one there is a dozen of them.

Unluckily, our python executable seems to not support virtualenv which somehow relies on a feature that moving (symlinking) the executable to a different location should change the value of sys.prefix (the directory where python searches for packages etc.). I will quickly change the python source to see if it's possible to add such a support. If not, well, there are two options: a) finalize kBuild port; b) somehow manage system-wide installation of packages needed for mozilla.

[SilvanScherrer]

If our python can't be made to work with virtualenv (which would be really bad), then i see only option a) as a solution.

[dmik]

Okay, in case of python 2.6 by Yuri Dario I see that there are lines like this in the patch he applies:

#ifndef __KLIBC__ // YD force hardcoded build prefix

These lines seem to force the hard-coded sys.prefix value (/@unixroot/usr) to be always used. Which obviously breaks the virtualenv functionality. I need to build Python myself to see if it's true. And also ask Yuri why he did that in the first place.

WRT Paul's builds, I don't know which patches he uses (no sources or diffs available on his site) but I think that the prefix detection is just completely broken there as it always needs PYTHONHOME to be set and simply refuses to start otherwise.

[dmik]

This is it. Undoing these Yuri's patches fixes the sys.prefix behavior but I get more problems. In particular, it complains that /@unixroot/usr/lib/python2.6/os.py doesn't start with C:/USR (C:/USR is what the normal sys.prefix value looks like with Yuri's patches disabled — it's in fact the same as /@unixroot/usr but the code that checks it is of course not aware of that).

[dryeo]

On 03/06/14 03:20 PM, Dmitriy Kuminov wrote:

WRT Paul's builds, I don't know which patches he uses (no sources or diffs available on his site) but I think that the prefix detection is just completely broken there as it always needs PYTHONHOME to be set and simply refuses to start otherwise.

The build of 2.7.3 I'm using came from Steven. I've always set PYTHONHOME and virtualenv almost works (never had to remove it) though always dropped the last part of the path, eg from FF15, ... PYTHONHOME is set. You must activate the virtualenv before using it New python executable in ./_virtualenv/bin/python.exe ERROR: The executable ./_virtualenv/bin/python.exe is not functioning ERROR: It thinks sys.prefix is u'g:/cc-release-15/mozilla/obj-fb' (should be u'g:/cc-release-15/mozilla/obj-fb/_virtualenv') ERROR: virtualenv is not compatible with this system or executable

[StevenLevine]

Dave, are you sure you are really using my build? The last build I did was back in May, 2013 and it was statically linked. Paul's build dynamically links to python27.dll. What I do recall doing is providing you with some patched .py files (transaction.py, urllib.py and util.py.

[dryeo]

On 03/06/14 07:41 PM, StevenLevine wrote:

Dave, are you sure you are really using my build?

My mistake, you said you'd been working with Paul to generate a stable python 2.7.3 hg 2.6.3 combo and pointed to build and fixed dll at Pauls site.

[dmik]

Well, in my case none of Paul's builds work (tried 2.7.2 and 2.7.5) — they fail to set the sys.prefix value virtualenv expects. For instance, 2.7.5 thinks it's / which is totally wrong. And anyway, if it gives you the errors you mention, it means virtualenv doesn't work for you. At best, it will copy all the stuff needed for building mozilla to some system location (where Python itself lives) which is from the virtualenv's POV is completely wrong.

In my case it seems that there are other configure problems that prevent it from further execution in my case. I.e. that this virtualenv failure is not fatal for configure (as your case also shows). Investigating them as well.

[dmik]

I'm a bit stuck with this task. There are too many things broken ATM that prevent virtualenv from working. First of all there is a bunch of LIBC problems (see http://trac.netlabs.org/libc/ticket/286). Then, there is a bunch of (partly related) python problems in both the RPM (Yuri's) and Paul's builds — primarily, the inability of both to dynamically set sys.prefix depending on python.exe location (the key feature virtualenv relies on).

Fixing this at the virtualenv-only level looks way too hackish to me — I would never commit something with that amount of dirty hacks and this would in turn not let anyone else build Mozilla (i.e. use the results of my work), which I consider as too bad.

I first need to fix Python itself (at least to some degree) so that only sane fixes in virtualenv are required. And now I'm a bit stuck with the Python build. The RPM build is not fully reproducible so far (fails when it comes to the libffi bits; actually, even earlier but I somehow managed to build it to that point). Looks like Yuri has some custom environment that is not fully reproduced with python.spec from .src.rpm... But I went quite far with this task so I want to finish it now.

What I need so far is to rebuild Python with no /@unixroot involved to make virtualenv work and see how it goes. If virtualenv works OK, then I have an idea on how to temporarily fix Python in an acceptable way (until the mentioned LIBC tickets are fixed — which will supposedly take quite a while) in a way that will let os.path.realpath work as it should and therefore use /@unixroot again.

Quite a tricky plan, huh?

[dmik]

I finally rebuilt Python, made sys.prefix work as desired and also almost made virtualenv work. It does install itself half way through and then fails because os.path.relpath is not found. In fact, this method is missing from both os2emx and os2knix flavors of python 2.6 as well as in python 2.7. Will implement it now.

The method was added in Python 2.6 so it must have been left out when someone updated the OS/2 build to 2.6 from earlier versions and missed this addition.

[dmik]

Okay, virtualenv now fully works, hooray! I can't commit my python patches though since net labs SVN is down again.

The next problem is that Python 2.6 lacks the collections.OrderedDict object (new in Python 2.7). This means that I will have to update the RPM build of Python to 2.7 in order to go further with the Mozilla build.

[wztest]

Good work, so 2.6 is fine now. Why is it required to update the rpm build?

[dmik]

Because the Paul's 2.7 build is not suitable for virtualenv (doesn't support dynamic sys.prefix w/o PYTHONHOME which is not allowed in virtualenv).

[wztest]

It means to simply go to Python 2.7 and add again the missing pieces, is there more missing from the current port?

[dmik]

I will better do the other way around: the RPM build is more valuable while the amount of work is basically the same.

[dmik]

Before merging with 2.7 I first decided to make sure Python 2.6 works with my patches in /@unixroot mode but now it fails to build. Even if I revert to what it was before. I'm stuck hard with it as I even don't understand how it could build before: the build process uses python that in turn calls setup.py that builds some parts of python (e.g. the binary _functools module). However, setup.py needs the site.py module which eventually refers to _functools which is not yet built! I.e. there is a circular dependency and I don't know how to break it.

[dmik]

Okay, I know how I got it build before. First time I used the same prefix /@unixroot/usr as the RPM build (which I already have installed). As a result, my newly built python was simply using the existing _functools module from the RPM installation (due to the hard-coded path) which was there just by accident. Then I changed the prefix to something else, but a new copy of _functools was already built by that time in my build tree (by setup.py) so it would not fail even given that the hard-coded path was changed.

This is definitely wrong. It is clearly stated in the Python README that the second phase of the build process (setup.py) uses the newly built python and it isn't mentioned anywhere that you should have a working installation of python in order to build a new one. Looks like a complete failure of the python guys to me.

[StevenLevine]

It is definitely not porter-friendly. Does this help?

http://wiki.osdev.org/Porting_Python

[dmik]

Steve, thanks for the link. Yes, perhaps this is what they do for bootstrapping it on a fresh platform.

However, I found a cleaner solution. The modules required for setup.py are actually statically linked into the python DLL. However, there were a few modules missing from there. I don't know why: perhaps the python maintainers just missed the point when they became necessary (because nobody builds a python on a python-free system perhaps). See http://trac.netlabs.org/rpm/changeset/385.

All my other python fixes are also in (starting from http://trac.netlabs.org/rpm/changeset/381).

Switching to 2.7 now.

[dryeo]

On 03/13/14 02:11 PM, Dmitriy Kuminov wrote:

All my other python fixes are also in (starting fromhttp://trac.netlabs.org/rpm/changeset/381).

Which source did you start from?

[dmik]

Not me. Yuri started it there. It's 2.6.5 + Paul's + Bird's patches. I'm upgrading it to 2.7.6 now.

[wztest]

In my view hardcoded pathes are never a good idea, esp. @unixroot in this case.

[dmik]

/@unixroot is not a hardcoded thingy — it may be changed dynamically. It gives us "the best of two worlds" (and simplifies porting Unix programs a lot).

[dmik]

Unfortunately, impossible to update today. Netlabs is unusably slow ATM so I can't commit the vendor branch, can't do merge etc. Additionally svn commit has just crashed with the following:

Transmitting file data ..............................................................................................................................svn: Commit failed (details fol
low):
svn: While preparing 'D:/Coding/rpm/python/vendor/PCbuild/pythoncore.vcproj' for commit
svn: Inconsistent line ending style

Killed by SIGSEGV
pid=0x290b ppid=0x0814 tid=0x0001 slot=0x00af pri=0x0200 mc=0x0001
C:\USR\BIN\SVN.EXE
SVN 0:000a2f66
cs:eip=005b:000b2f66      ss:esp=0053:0043fc10      ebp=0043fc28
 ds=0053      es=0053      fs=150b      gs=0000     efl=00010202
eax=20202020 ebx=20202020 ecx=00000000 edx=1b00d0c8 edi=0043fdc0 esi=00000000
Process dumping was disabled, use DUMPPROC / PROCDUMP to enable it.

I don't know if the crash is caused by line endings or slow netlabs. Impossible to wait for several more hours to check this now. I need the netlabs speed to get back to normal first.

[dmik]

Committed vendor update of Python 2.7 to netlabs. Switching to the merge phase.

BTW, the problem with the svn crash had gone after I fixed line endings in PCbuild/pythoncore.vcproj. This is something to look at by Paul Smedley.

[wztest]

In case there are netlabs svn problems try to contact adrian. When you have problems to commit changes the same applies for checkouts? About the crash with svn, the hint of the problem with line ending was already in the crash screen: "svn: Inconsistent line ending style". The fix was to use unix style line endings?

[psmedley]

I fixed some apr file_io bugs in order to get svn 1.7 working. A test build is available fromhttp://smedley.id.au/tmp/subversion-1.7.16-os2-20140310.zip

[StevenLevine]

Regarding line endings. The rpm/yum repos use svn:eol-style native. If there is a line ending problem, the file needs to have OS/2 style line endings.