Update Firefox to 38 ESR

[SilvanScherrer]

update the current firefox 31 ESR to latest 38 ESR release.

[dmik]

Updated vendor-esr in 8112532fbc9147cf535799e402c55ad5a8c2359f to FIREFOX_38_1_0esr_RELEASE. There is quite a bit of changes, I must say: 69,311 changed files with 33,864 additions and 35 deletions. As usual, they renamved/moved a lot of stuff but also a lot of modifications here and there plus further improvement of the build system. I will have to look through sensitive places.

Also, an attempt to merge will further show how many problems we may have making this build work on OS/2. Stay tuned.

[dmik]

Merging gave us 119 conflicts in total. Solving them.

[dmik]

I've patched around 2/3 but there is also some change in the python-based build subsystem (expected) which affects OS/2. In particular, the old way of specifying the short DLL name doesn't work any more. They use the python function to produce DLL names now and I need to add OS/2 support there.

[dmik]

Another interesting new thing is OOP support for the print dialog (i.e. running all the print magic in a separate process). This is to protect FF from crashes inside the print driver which are quite typical on Windows and so it is enabled there by default (using PROXY_PRINTING). The same could be used on OS/2 as well (to protect from taking down the whole Firefox like in #75) but unfortunately OOP isn't yet fixed so makes no sense to enable it now. I made a note in #106.

[dmik]

Another thing I spotted is that they removed a bunch of DOS encodings (actually all IBM* ones except CP866 and some Thai) that are not "Web-exposed", the relative ticket is https://bugzilla.mozilla.org/show_bug.cgi?id=997124. On OS/2 this, in addition to the well-known ones like IBM850 (Central European) also includes removal of several OS/2-specific encodings (IBM869 which is Greek, IBM1125 which is Ukrainian and IBM1131 which is Belorussian). From what I understand, removing this will break OS/2 locale support (intl/locale/os2/os2charset.properties) for these languages (that depends on these encodings) in terms of converting clipboard, file names etc. This means that we should leave them all in — this is what I will do for now.

[dryeo]

Looks like they'll need restoring for Thunderbird as well. I notice that https://bugzilla.mozilla.org/show_bug.cgi?id=711101#c3 mentions that we use a different "Shift_JIS" on OS/2 then elsewhere

[dmik]

Yup. Anyway, I restored it right away (as it causes a merge conflict with the mentioned OS/2 charsets that I restored when upgrading from 24 to 31). However, I second the approach from the Mozilla bug stating that the set of charsets should be consistent across platforms and hence I removed ifdef XP_OS2 around those 3 charsets.

[dmik]

Did another batch of merges and found out that they performed some significant changes in the system exception handling in JS (the one I finally fixed to make it not crash in esr31). I have to redo some stuff...

[dmik]

Also my fixes in character set conversion in JS need some adjustment due to significant changes to some functions.

[dmik]

Hmm, some how my fixes landed up in #132 instead of this issue. That's wrong. Moving them here.

From what I also see in the code, they switched to C++11 mode. This means that a lot of #ifdefs supporting older compilers are gone. I hope this will not result into too many build break on OS/2 (but there will be some due to missing defines, for sure — those are easily fixable though).

A small bunch of files is left that were patched by me but deleted upstream. I need to figure out if these patches are still relevant and move them to a new location.

[dmik]

Btw, they also heavily resorted some things (deleted a few files etc). in the FFmpeg code which I attempted (unsuccessfully) to make it work on OS/2. This needs some patch reorganization as well. This, in addition with the build system changes, seems to be last thing to do. The rest is merged.

[dmik]

Ah and they also changed the platform specific TLS code location in gfx/angle that also needs to be aligned.

[dmik]

Re FFMPEG see #102. In this merge, I will drop parts of https://github.com/bitwiseworks/mozilla-os2/commit/29677e4d5baadb814e5cf7fce020866844da3cb2 I did earlier (in particular, FFmpegFunctionList.h and FFmpegRUntimeLInker.cpp) because it's questionable how to port it to OS/2 (the new code tries to import every function from the avformat DLL but on OS/2 it doesn't contain all needed exports). In fact, only the DecoderTraits.cpp experimental patch will survive the merge. The rest will be done within #102.

[dmik]

Something strange with the exception handlers in JS in ERS38. In earlier versions these handlers were used to process exceptions (page faults) caused by setting some JS memory to protected state (no access) to interrupt the currently executed Asm/Jit code. The handlers would catch these exceptions, set it the memory back to non-protected state and continue execution. However, in the current version I see that the protect/unprotect functions (AsmJSModule::[un]protectCode) have disappeared and that exception handlers only do some meaningful stuff in 64-bit mode, by always returning false in 32-bit mode (which means the process will eventually terminate due to the handled page fault). The protect/unprotect code seems to have moved to GCRuntime::[un]protectRelocatedArenas but this only works in DEBUG mode and it seems that it intentionally makes a condition for a page fault (and a subsequent application crash) if something goes wrong.

All in all, this means that in 32-bit mode the exception handler is completely useless (does nothing). This is strange. But I can't really figure out what's going on there on merge. I will blindly do it as on other platforms (i.e. keep it useless) and then the real build and run will show things.

[dmik]

To the previous comment: the situation is the same on all platforms, including windows, mac and linux — they do nothing in exception handlers in 32 bit. Which looks a bit strange as it's unclear now how they implement kicking into the asm/jit code... HMM, I checked the mozilla-central version and I see that it is hugely redone compared to ESR38: http://hg.mozilla.org/mozilla-central/file/29b2df16e961/js/src/asmjs/AsmJSSignalHandlers.cpp. In particular, there are no 64-bit ifdefs meaning the old behavior is back. Looks like the code in ESR38 is just bogus.

[dmik]

Here is the latest version from ESR38: http://hg.mozilla.org/releases/mozilla-esr38/file/e72e9054d9eb/js/src/asmjs/AsmJSSignalHandlers.cpp. And BTW there were a couple of ESR38 updates already, 38.1.1 and 38.2.0 (the latest one). So I'm not sure I understand why the right code is still absent in 32-bit mode there. Either they achieve it in a completely different way or it's a bug indeed which is still not fixed in this ESR.

BTW, we should update to 38.2.0 I guess, anyway.

[dryeo]

On 08/20/15 06:44 AM, Dmitriy Kuminov wrote:

BTW, we should update to 38.2.0 I guess, anyway.

38.1.0 has a major cross-platform vulnerability in the js pdf viewer (allows full access to the file system) and 38.2.0 includes stability fixes.

[dmik]

Yup, I read it. But the exception stuff is as in 38.1.0.

[dmik]

Another thing to mention is that I had to adjust the JavaScript string conversion code I did in #114 because they changed this part as well.

[dmik]

The build system has been adjusted as well. SHORT_LIBNAME is deprecated now and one uses e.g. XPCOMBinaryComponent('browsercomps', short_name='brwsrcmp') instead. I didn't test my changes yet so there may be bugs (will fix them when building).

The resulting merge is committed in a97c6193783c0aa85aedc0cd02b5aaf602e59b3c. I will now merge the latest 38.2 changes (should be fast and easy) and then try to build the beast.

[dmik]

I updated vendor-esr to 38.2.0 in c8769f7e0f499179ee6a35e30511959a427d2b26 and merged it (flawlessly, except one minor conflict) in f61cc7ae1f8a0f80051e1ccbf20d70e29db5d255. Will now try building the beast.

This defect will be left open as a meta-ticket for major build issues I encounter. I will close it after sorting them out.

[wztest]

It's very strange that the ESR branch contains this kind of broken code. Any news from the first build?

[dmik]

Not much so far, some stupid python virtual env breakage.

[dmik]

The problem is in the pip module. The 38.x version uses pip 6.0.6 while 31.x was using 1.5.4.. The bad thing is that the module is supplied as a zip archive ("wheel") so we will have to fix it, repack and commit back as zip.

[dmik]

Created #135 for the virftualenv issue.

[dryeo]

On 08/25/15 07:58 AM, Dmitriy Kuminov wrote:

I updated |vendor-esr| to 38.2.0 in c8769f7 https://github.com/bitwiseworks/mozilla-os2/commit/c8769f7e0f499179ee6a35e30511959a427d2b26 and merged it (flawlessly, except one minor conflict) in f61cc7a

38.2.1 has been released due to a security hole in 38.2

[dmik]

Aha. Nice. Will have to pick it up as well.

[dmik]

Meanwhile, I fixed all failures related to moz.build files so now configure finishes w/o errors. I will update it to 38.2.1 and then try to build.

[dmik]

Merged 38.2.1 in efa9ad44408894384bc2b4c8e849bd8cd8b09b7f. No conflicts whatsoever.

[dmik]

Built nspr and some helpers, got a strange failure in js/src/Makefile, unmatched endif. Something went wrong as this is a generated file.

[dmik]

Ok, that was a tiny merge typo. Fixed locally, will try to build further and see if there are more thing like that, before committing.

[dmik]

Now I get some strange error when building some RDF code:

c++ -o nsRDFResource.o -c -I../../dist/stl_wrappers -I../../dist/system_wrappers -include D:/Coding/mozilla/master/config/gcc_hidden.h -DAB_CD=en-US -DNO_NSPR_10_SUPPORT -ID:/Coding/mozilla/master/rdf/util -I.  -I../../dist/include   -ID:/Coding/mozilla/master-build/dist/include/nspr -ID:/Coding/mozilla/master-build/dist/include/nss         -idirafter D:/Tools/OS2TK45/h -DMOZILLA_CLIENT -include ../../mozilla-config.h -Uunix -U__unix -U__unix__ -MD -MP -MF .deps/nsRDFResource.o.pp -idirafter D:/Tools/OS2TK45/h -Wall -Wempty-body -Woverloaded-virtual -Wsign-compare -Wwrite-strings -Wno-invalid-offsetof -Wcast-align -march=i486 -mtune=i686 -fno-exceptions -fno-strict-aliasing -Zomf -fno-rtti -fno-exceptions -fno-math-errno -std=gnu++0x -pthread  -DNDEBUG -DTRIMMED -g -O3 -fomit-frame-pointer      D:/Coding/mozilla/master/rdf/util/nsRDFResource.cpp
In file included from ../../dist/include/jspubtd.h:19:0,
                 from ../../dist/include/nsCycleCollectionTraversalCallback.h:10,
                 from ../../dist/include/nsCycleCollectionNoteChild.h:13,
                 from ../../dist/include/nsCOMPtr.h:32,
                 from D:/Coding/mozilla/master/rdf/util/nsRDFResource.h:9,
                 from D:/Coding/mozilla/master/rdf/util/nsRDFResource.cpp:6:
../../dist/include/jspubtd.h:35:21: error: 'dllimport' implies default visibility, but 'struct JS::CompileOptions' has already been declared with a different visibility
 class JS_FRIEND_API(CompileOptions);
                     ^
../../dist/include/jstypes.h:85:45: note: in definition of macro 'JS_FRIEND_API'
 #  define JS_FRIEND_API(t)   MOZ_IMPORT_API t
                                             ^
../../dist/include/jspubtd.h:36:21: error: 'dllimport' implies default visibility, but 'struct JS::ReadOnlyCompileOptions' has already been declared with a different visibility
 class JS_FRIEND_API(ReadOnlyCompileOptions);
                     ^
../../dist/include/jstypes.h:85:45: note: in definition of macro 'JS_FRIEND_API'
 #  define JS_FRIEND_API(t)   MOZ_IMPORT_API t
                                             ^
../../dist/include/jspubtd.h:37:21: error: 'dllimport' implies default visibility, but 'struct JS::OwningCompileOptions' has already been declared with a different visibility
 class JS_FRIEND_API(OwningCompileOptions);
                     ^
../../dist/include/jstypes.h:85:45: note: in definition of macro 'JS_FRIEND_API'
 #  define JS_FRIEND_API(t)   MOZ_IMPORT_API t
                                             ^
../../dist/include/jspubtd.h:38:21: error: 'dllimport' implies default visibility, but 'struct JS::CompartmentOptions' has already been declared with a different visibility
 class JS_PUBLIC_API(CompartmentOptions);
                     ^
../../dist/include/jstypes.h:77:45: note: in definition of macro 'JS_PUBLIC_API'
 #  define JS_PUBLIC_API(t)   MOZ_IMPORT_API t
                                             ^
../../dist/include/jspubtd.h:104:21: error: 'dllimport' implies default visibility, but 'struct JSTracer' has already been declared with a different visibility
 class JS_PUBLIC_API(JSTracer);
                     ^
../../dist/include/jstypes.h:77:45: note: in definition of macro 'JS_PUBLIC_API'
 #  define JS_PUBLIC_API(t)   MOZ_IMPORT_API t
                                             ^
../../dist/include/jspubtd.h:211:21: error: 'dllimport' implies default visibility, but 'class JS::AutoGCRooter' has already been declared with a different visibility
 class JS_PUBLIC_API(AutoGCRooter)
                     ^
../../dist/include/jstypes.h:77:45: note: in definition of macro 'JS_PUBLIC_API'
 #  define JS_PUBLIC_API(t)   MOZ_IMPORT_API t
                                             ^

I don't really understand what is this bitching about. The files themselves haven't changed since ers31. So this must be either changed compiler options or some new conditional define logic. Something like declared as dllexport and then declared again as dllimport.

[dmik]

Fixed in 12a276b01ce16b1efc950290b7cb233287c09f66. The trick was that the guilty declaration was included via the -include switch (which is generally evil).

[dmik]

The esr38 branch got a kewl hack in the end of 2013 described in this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=938437. The idea of this hack is to make the linker automatically gather pointers to static components (the ones defined inside XUL.DLL) into an array (the global kPStaticModules variable) instead of maintaining this array's contents manually via the initializer list in a separate file (former nsStaticXULComponents.cpp, now deleted). The rationale is to free the developers from updating this separate file each time a new component is made/unmade static.

This hack requires support for the section attribute (__attributre((section("BLAH")))__) in GCC but our OS/2 target doesn't support this attribute (because gas in a.out mode doesn't support .section directives since sections are not supported by the a.out format per se).

Fortunately there is an EMX hack in emxomfld that provides this specific functionality to create such arrays at link time (what a luck). It is used e.g. by LIBC to gather all CRT init functions from all over the source files (the _CRT_INIT1 macro) and then automatically call them at program startup (so a very similar purpose to what the Mozilla guys want). I will try to utilize this approach in Mozilla.

[dmik]

The .section problem is solved. Now it fails when compiling Japanese unicode conversion tables. Strange, need to figure out what's going on.

[dmik]

The Japanese break was easy to fix, see http://hg.mozilla.org/releases/mozilla-esr38/rev/52b0ec2f0522 where they dropped some deprecated IBM943 stuff and 0342010c52a8c4b6ebbdf31b9aec05331199f561 where I restored it (I'm not a Japanese code page expert so someone with better knowledge should check if the current state is accurate i.e. matches what we had in ff24 and earlier).

There are now small build breaks here and there, fixing them.

[wztest]

Funny that they recognized that they removed some required OS/2 specific code. Good luck for the next hopefully only small bug fixes.

[dmik]

Going further. Discovered quite a bit of gfxFont related changes in the new code which are not as minor as the above small fixes. Working on them now.

[dmik]

Ported gfx changes in 36dc4a003b1fbb336062a9fd5068f2c07ef64f54, hopefully it will work.

[dmik]

I'm now building the widget subdirectory and the number of changes there is rather big. So will be the next commit (or two).

[dmik]

The widget part is done. We are close to the end.

[dmik]

There is a limitation of the new logic collecting used libraries for targets. They are defined with USE_LIBS in moz.build but this var contains references to python definitions rather than real library names. These definitions are then processed and necessary platform specific defaults are added to them, including the library prefix and suffix. LIB_SUFFIX' islibon OS/2 and it generally works. However, there islibffibuilt from the original sources and the result on OS/2 is namedffi.a(sinceldis used). However, due toLIB_SUFFIX, python makes itffi.libinstead and fails because it cannot find this file. In the previous versions there wasSHARED_LIBRARY_LIBSwhere we specifiedffi.a` manually on OS/2 but now this mechanism has gone and it's really strange that they didn't add support for overriding suffixes (and other bits) in the new mechanism. I need to think where to fix that.

[dryeo]

Originally I patched the makefile to run emxomf on the aout ffi.a to get ffi.lib. Can something similar be done?

[dmik]

Not quite, because:

1. We use autotools now to generate configure and libtool for libffi instead of versioning these files.
2. There is a bug in libtool (has been there forever), see http://trac.netlabs.org/ports/ticket/72.

I hacked libtool in libffi manually for the time being. Later I will release a new libtool package.

[dmik]

Turns out that JS builds after everything else in esr38 (it used to be built at the beginning, after NSPR). I did a lot of blind porting there when merging and there were a few syntax minor build breaks. However, they also changed some other memory management functions (Memory.cpp) which didn't cause any conflicts so I need to port them now (doesn't look too big)

[dmik]

I adopted Memory.cpp changes to make it build. However, a deeper analysis of the new code may be needed. This part is responsible for how memory is allocated and freed for JS and may be related to #55. Will postpone it for after the first build of esr38 is done.

Also, esr38 is by default built in unified source mode where they generate synthetic source files that include a group of real sources and then build these sources (perhaps, this is done to improve build speed, I haven't investigated). This exposes the infamous os2.h single inclusion problem (i.e. when this file is included once with a restricted set of #define INCL_* options, any other attempt to include it with potentially a different set of options will be ignored resulting in unresolved references at compile time). To make this unified source mode work, we must include everything from os2.h (using something like #define INCL_DOS/#define INCL_PM/etc. We already do it in several places but we should actually do it everywhere (as a rule of thumb). I did that for JS for now.

[dmik]

Encountered a new library in esr38, media/libstagefright. Used for some audio/(video?) processing. Need to adapt it to OS/2.

[LewisR]

libstagefright, as in the one patched in FF 40? http://www.eweek.com/enterprise-apps/firefox-40-features-updates-patches-including-one-for-stagefright.html

Chance of backporting whatever was fixed to tighten it up before releasing something we might not want to send out into production?

[dmik]

@LewisR 38 is ESR (extended support release) and there is FIREFOX_38_3_0esr_RELEASE already (released on 16.09). So if it's critical it is there as well, I'm sure. We will update to this no doubt. And generally back porting from other (especially newer) branches is a bad idea. Thanks for pointing out though.

[LewisR]

I see that the fixes for libstagefright were (back)ported to 38.2ESR (https://www.mozilla.org/en-US/security/advisories/mfsa2015-83/).