Make it build with kbuild

[dmik]

kBuild is a much more adequate build system than the autoconf hell. Mozilla is big, building it with autoconf is very inconvenient so we need to port it to kbuild as the first step.

[dmik]

I'm a bit stuck with this: can't cleanly configure it with ./configure to get the sample makefiles and generated headers. It fails at various stages, I have to sort them all.

Yes, this is why I want to get rid of autoconf. It has so many problems in general and on OS/2 in particular that it makes it very very difficult to live with.

[dmik]

Configure-style Makefiles are built, doing the kmk side.

[dmik]

HMM, somehow my comment wasn't added yesterday, here it is:

There is a small generic problem: the Mozilla soucres assume their headers are already copied to the corresponding subdirs of the "dist" area and include them from these subdirs (while the headers originally live next to the sources). I think this is a bad practce but we obviously have to deal with that (to avoid changing each source).

The problem is that kBuild doesn't expect such a situation so it only can execute INSTALLS targets after the build stages, not before. We will have to write a custom rule for that.

[dmik]

Anyway, there is quite an easy solution to that (I had to figure it out though), see commit da09ed63e89244f356b7257319ff48035a2619f0 for that.

[dryeo]

IIRC some of the headers are generated during the build process from the idl files using the python script xpcom/idl-parser (used to be xpidl.exe which is where the dependencies on glib and libidl come from) during the export phase of the build. Actually looking at my build.log for 17esr there are a few python scripts involved in generating some of the headers. Some of the headers also go to the dist/sdk directory which may still be needed for building some extensions such as enigmail.

[dmik]

Yeah, sure, python will be used as designed. Generation of headers before compiling is also not a question for kBuild. We did it when adding Qt3/Qt4 support into it (it works quite nicely btw, a good alternative to qmake).

[dmik]

Got further. Now kmk can generate various settings and headers in a nice manner (using perl/python — just as in the original build environment).

[dryeo]

Not sure if you've already done this but js (mozjs.dll) should be built shared right from the beginning to make xul.dll smaller (needed for debug symbol builds) and avoid breaking binary extensions such as lightning. This was done with ac_add_options --enable-shared-js doing the configure based build.

[dmik]

Thanks! I haven't built it yet but soon.

[dmik]

JFTR. The original build system is completely a show stopper, just bumped into it again. You can properly build it only from root, because the upper makefiles define things used by nesed makefiles and if you try the nested one directly, it will fail in many cases. E.g. you can't build NSPR from its own directory. So it is very good that we're changing it to kbuild.

BTW the very same problem exists in OpenJDK and it also makes a dev suffer. I don't know how these linux people can live like that (well I also don't know how e.g. African people can live naked in sands and durt)...

[dryeo]

It isn't that bad, usually you only have to build from the top directory after clobbering your tree, which once things were working wasn't too often, usually only when the configuration or makefiles changed. NSPR used to have os2-gcc.mk and os2-vac.mk for standalone builds but as they were unmaintained...

[dmik]

Well, if you are a dev, you frequently change one or two files here and there and you don't want to initiate a full build that does many unnecessary things (regenerates generated files, copies headers etc). In kmk you can run make from any dir with Makefile.kmk and it will only rebuild (and reinstall, if requested) what's actually changed.

[dmik]

Dave, do you by chance know if MULTIPLE/NONSHARED is actually necessary for NSPR4.DLL and other NSPR DLLs? This causes each process to have its own copy of NSPR data which in particular means that if you have two Firefox processes running, there will be two distinct NSPR setups that can't cooperate. I'm not sure that this is the right thing to do so I leave this MULTIPLE/NONSHARED parameter out and will add it back only if we find it's really necessary.

[dryeo]

Apparently it's been there since the Netscape days, at least it was in rev 3.6 when Henry got GCC and VACPP support checked in (http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&file=rules.mk&branch=&root=/cvsroot&subdir=mozilla/nsprpub/config&command=DIFF_FRAMESET&rev1=3.6&rev2=3.7) so I'd guess it's just left over from when only one browser instance could be ran. In reply to your previous message, kmk doesn't seem much different then the configure && make depends && cd toworkdir and make way (configure and make depends only needs to be run once) It's just a matter of the rules and granted Mozilla has gotten really sloppy in some ways since they switched to XUL builds they would have done the same thing with whatever type of make they used. With kmk don't you still have to generate your build tree or are you doing in tree builds?

[dmik]

Strange. My last comment has lost somehow. Here it is:

Thanks for the info about the DLLs! We will go the share way for now.

Kmk is different, just try to feel the difference. You don't need to do separate depends stages or create the output directory tree: it's all done as needed. With kmk, you start the make command from within the source tree, but all output goes to a dedicated tree (which defaults to /out but you can change it to whatever you want). You can run kmk from any subdirectory within the source tree that contains Makefile.kmk and only this subdirectory will be rebuilt/reinstalled/whatever (only the changed files, of course). Also, kmk fully controls output so you will only see what you want to see during the build (by default it's only one nice per each compilation unit w/o any annoying 1000km long command lines etc). And many other benefits.

[dmik]

BTW, I have to fix the configure build as well while I go since I need it to make sure that I won't miss anything in kmk. This is almost done, I'm past the xpcom component already. Just a few things left.

[dmik]

In addition to all other annoying things, the Mozilla sources have dirty line endings with trailing spaces in virtually every file. Looks like their devs miss the basics of the programming culture. This annoys a lot. Sigh -(

[dmik]

I reached the build of the new version of the Angle library in gfx. I will try to align its OS/2 part so that it builds instead of rolling it back to the old version. Its new functionality may be definitely required later and our roll back won't be ever accepted as a patch for sure.

[dmik]

Regarding the failed Angle build. I get some strange error complaining that the prototype for pp::Preprocessor::init() does match any in class while visually there is an exact match. Dave, is that what you were seeing too? This doesn't look like a missing GCC feature but rather a bug to me...

[dryeo]

Yes, that was the problem. I don't know if bug or feature better describes it but apparently the behaviour of G++ has changed. There are few similar build failures and comparing to older versions the difference was whether arguments were in the function prototype in the header file. For example I fixed a similar errors in gfx/gl by doing something like

diff --git a/gfx/gl/GLContextProviderImpl.h b/gfx/gl/GLContextProviderImpl.h
--- a/gfx/gl/GLContextProviderImpl.h
+++ b/gfx/gl/GLContextProviderImpl.h
@@ -62,16 +62,16 @@
     CreateOffscreen(const gfxIntSize& aSize,
                     const ContextFormat& aFormat = ContextFormat::BasicRGBA32Format,
                     const ContextFlags aFlags = GLContext::ContextFlagsNone);

     /**
      * Get a pointer to the global context, creating it if it doesn't exist.
      */
     static GLContext *
-    GetGlobalContext( const ContextFlags aFlags = GLContext::ContextFlagsNone);
+    GetGlobalContext();

     /**
      * Free any resources held by this Context Provider.
      */
     static void
     Shutdown();
 };
diff --git a/gfx/gl/GLContextProviderOSMesa.cpp b/gfx/gl/GLContextProviderOSMesa.cpp
--- a/gfx/gl/GLContextProviderOSMesa.cpp
+++ b/gfx/gl/GLContextProviderOSMesa.cpp
@@ -247,17 +247,17 @@
     {
         return nullptr;
     }

     return glContext.forget();
 }

 GLContext *
-GLContextProviderOSMesa::GetGlobalContext(const ContextFlags)
+GLContextProviderOSMesa::GetGlobalContext()
 {
     return nullptr;
 }

 void
 GLContextProviderOSMesa::Shutdown()
 {
 }

This was basically reverting to an earlier version. I tried similar stuff in angle and got various errors all about missing prototypes when I could see matches. Our GCC is getting old and I'd guess the code would work in new enough versions or perhaps it's a reflection of the code being meant for VC++ though I assume it compiles with at least mingw.

[dryeo]

Hmm, the markup changed some plus and minus signs to bullet points.

[dmik]

Dave, I've fixed markup in your comment above: you should start and end pre-formatted blocks with thee backticks, not with three dots. And to get syntax highlighting, the name of the syntax mode should immediately follow the opening triplet of backticks mode where mode is syntax mode.

[dmik]

Okay, found something more. The problem is not the old GCC version as here is nothing new (language feature wise) in this library. It must be either a generic GCC 4.4.6 crossplatform bug (unlikely) or a big in my patch dealing with _System and friends. GCC 4.5.2 works well on this souce, as well as GCC 3.3.5 CSD4. I have to check that I'm afraid.

Dave, do you have some GCC 4.4.6 build from Paul (i.e. not from RPM) where my fixes are not yet included? My internet connection is very weak now and I can't download that ATM.

[dmik]

BTW, the problem in my example with Preprocessor goes away if I put the include before including "Preprocessor.h". This means that somehow changes the context so that after it visually the same declaration produces a different result incompatible with the previous one. This is kind of weird. We're not going to work around it that way. The proper solution is to fix GCC.

I'm building such files with GCC 4.5.2 for now.

[dryeo]

I got my 4.4.6 directly from Paul. Not all versions I got from Paul worked and probably your fixes were not included. I just reverted to 4.4.1 and angle built fine with a small patch. 3.3.5 is broken for building Mozilla, cross-platform bug fixed in 3.4.x that Mozilla refused to accept workarounds for. I did try an early build of 4.5.2 and had the same problem. Should I ask Paul for the version you're using? Here's my patch

# HG changeset patch
# Parent 2d992544f61d5c23186bed68ac5b64a02a7c01e6
# User Dave Yeo <daveryeo@telus.net>
Enable angel to use nspr on OS/2

diff --git a/gfx/angle/Makefile.in b/gfx/angle/Makefile.in
--- a/gfx/angle/Makefile.in
+++ b/gfx/angle/Makefile.in
@@ -93,16 +93,18 @@
         VersionGLSL.cpp \
         OutputESSL.cpp \
         OutputGLSLBase.cpp \
         TranslatorESSL.cpp \
         $(NULL)

 ifeq ($(MOZ_WIDGET_TOOLKIT),windows)
 CPPSRCS += ossource_win.cpp $(NULL)
+else ($MOZ_WIDGET_TOOLKIT),os2)
+CPPSRCS += ossource_nspr.cpp $(NULL)
 else
 CPPSRCS += ossource_posix.cpp $(NULL)
 endif

 CSRCS   = \
         atom.c \
         cpp.c \
         cppstruct.c \
diff --git a/gfx/angle/src/compiler/osinclude.h b/gfx/angle/src/compiler/osinclude.h
--- a/gfx/angle/src/compiler/osinclude.h
+++ b/gfx/angle/src/compiler/osinclude.h
@@ -15,16 +15,18 @@
 #if defined(_WIN32) || defined(_WIN64)
 #define ANGLE_OS_WIN
 #elif defined(__APPLE__) || defined(__linux__) || \
       defined(__FreeBSD__) || defined(__OpenBSD__) || \
       defined(__sun) || defined(ANDROID) || \
       defined(__GLIBC__) || defined(__GNU__) || \
       defined(__QNX__)
 #define ANGLE_OS_POSIX
+#elif defined(__OS2__)
+#define ANGLE_USE_NSPR
 #else
 #error Unsupported platform.
 #endif

 #if defined(ANGLE_USE_NSPR)
 #include "prthread.h"
 #elif defined(ANGLE_OS_WIN)
 #define STRICT

[dryeo]

Of course above is wrong, that's what I get for not rerunning configure :) anyways see you already did it right.

[dmik]

Yes, already did. Anyway, this all reminds me of similar problems, this GCC/2 bug may be related to this. Let's postpone it so far.

[dmik]

Dave, your diff from this comment https://github.com/bitwiseworks/mozilla-os2/issues/2#issuecomment-18951281 is not correct. The proper one I've just pushed in. Seems that GLContextProviderNull is not part of any supported mozilla build so they forgot to change a line there. See cf515773ae04695b49aacc3c7191b8bdf60e32ad.

[dmik]

Disabled some more IPC related code. And there seems to be even more.

[dmik]

Another problem: they seem to have changed something in the widgets code, for instance NS_ACTIVATE is missing now but still referenced in the OS/2 code. Investigating. Created #15 for that.

[dryeo]

Yes, there seems to be a few changes to widget/os2. The mouse handling has also changed, https://bugzilla.mozilla.org/show_bug.cgi?id=719320. Another bug where OS/2 widget code was mentioned was https://bugzilla.mozilla.org/show_bug.cgi?id=701760. Somewhere I also have a patch from Rich where he redid quite a bit of the widget code which was used for the early 10 builds accidentally. It had the focus bug. Should there be a new issue raised about fixing the widget code? Another thing that could be considered is adding gstreamer support. This would allow using libavcodec to play H264 videos. Porting gstreamer is probably outside the scope of here. https://bugzilla.mozilla.org/show_bug.cgi?id=422540

[dmik]

Dave, see #15 and #16. If you find a patch for the focus thing, it will save me some time, so you are welcome!

[dmik]

I'm back to this issue for some time since I need easy and fast rebuilds for further development (this includes the debug build).

[dmik]

As a reminder of why this task is important:

1. It will get us a convenient and fast build environment. That in particular will let us build the debug version etc.
2. While doing this I'm creating many additional units for kmk that may be later used to port other configure (and not only configure) based projects to kmk, e.g. OpenJDK.

So it's worth the effort. Much more worse than hacking configure in order to get the debug build out of it, for example.

[dryeo]

On 08/23/13 10:04 am, Dmitriy Kuminov wrote:

1. It will get us a convenient and fast build environment. That in particular will let us build the debug version etc.

Why would kmk help the debug/debug-symbol build? Configure by default already adds debug-symbols (-g) unless disabled and --enable-debug always worked until xul grew to big (even emxomfar was starting to fail in gtklayout before the switch to fake libs)

[dmik]

So you tried to get the debug build with fake libs and it worked?

[dryeo]

On 08/23/13 05:00 pm, Dmitriy Kuminov wrote:

So you tried to get the debug build with fake libs and it worked?

Only the debug-symbol build worked with Firefox 10.12. The trick was separating the JavaScript library (mozjs.dll) to bring XUL down in size. Worked with Steven Levine to figure out that somewhere there is a 32 MB object limit, Steven was planning on examining wlink to see if it could be worked around but he is busy. Debug builds stopped working long before fake libs were introduced, basically when they switched to xul builds and even before it was hit and miss with emxomfar erroring out due too large page tables (64kb limit) The IPC stuff has probably inflated xul again

[dryeo]

On 08/23/13 05:00 pm, Dmitriy Kuminov wrote:

So you tried to get the debug build with fake libs and it worked?

Just to be clear, a debug build spews lots of stuff to stdout/stderr, has lots of popups about asserts giving the option use the debugger or continue and is huge whereas the debug-symbol build is just adding -g, removing strip and the ilink optimize stuff.

[dmik]

Okay, thanks for the info. We will see soon. As I really need a debug build for further work.

Also don't forget that nobody supports autoconf on OS/2 (and we have to stick to very old versions and use various hacks to make it work) while kmk is constantly supported. Further updates of the list of source files in make targets to be in sync with the upstream is nothing compared to that. This is another reason behind the switch.

[abwillis]

Mozilla uses very old autoconf anyhow... Why doesn't Mozilla support autoconf 2.5x?

Simply put, autoconf 2.5x does not offer anything to make the upgrade worth the effort. Autoconf 2.5x is not backwards compatible with autoconf 2.13 and the additional restrictions made by the newer versions of autoconf would require a major rewrite of the Mozilla build system for questionable gain.

Some of the 2.13 features, such as the ability to pass additional arguments to sub-configures, are not available in 2.5x. People have asked repeated about those features on the autoconf mailing list without any favorable response. Rewriting the configures of the sub-projects of Mozilla (NSPR & LDAP) is not an acceptable tradeoff. The sub-projects are also standalone projects and forking an entire codebase because of a build system incompatiblity is silly.

[abwillis]

The above was quoted from: https://developer.mozilla.org/en-US/docs/Developer_Guide/Mozilla_build_FAQ

[dryeo]

On 08/26/13 01:27 am, Dmitriy Kuminov wrote:

Okay, thanks for the info. We will see soon. As I really need a debug build for further work.

How large is the debug xul.dll object? lxlite -c:exemap -vf- xul.dll and look at the line ├ Object Table ├─────────────────────────────────────────────────────────── ├ ## ─ Base ─── Size ──R─W─E─Res─Dis─Shr─Pre─Inv─Swp─Rsd─Loc─A16─32B─Cnf─IOP─ │ 1 00010000 01CD7E40 ¹ ¹ ¹

where 01CD7E40, 30,244,416 bytes is the object size. If it is not much over 32MBs you might just be able to disable the multimedia stuff or perhaps split out Cairo/Pixman to bring it down. -Os might also help though it is nice to avoid any optimizations. I also found the debug-symbol build didn't align the SSE[2] instructions correctly so that has to be worked around. I'll have to get you to post a tarball, preferably compressed with xz somewhere for me to spend a couple of evenings downloading.

[dmik]

Dave, a tarball of the git tree? Okay, sure, just tell me where to upload it.

Andy, correct, but that doesn't help us a lot. Nobody properly maintains it any way and there is also a tool chain which is not regularly maintained.

[dmik]

Built all modules from netwerk and related headers (this is another big piece).

[dmik]

See #9 about the debug XUL build.

[dmik]

Postponing it to GA. Not necessary for the beta but should be done eventually to simplify further maintenance.

[dmik]

This task is really abandoned now — too much work and it has to be aligned each time upstream changes (which is also too much and error prone). We use the original build system now and will continue to do so in the near future.

In order to not get it on the way any more (when searching across files, merging, etc), I removed all kBuild stuff in ea660a967cb61b8423c4096c887f18422f26ac97.