search

bitwiseworks / qtwebengine-chromium-os2

Port of Chromium and related tools to OS/2
9 stars 2 forks source link

Make Chromium build #3

Closed dmik closed 4 years ago

dmik commented 4 years ago

This is a continuation of https://github.com/bitwiseworks/qtwebengine-os2/issues/1 dedicated to building the original Chromium source code.

Note that the build process itself is intended to be started using qmake makefiles from within the qtwebengine repository. But qmake does no more than simply runs gn to generate ninja build files and then runs ninja. I.e. prepares and starts a native Chromium build process. While this process can be then started manually from $(BLDROOT)/qtwebengine/src/core/debug (or release) by calling ninja like this:

cd "$(BLDROOT)"/qtwebengine/src/core && "$(BLDROOT)"/qtwebengine/src/3rdparty/ninja/ninja.exe -v  -C "$(BLDROOT)"/qtwebengine/src/core/$(BUILD_TYPE) QtWebEngineCore

it's still recommended to use qmake makefiles to avoid any de-synchronization with Qt sources.

dmik commented 4 years ago

Then next rather big step is to port the message_pump machinery. I actually already did that when porting Mozilla (check https://github.com/bitwiseworks/mozilla-os2/commits/master/ipc/chromium/src/base and around) but Mozilla uses a very old version of Chromium and a lot has changed since then so some aligning is needed.

dmik commented 4 years ago

The message_pump machinery is mostly done (a commit will follow). Then ext task is thread_local_storage. In old Chromium in Mozilla we used pthread but now I have to do it natively as pthread destructor handling is not perfect (and will not work in some cases). And Chromium performs its own processing anyway.

dmik commented 4 years ago

I'm getting build issues in utf_string_conversions.cc because std::is_integral<wchar_t>::value is false with the new GCC 9.x although it must be true of course. This might require more some more GCC work within https://github.com/bitwiseworks/gcc-os2/issues/16, sigh...

dmik commented 4 years ago

With LIBC and GCC fixed, the string functions build fine now. Next stops besides pthread TLS are these (all expected):

dmik commented 4 years ago

LaunchProcess requires some tricks to be done. In POSIX, they sometimes want the child to inherit a specific FD of the parent. Sometimes this FD is required to be inherited under a specific file number in the child (i.e. not the original one). All this is achievable (with dup2) but not out of the box. The main problem is thread safety. I fear Chromium starts processes from many threads. So doing it w/o locking might mix things up. Also, inheritance is a big question. On OS/2 all file handles are inherited by default and this is surely not what Chromium wants. In spawn2 inheritance may be completely suppressed with P_2_NOINHERIT flag, but this makes it impossible to inherit specific descriptors. I think that the simplest way to go here is to extend LIBCx spawn2 to allow for selective inheritance of specified file handles.

dmik commented 4 years ago

With the new LIBCx spawn2 update, doing LaunchProcess was quite easy (and much easier than Unix or even Windows code!).

I've also stumbled upon https://github.com/psmedley/gcc/issues/34 when building histogram code:

D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/metrics/histogram.cc:350:34: error: 'round' is not a member of 'std'; did you mean 'round'?
  350 |     next = static_cast<int>(std::round(exp(log_next)));
      |                                  ^~~~~
In file included from C:/usr/include/c++/9/math.h:30,
                 from C:/usr/include/c++/9/cmath:45,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/numerics/checked_math_impl.h:12,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/numerics/checked_math.h:13,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/numerics/safe_math.h:8,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/time/time.h:63,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/metrics/histogram_base.h:20,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/metrics/bucket_ranges.h:30,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/metrics/histogram.h:81,
                 from D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/base/metrics/histogram.cc:10,
                 from gen/base/base_jumbo_12.cc:5:
C:/usr/include/math.h:289:8: note: 'round' declared here
  289 | double round(double);
      |        ^~~~~

It can be worked around the way I did it in QtDeclarative but I guess it's time to fix GCC instead as it should be easy and will free us from working around these bits in other software.

SilvanScherrer commented 4 years ago

https://github.com/bitwiseworks/gcc-os2/issues/15 weil known id say

dmik commented 4 years ago

With https://github.com/bitwiseworks/gcc-os2/issues/15 closed, histogram code builds fine. I will rebuild QtDeclarative too.

dmik commented 4 years ago

Just as a side note. It's kind of hard to work on things because there is basically a single Chromium "makefile" which builds everything in parallel and in random order. There is no way to tell "build me only threading sources" etc. So I have to grab individual gcc calls and run them manually if I want to work specifically on them. This is very inconvenient. Takes too much time to sort these things out.

There is some level of control via build.ninja's build statements but it also requires some manual lookup and execution. Not very helpful.

dmik commented 4 years ago

Moved a bit further. With latest pthread fixes TLS code also builds fine out of the box (and should be thread-safe). I'm seeing lots of small build breaks here and there. Will commit them next once done.

dmik commented 4 years ago

With the above commits, the base Chromium module completely builds (producing a static library libbase.a). The next big stop is the gfx module (obviously) where I will need to add PM integration. Perhaps, I will create a separate ticket for that.

Before going with gfx, I will release LIBCn, LIBCx and GCC as there have been a lot of new functionality that is useful for other apps as well. And I don't expect that further Chromium code will require a lot more changes to LIBCn etc.

dmik commented 4 years ago

From now on, I will be creating separate tickets for other modules/parts of Chromium for better readability. This will be a generic meta ticket open until we build everything.

dmik commented 4 years ago

Yasm is still getting used here and there (e.g. for blink/renderer/platform/heap/asm). I need to get nasm used instead. I wonder how much they are compatible option-wise.

dryeo commented 4 years ago

On 04/13/20 06:23 AM, Dmitriy Kuminov wrote:

Yasm is still getting used here and there (e.g. for |blink/renderer/platform/heap/asm|). I need to get nasm used instead. I wonder how much they are compatible option-wise.

They're pretty compatible. NASM is a bit more picky when it comes to the include path, -I and IIRC, might need -P to pre-include.

dmik commented 4 years ago

Yasm is worked around — I replace it with nasm so that no individual build file patching is needed (but alignment of .asm files might be needed due to tiny yasm/nasm differences). Will commit after some testing.

The next build break is gpu code. I hope there is some sort of placeholders for platforms that don't support it.

dmik commented 4 years ago

BTW, regarding nasm, we have 2.13 and it is indeed picky about -I options: it expects the include path to end with a slash and doesn't append one if it's missing. This would completely break %include processing if paths in this directive did not start with a slash. I've just hit that. In 2.14 they fixed this (see -I docs at https://nasm.us/doc/nasmdoc2.html) but since our RPM is still at 2.13, this fix is not there. I will fix chromium build files for now but we should surely update our nasm port as well.

dmik commented 4 years ago

Another show stopper: FFMPEG. Chromium needs avcodec_send_packet and avcodec_receive_frame We have 2.8.6 and both are missing from there. From what I see in their git, version 3.1 already has it (version 3.0 does not). So we need at least version 3.1. The current one is 4.2.2. We definitely need a fresh port.

dryeo commented 4 years ago

On 04/27/20 10:21 AM, Dmitriy Kuminov wrote:

Another show stopper: FFMPEG. Chromium needs |avcodec_send_packet| and |avcodec_receive_frame| We have 2.8.6 and both are missing from there. From what I see in their git https://git.ffmpeg.org/gitweb/ffmpeg.git/tags, version 3.1 already has it (version 3.0 does not). So we need at least version 3.1. The current one is 4.1.5. We definitely need a fresh port.

Should build fine with needing some includes adjustment for getaddressinfo() and friends in libavformat/os_support.h,something like +#if defined (OS2) && defined (HAVE_GETADDRINFO) +#include <libcx/net.h> +#endif + Dave ps don't forget to leave the older ffmpeg libs in the tree, and ffplay needs sdl2 now.

dmik commented 4 years ago

@dryeo ok it builds out of the box with the above LIBCx addition indeed. As far as I see, all our OS/2 configure patches from https://github.com/bitwiseworks/FFmpeg are upstream now. Was it you who pushed them? Thanks anyway.

BTW I'm not sure I got your "leave the older ffmpeg libs in the tree", what do you mean? Old DLL versions when installing a newer RPM? Well, it's a question on how to do it. If backward compatibility is supported, we may provide forwarders but I'm not sure. Needs thinking. What exactly do you need this for? Mozilla?

Re ffplay, seems that we will have to not provide it for the time being because we don't have SDL2 ATM.

dryeo commented 4 years ago

On 04/28/20 05:09 PM, Dmitriy Kuminov wrote:

@dryeo https://github.com/dryeo ok it builds out of the box with the above LIBCx addition indeed. As far as I see, all our OS/2 configure patches from https://github.com/bitwiseworks/FFmpeg are upstream now. Was it you who pushed them? Thanks anyway.

FFmpeg has supported OS/2 for the longest time due to patches KOMH and I pushed years back. I'd push the patch I posted but unsure how long before the headers are fixed and KOMH is actually the official OS/2 maintainer and I'm not sure he uses libcx, otherwise I'd add it to our configure section as a network lib. BTW, you might want to disable AVX as the OS4 kernel supports it but our GCC doesn't align it correctly. People were getting crashes in Firefox using the RPM FFmpeg libs.

BTW I'm not sure I got your "leave the older ffmpeg libs in the tree", what do you mean? Old DLL versions when installing a newer RPM? Well, it's a question on how to do it. If backward compatibility is supported, we may provide forwarders but I'm not sure. Needs thinking. What exactly do you need this for? Mozilla?

Yes Mozilla needs the older DLLs. As the libs are a separate package and they are versioned by name so no conflict with newer ones, perhaps the package name can just be changed to moz_ffmpeg_libs or such?

Re ffplay, seems that we will have to not provide it for the time being because we don't have SDL2 ATM.

There's a port on Hobbes but it might be an OpenWatcom build so needs import libs, sdl2.config and pkgconfg files to work, it is built with the right calling convention

dmik commented 4 years ago

@dryeo thanks for the clarification — I remembered now that KOMH is the maintainer (we'd even crossed over its mailing list on that subject) but your comments confused me. Re AVX, IIRC, it's not a GCC problem, it's a linker problem, see https://github.com/bitwiseworks/gcc-os2/issues/11. Re old DLLs, we will most likely provide a usual -legacy RPM package with them.

dmik commented 4 years ago

Got something really sad. It had to happen one day. No idea on how to fix it for now other than rename some files (which is bad-bad-bad). May be some relative path magic will help.

g++ -MMD -MF obj/gpu/command_buffer/service/gles2_sources/gles2_sources_jumbo_1.o.d -DUSE_AURA=1 -DNO_TCMALLOC -DCHROMIUM_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DGPU_GLES2_IMPLEMENTATION -DRASTER_IMPLEMENTATION -DWEBGPU_IMPLEMENTATION -DWEBP_EXTERN=extern -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DUSE_SYSTEM_ZLIB=1 -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/mesa_headers -I. -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/khronos -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/gpu -Igen -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/libwebp/src -Igen -Igen -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/skia/config -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/skia/ext -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/c -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/config -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/core -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/docs -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/effects -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/encode -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/gpu -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/pathops -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/ports -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/utils -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/codec -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/src/gpu -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/src/sksl -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/modules/skottie/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/angle/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/angle/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/angle/src/common/third_party/base -Igen/angle -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/angle/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/re2/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/ced/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/mesa_headers -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -Zomf -m32 -msse2 -mfpmath=sse -mmmx -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-dangling-else -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -O0 -fno-omit-frame-pointer -g2 -std=c++14 -Wno-narrowing -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -fno-exceptions -fno-rtti -c gen/gpu/command_buffer/service/gles2_sources_jumbo_1.cc -o obj/gpu/command_buffer/service/gles2_sources/gles2_sources_jumbo_1.o
gen/gpu/command_buffer/service/gles2_sources_jumbo_1.cc:15:10: fatal error: ../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/mesa_headers/../../../../../qt5/qtwebengine/src/3rdparty/chromium/gpu/command_buffer/service/gles2_cmd_apply_framebuffer_attachment_cmaa_intel.cc: File name too long
   15 | #include "../../../../../qt5/qtwebengine/src/3rdparty/chromium/gpu/command_buffer/service/gles2_cmd_apply_framebuffer_attachment_cmaa_intel.cc"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
StevenLevine commented 4 years ago

I would vote for normalizing the paths when needed before passing them to the file system. Anything of the form

/component/..

can be mapped to

/

after checking that the component exists.

Dealing with leading ../'s is similar once the current directory is prepended to the relative path.

Orginally I was thinking that the normalization would be done in the compiler, but it might be better to do this normalization in kLIBC. To my way of thinking it is a variation on the path rewriting that already exists. The normalization only needs to be done if the length of the current directory and the relative path exceeds the limit. Otherwise, the file system will handle the normalization for us.

ydario commented 4 years ago

I assume gles2_cmd_apply_framebuffer_attachment_cmaa_intel.cc is a real file on the disk, so its absolute path does not exceed 260 chars for sure; given this, the relative path can be handled: I had similar issues in the past and they was due to some bad calculation about max path size. In this case there are a lot of ../ rel path changes that can break bad code. Probably resolving the path to its absolute path can fix this.

dmik commented 4 years ago

The question is where to normalize. The failure (most likely) happens when the C preprocessor attempts to normalize the following include path:

../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/mesa_headers/../../../../../qt5/qtwebengine/src/3rdparty/chromium/gpu/command_buffer/service/gles2_cmd_apply_framebuffer_attachment_cmaa_intel.cc

The first part of it comes from some -I command line option and the second part is the include statement itself.

My guess is that it uses LIBC for that (something like abspath or realpath). If it's the case and the failure comes from LIBC (which is my current guess) we are lucky more or less as it's a question of fixing LIBC then. Not a very easy task but doable. I will dig more.

Resolving the path to the absolute in the source is also an option but it's tricky as too much has to be reviewed in the chromium build system (and jumbo file generation). And will not be a universal solution since any case like the above in any other project will just fail too.

dmik commented 4 years ago

As the above link indicates, it's a LIBC problem which I reproduced in a simple isolated test case.

BTW this is one of those unexpected show stoppers we didn't even mention yesterday, @StevenLevine.

dmik commented 4 years ago

Another weird failure:

g++ -MMD -MF obj/net/net/http_stream_factory_job.o.d -DUSE_AURA=1 -DNO_TCMALLOC -DCHROMIUM_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DOPENSSL_NO_ASM -DDLOPEN_KERBEROS -DNET_IMPLEMENTATION -DENABLE_BUILT_IN_DNS -DUSE_SYSTEM_ZLIB=1 -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium -Igen -Igen -Igen -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/boringssl/src/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/ced/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/brotli/include -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -Zomf -m32 -msse2 -mfpmath=sse -mmmx -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-dangling-else -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -O0 -fno-omit-frame-pointer -g2 -std=c++14 -Wno-narrowing -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -fno-exceptions -fno-rtti -c ../../../../../qt5/qtwebengine/src/3rdparty/chromium/net/http/http_stream_factory_job.cc -o obj/net/net/http_stream_factory_job.o
emxomf: Record too long
D:\Temp/ccmqxTwe.s: Assembler messages:
D:\Temp/ccmqxTwe.s: Fatal error: emxomf failed

Some digging shows that if I remove -Zomf then GCC is able to successfully create a 10MB a.out file. Converting this file to OMF with emxomf fails with the same error (Record too long). I bet this must be some C++ debug (HLL) stuff that doesn't fit some ancient OMF record buffer again... (I recall I saw that already).

dmik commented 4 years ago

No, it's definitely not the debug info. I've tried replacing -g2 with -s, got an a.out file of just 400KB but the conversion to OMF still fails with the same error. It's going to be more difficult to fix.

dmik commented 4 years ago

Compiled a thousand more compilation units with emxomf fixed. Now getting some python issues:

C:/USR/BIN/python.exe ../../../../../qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit.py -i ../../../../../qt5/qtwebengine/src/3rdparty/chromium/ui/webui/resources/webui_resources.grd build -o gen/ui/resources --depdir . --depfile gen/ui/resources/webui_resources_grd_stamp.d --write-only-new=1 --depend-on-stamp -D _chromium -E CHROMIUM_BUILD=chromium -D use_aura -D optimize_webui=false -E root_gen_dir=gen -f ../../../../../qt5/qtwebengine/src/3rdparty/chromium/tools/gritsettings/resource_ids --assert-file-list=obj/ui/resources/webui_resources_grd_expected_outputs.txt
parse exception: run GRIT with the -x flag to debug .grd problems
Traceback (most recent call last):
  File "../../../../../qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit.py", line 15, in <module>
    sys.exit(grit.grit_runner.Main(sys.argv[1:]))
  File "D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit/grit_runner.py", line 256, in Main
    return toolobject.Run(options, args[1:])
  File "D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit/tool/build.py", line 245, in Run
    target_platform=target_platform)
  File "D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit/grd_reader.py", line 198, in Parse
    xml.sax.parse(filename_or_stream, handler)
  File "C:/USR/lib/python2.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "C:/USR/lib/python2.7/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "C:/USR/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "C:/USR/lib/python2.7/xml/sax/expatreader.py", line 210, in feed
    self._parser.Parse(data, isFinal)
  File "C:/USR/lib/python2.7/xml/sax/expatreader.py", line 307, in end_element
    self._cont_handler.endElement(name)
  File "D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/tools/grit/grit/grd_reader.py", line 96, in endElement
    xml.sax.parse(partname, GrdPartContentHandler(self))
  File "C:/USR/lib/python2.7/xml/sax/__init__.py", line 33, in parse
    parser.parse(source)
  File "C:/USR/lib/python2.7/xml/sax/expatreader.py", line 102, in parse
    source = saxutils.prepare_input_source(source)
  File "C:/USR/lib/python2.7/xml/sax/saxutils.py", line 328, in prepare_input_source
    basehead = basehead.decode(encoding)
TypeError: decode() argument 1 must be string, not None

This is kind of strange as the error comes from the Python's built-in SAX parser. The parser queries the encoding for file names with sys.getfilesystemencoding() and, according to the docs, this method is allowed to return None to indicate that the system default encoding is used for file names (which is true on OS/2). However, string.decode() which is then called with the returned encoding does not accept None as the encoding value (but does accept omitting the encoding argument completely to indicate just the same — the default encoding). But SAX doesn't expect that. Strange. To my understanding, both parties are wrong here.

dmik commented 4 years ago

With the latest fixed python and above patches, I got much further. It's just about 7600 compilation units to go against the initial amount of ~15600. And #7 is almost done.

dmik commented 4 years ago

With the above fixes and some local modifications I now went from 7600 down to 3800 compilation units left to go. Not bad for a week of work. Local modifications include porting pdfium and v8 components to OS/2 and are not yet finished so I will commit them next week once done.

dmik commented 4 years ago

I've ported v8 and pdfium to OS/2, all compiles but as part of the build v8 BUILD.gn runs some mksnapshot.exe it built earlier that generates embedded assembly or such and that fails like that:

FAILED: gen/v8/embedded.S gen/v8/snapshot.cc 
C:/USR/BIN/python.exe ../../../../../qt5/qtwebengine/src/3rdparty/chromium/v8/tools/run.py ./mksnapshot --turbo_instruction_scheduling --embedded_src gen/v8/embedded.S --embedded_variant Default --random-seed 314159265 --startup_src gen/v8/snapshot.cc

#
# Fatal error in D:/Coding/qt5/qt5/qtwebengine/src/3rdparty/chromium/v8/src/allocation.cc, line 254
# Debug check failed: result.
#
#
#
#FailureMessage Object: 0x208a734

Some DosSetMem fails or such, I need to debug.

dmik commented 4 years ago

I've fixed the above error (it took a lot of time because each change in the source file would cause around 200 files to be recompiled that took 15 to 20 min). Now there is no memory errors and the assembly file is generated fine (12 megabytes!). I had to fix a couple of things there (.section GAS attribute is not supported by a.out), however GAS still fails to generate an object file out of it. It fails with the following:

as: obj/v8/v8_snapshot/embedded.o: can not represent section `.debug_line' in a.out object file format
D:\Temp/ccDhyqmx.s: Assembler messages:
D:\Temp/ccDhyqmx.s: Fatal error: can't write 1882 bytes to section .debug_line of obj/v8/v8_snapshot/embedded.o: 'nonrepresentable section on output'

I can't say what it is w/o looking into GAS sources...

The generated assembly itself does not contain such a section so it must be created implicitly somehow.

dmik commented 4 years ago

I've sorted out all GAS problems and flushed all the current patches above. Now both V8 (Google JS engine) and PDFium (PDF viewer based on JS) build fine. It's about 2000 source units left to compile. Hard to tell more exactly as I'm having a full rebuild now due to numerous changes here and there (and especially the C++ standard change which causes rebuilding of all C++ code).

dmik commented 4 years ago

I’m having some really strange problems now with my hard disk. Running gn.exe leads to either a hard lock or a trap with automatic reboot and disk checking (with some FS problems found). Happened three times in a row already.

dmik commented 4 years ago

Still hanging at the very beginning of ninja invocation. It was doing so while trying to build a particular file but building this file b hand worked fine and now it hung again. On a different file. No idea.

I’ve run chkdsk /f:3 and that found one bad block (see the screenshot attached). Can it be that my SSD is dying? Who knows. I doubt there are tools for OS/2 to check its health. It will be really bad if it’s the SSD. It’s relatively young. Less than two years...

1A1B9391-B0BD-4130-AA63-50671091F48D

dmik commented 4 years ago

I’m also getting CHKDSK. logredo failed (rc=-272). CHKDSK continuing. While booting up after such a hang (it’s a hard hang, only cold reboot cures it so this smells really like a low level disk issue to me...).

dryeo commented 4 years ago

You can use smartahci which may tell you something about its health. I also try to trim my SSD regularly by booting to Mint, mounting all the partitions and running "sudo fstrim -a -v" IIRC.

StevenLevine commented 4 years ago

The chkdsk errors are expected after a random trap and hang. The redo log can and will get corrupted when a busy system hangs or traps.

The trap screens might provide some clues as to what is failing.

I would not automatically attribute your failure to a disk issue. It could be a lot of things since this system as a whole is probably working hard.

dmik commented 4 years ago

Further observation shows that its the same file though, file_system_url_loader_factory.cc. Even if I build it by hand, ninja still wants to rebuild it all the time. It actually starts gcc for that and I see the file gets zeroed again but then this gcc process never finishes. Why so, I'm not sure. Something on the other core is preventing it from doing that.

So, might be not a hard disk issue indeed.

dmik commented 4 years ago

Hmm, it's really strange but seems that I was right about the cores. Leaving only one CPU with /MAXCPU:1 cured it. Still, no idea what it was. It might be some process watching code that screws LIBC/whatever waiting for multiple children. Or it might be several EMXOMF invocations figthting for a temp file or such. Currently I don't have any idea on what could be the cause. It will be really hard to debug that I suppose. I will try to skip that tor now and go further until I hit that again (I hope I won't).

dmik commented 4 years ago

Now it's trying to compile two next compilation units but it's taking forever for some reason (why two I don't know, IIRC ninja figures out the number of CPUs via sysconf (which we support now) and should only use one thread). I see two AS.EXE instances and two EMXOMF.EXE instances. One AS->EMXOMF pair per each compilation unit. The TEMP dir proves that:

cc8ZoA5lasXXXXXX                                                       ³31ÿ518ÿ231³02.06.20³ 19:00º
ccbezu6P.s                                                             ³36ÿ771ÿ545³02.06.20³ 18:53º
cccxjjCUasXXXXXX                                                       ³35ÿ304ÿ448³02.06.20³ 18:53º
ccySKDde.s                                                             ³33ÿ064ÿ932³02.06.20³ 19:00º

It looks like EMXOMF somehow interfere and don't let each other finish (100% CPU load). In SMP mode they crash the FS driver it seems.

dmik commented 4 years ago

Well, no, it works but VERY slowly with one core. 6 compilation units in 1.5 hours. Hmmmm.

Switching to 4 cores immediately brings the problem back. Sigh.

dmik commented 4 years ago

Ok, limiting ninja threads to just 2 with -j 2 makes it work even on SMP but I really wonder whats that. Perhaps out of memory when too many EMXOMF instances are trying to convert too fat AOUT files (as you see above both assembly files and the resulting AOUT files are more than 30 MB in size!). Or some out of resources condition in the kernel. It's really strange why I haven't seen this before. The resulting OMF files are not the largest ones out there. But may be they have too much debug info as I'm getting this:

emxomf warning: Input file 'D:\Temp/ccngSn4aasXXXXXX' has more HLL debug types than we can index in PUBDEF and EXTDEF records.

for all files that make it hang. I wonder which tools can be used to debug this sort of things. (and not a COM port or anything like that).

PS. By default, ninja uses 6 threads in 4 core mode. Perhaps it adds 2 to the number of cores (as with one core it used at least 2 or maybe 3). But lowering it down to 4 threads with -j 4 makes the kernel also hang.

dmik commented 4 years ago

BTW, judging by the .s (asm) file contents, there is an ENORMOUS amount of debug info (.stabs/.stabd directives). For a 36.3 MB file, there is 35.7 MB of stabs (and only 0.6 MB of real assembly). So it's not a surprise that EMXOMF gets crazy trying to build OMF HLL out of that. It really smells like a pretty hard OOM condition somewhere.

StevenLevine commented 4 years ago

FWIW, if you think you have a OOM condition, Theseus can help you.

For file contention, the OS/2 trace facility can often help you.

Steven

--

"Steven Levine" steve53@earthlink.net Warp/DIY/BlueLion etc. www.scoug.com www.arcanoae.com www.warpcave.com

dmik commented 4 years ago

Moved to a separate issue. Theseus is great but not very useful under heavy load.

StevenLevine commented 4 years ago

When using tools like Theseus on a heavily loaded systems, it is recommended that they be run at time-critical priority.

dmik commented 4 years ago

I wonder then how to run Theseus at time-critical priority... Some nice-like command line tool?

I worked around the OOM condition in EMXOMF but this didn't solve all the problems. Now I'm hitting OOM in CC1PLUS.EXE while compiling (i.e. generating that huge assembly):

g++ -MMD -MF obj/content/browser/browser/render_frame_devtools_agent_host.o.d -DUSE_VIZ_DEVTOOLS -DUSE_AURA=1 -DNO_TCMALLOC -DCHROMIUM_BUILD -DTOOLKIT_QT -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DCONTENT_IMPLEMENTATION -DWEBP_EXTERN=extern -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DHAVE_PTHREAD -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSK_SUPPORT_GPU=1 -DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\" -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_PLATFORM_CHROMIUM=1 -DOPENSSL_NO_ASM -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_POSIX -DWEBRTC_OS2 -DWEBRTC_NO_INET6 -DABSL_ALLOCATOR_NOTHROW=1 -DNO_MAIN_THREAD_WRAPPING -DV8_ENABLE_CHECKS -DV8_DEPRECATION_WARNINGS -DUSING_SYSTEM_ICU=1 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_STATIC -DSQLITE_ENABLE_FTS3 -DSQLITE_DISABLE_FTS3_UNICODE -DSQLITE_DISABLE_FTS4_DEFERRED -DSQLITE_ENABLE_ICU -DSQLITE_SECURE_DELETE -DSQLITE_THREADSAFE=1 -DSQLITE_MAX_WORKER_THREADS=0 -DSQLITE_MAX_MMAP_SIZE=268435456 -DSQLITE_DEFAULT_FILE_PERMISSIONS=0600 -DSQLITE_DEFAULT_MEMSTATUS=1 -DSQLITE_DEFAULT_PAGE_SIZE=4096 -DSQLITE_DEFAULT_PCACHE_INITSZ=0 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS -DSQLITE_OMIT_DEPRECATED -DSQLITE_OMIT_PROGRESS_CALLBACK -DSQLITE_OMIT_SHARED_CACHE -DSQLITE_USE_ALLOCA -DSQLITE_OMIT_ANALYZE -DSQLITE_OMIT_AUTOINIT -DSQLITE_OMIT_AUTORESET -DSQLITE_OMIT_COMPILEOPTION_DIAGS -DSQLITE_OMIT_COMPLETE -DSQLITE_OMIT_DECLTYPE -DSQLITE_OMIT_EXPLAIN -DSQLITE_OMIT_GET_TABLE -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_DEFAULT_LOOKASIDE=0,0 -DSQLITE_OMIT_LOOKASIDE -DSQLITE_OMIT_TCL_VARIABLE -DSQLITE_OMIT_REINDEX -DSQLITE_OMIT_TRACE -DSQLITE_OMIT_UPSERT -DSQLITE_OMIT_WINDOWFUNC -DSQLITE_HAVE_ISNAN -DSQLITE_ENABLE_LOCKING_STYLE=0 -DSQLITE_ENABLE_API_ARMOR -DUSE_SYSTEM_ZLIB=1 -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium -Igen -Igen -Igen -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/libwebp/src -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/khronos -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/gpu -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/libyuv/include -Igen -Igen -Igen -Igen -Igen -Igen -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/ced/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/skia/config -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/skia/ext -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/c -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/config -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/core -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/docs -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/effects -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/encode -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/gpu -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/pathops -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/ports -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/utils -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/include/codec -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/src/gpu -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/src/sksl -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/skia/modules/skottie/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/leveldatabase -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/leveldatabase/src -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/leveldatabase/src/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src -Igen/protoc_out -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/boringssl/src/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/webrtc_overrides -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/webrtc -Igen/third_party/webrtc -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/abseil-cpp -Igen/third_party/metrics_proto -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/libwebm/source -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/mesa_headers -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/v8/include -Igen/v8/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/angle/src/common/third_party/base -Igen/angle -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/brotli/include -I../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/re2/src -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -funwind-tables -Zomf -m32 -msse2 -mfpmath=sse -mmmx -Wall -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-deprecated-declarations -Wno-comments -Wno-dangling-else -Wno-packed-not-aligned -Wno-missing-field-initializers -Wno-unused-parameter -O0 -fno-omit-frame-pointer -g2 -std=gnu++14 -Wno-narrowing -Wno-attributes -Wno-class-memaccess -Wno-subobject-linkage -Wno-invalid-offsetof -fno-exceptions -fno-rtti -c ../../../../../qt5/qtwebengine/src/3rdparty/chromium/content/browser/devtools/render_frame_devtools_agent_host.cc -o obj/content/browser/browser/render_frame_devtools_agent_host.o

cc1plus.exe: out of memory allocating 65536 bytes after a total of 65536 bytes

I will look at that in Theseus. But I already noticed earlier that CC1PLUS.EXE was taking more than 1G of "shared" memory on heavy source files. Quite expectable I would say.

BTW, as source file locations show, all the OOM problems started to happen when I came to building the browser component of Chromium. I guess it is its major component so no surprise here. Its sources drag LOTS of stuff from all other support libraries.

dmik commented 4 years ago

The above was solved by changing VAL from 1536 to 3072. Going further.