Update to Qt 5.15

[dmik]

This is the latest (and in fact, last) line of the Qt 5 series. Bad news is that we can only update to 5.15.2 (4 months old) because the 5.15.3 update is only available to commercial customers. Looks like a strange decision by The Qt Company but we have what we have (see the original announce of 5.15.3 and also the blog post from Jan 2020 it refers to). Good news is, from what I see, the 5.15.3 branch is still available to everyone so we can cherry pick stuff from there by ourselves.

But so far we will update to 5.15.2 and then we will see.

[dmik]

Also, as comments mention and I see online, they provide access to the 5.15.3 branch for Qt WebEngine, perhaps for GPL reasons or such. We might decide to update Chromium to a more recent version (87.x) instead of 83.x which 5.15.2 has.

[dmik]

With 5d00d6d23483257cb406939244372442004d6e91, the root (this) repo and all 8 submodules we have ported to OS/2 so far have been updated to version 5.15.2 and successfully merged with our OS/2 work.

Now it's build time. I expect several extra merge tasks in Qt Base and Qt WebEngine but nothing serious (e.g. re-amalgamate the new in-tree sqlite version in Chromium). I will be tagging this ticket in respective commits.

[dmik]

With the above commits, qtbase fully builds including tests, now testing time.

[dmik]

In general, Qt Base appears to function well, here's a traditional texteditor example screenshot:

[dmik]

Auto test results so far (by module):

cmake - some failures

63% tests passed, 11 tests failed out of 30

Total Test time (real) = 603.46 sec

The following tests FAILED:
      4 - test_dependent_modules (Failed)
      5 - test_needsquoting_dirname (Failed)
      9 - test_qtmainwin_library (Failed)
     10 - test_dbus_module (Failed)
     13 - test_add_binary_resources_delayed_file (Failed)
     20 - module_includes (Failed)
     22 - test_openglextensions_module (Failed)
     23 - test_opengl_lib (Failed)
     24 - test_interface (Failed)
     28 - test_QTBUG-63422 (Failed)
     29 - test_import_plugins (Failed)
Errors while running CTest

concurrent - OK

Total test units : 6
Tests PASSED : 67
Tests SKIPPED : 0
Tests FAILED : 0
Tests expectedly failed : 0

corelib - one failure

Total test units : 14
Tests PASSED : 1350
Tests SKIPPED : 0
Tests FAILED : 1
Tests expectedly failed : 1

dbus - can't start a single test, fails in init like this

********* Start testing of tst_QDBusConnection_Delayed *********
Config: Using QtTest library 5.15.2, Qt 5.15.2 (i386-little_endian-ilp32 shared (dynamic) debug build; by GCC 9.2.0 20190812 (OS/2 RPM build 9.2.0-5.oc00)), os2 Warp 4.5
PASS   : tst_QDBusConnection_Delayed::initTestCase()
FAIL!  : tst_QDBusConnection_Delayed::delayedMessages() 'session.isConnected()' returned FALSE. ()
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/dbus/qdbusconnection_delayed/tst_qdbusconnection_delayed.cpp(68)]
PASS   : tst_QDBusConnection_Delayed::cleanupTestCase()
Totals: 2 passed, 1 failed, 0 skipped, 0 blacklisted, 53ms
********* Finished testing of tst_QDBusConnection_Delayed *********

gui - one failure

Total test units : 14
Tests PASSED : 412
Tests SKIPPED : 0
Tests FAILED : 1
Tests expectedly failed : 1

network - hangs in one test like this so not complete:

********* Start testing of tst_QHttpNetworkConnection *********
Config: Using QtTest library 5.15.2, Qt 5.15.2 (i386-little_endian-ilp32 shared (dynamic) debug build; by GCC 9.2.0 20190812 (OS/2 RPM build 9.2.0-5.oc00)), os2 Warp 4.5
PASS   : tst_QHttpNetworkConnection::initTestCase()
XFAIL  : tst_QHttpNetworkConnection::options() not tested yet
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/network/access/qhttpnetworkconnection/tst_qhttpnetworkconnection.cpp(121)]
PASS   : tst_QHttpNetworkConnection::options()
FAIL!  : tst_QHttpNetworkConnection::get(success-internal) 'reply->bytesAvailable()' returned FALSE. ()
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/network/access/qhttpnetworkconnection/tst_qhttpnetworkconnection.cpp(209)]
FAIL!  : tst_QHttpNetworkConnection::get(failure-path) 'reply->bytesAvailable()' returned FALSE. ()
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/network/access/qhttpnetworkconnection/tst_qhttpnetworkconnection.cpp(209)]
FAIL!  : tst_QHttpNetworkConnection::get(failure-protocol) 'reply->bytesAvailable()' returned FALSE. ()
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/network/access/qhttpnetworkconnection/tst_qhttpnetworkconnection.cpp(209)]
FAIL!  : tst_QHttpNetworkConnection::head(success-internal) 'reply->isFinished()' returned FALSE. ()
   Loc: [D:/Coding/qt5/qt5/qtbase/tests/auto/network/access/qhttpnetworkconnection/tst_qhttpnetworkconnection.cpp(159)]
make[5]: Leaving directory `D:/Coding/qt5/qt5-dev-build/qtbase/tests/auto/network/access/qhttpnetworkconnection'

other - a bunch of failures

Total test units : 5
Tests PASSED : 119
Tests SKIPPED : 2
Tests FAILED : 40
Tests expectedly failed : 0

printsupport - OK

Total test units : 2
Tests PASSED : 6
Tests SKIPPED : 0
Tests FAILED : 1
Tests expectedly failed : 0

sql - OK

Total test units : 13
Tests PASSED : 251
Tests SKIPPED : 87
Tests FAILED : 0
Tests expectedly failed : 7

testlib - tons of failures

Total test units : 4
Tests PASSED : 40
Tests SKIPPED : 0
Tests FAILED : 1254
Tests expectedly failed : 0

tools, asserts like this, incomplete:

QFATAL : tst_qmakelib::ioUtilResolve(abs-abs) [34755-1:16:40:39.997] unknown:130: ASSERT failure in IoUtils::resolvePath: "/a/unix/dir", file D:/Coding/qt5/qt5/qtbase/qmake/library/ioutils.cpp, line 130
FAIL!  : tst_qmakelib::ioUtilResolve(abs-abs) Received a fatal error.

widgets - some fail

Total test units : 2
Tests PASSED : 27
Tests SKIPPED : 0
Tests FAILED : 2
Tests expectedly failed : 0

xml - OK

Total test units : 4
Tests PASSED : 2015
Tests SKIPPED : 4
Tests FAILED : 0
Tests expectedly failed : 45

Not that much to fix.

[dmik]

With the above fixes, all cmake tests run (with one test to fix within bitwiseworks/cmake-os2#3.

[dmik]

With corelib and gui test suites made succeed w/o a single failure, I guess it's more than enough to move forward. Network and widgets tests will be finished within bitwiseworks/qtbase-os2#74 and bitwiseworks/qtbase-os2#82, respectively. Note that we never had these two test suites succeed (for the previous version 5.13, I mean), and not even gui (before now). The rest of QtBase test suites we also never completed so far. The problem is that running all the tests (there are hundreds thousands of them) is just beyond capabilities of a single developer. And it's not needed that much given that the core part is tested and works.

In the mean time I successfully built the qtdeclarative module which is a must for qtwebengine. It's testing was never finished too and will be done within bitwiseworks/qtdeclarative-os2#3 later. Other modules (qtsvg, qtwebchannel, qtwebsockets, etc.) are rather small and should not cause any trouble.

[dmik]

With the above changes, Chromium builds successfully now in debug mode. However, in release mode I get a strange crash from one helper process:

C:/USR/BIN/python.exe ../../../../../qt5/qtwebengine/src/3rdparty/chromium/tools/protoc_wrapper/protoc_wrapper.py protos/perfetto/common/android_log_constants.proto protos/perfetto/common/commit_data_request.proto protos/perfetto/common/data_source_descriptor.proto protos/perfetto/common/descriptor.proto protos/perfetto/common/gpu_counter_descriptor.proto protos/perfetto/common/observable_events.proto protos/perfetto/common/sys_stats_counters.proto protos/perfetto/common/trace_stats.proto protos/perfetto/common/tracing_service_state.proto protos/perfetto/common/track_event_descriptor.proto --protoc ./protoc.exe --proto-in-dir ../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/perfetto/ --plugin protozero_plugin.exe --plugin-out-dir gen/third_party/perfetto/ --plugin-options wrapper_namespace=pbzero
[libprotobuf FATAL ../../../../../qt5/qtwebengine/src/3rdparty/chromium/third_party/protobuf/src/google/protobuf/compiler/subprocess.cc:322] fork: Function not implemented

Killed by SIGABRT
pid=0x00a0 ppid=0x009f tid=0x0001 slot=0x008f pri=0x0200 mc=0x0001 ps=0x0010
D:\CODING\QT5\QT5-DEV-BUILD\QTWEBENGINE\SRC\CORE\RELEASE\PROTOC.EXE
Creating 00A0_01.TRP
Protoc has returned non-zero status: -6

It's really weird that the debug version of the same exe works like a charm. I guess I will have to change fork to spawn2.

[dmik]

From what I see in the LIBC sources, ENOSYS is returned either when it fails to get the module list's head or when -Zfork is missing. But -Zfork is on by default ATM so I really wonder what could go wrong there.

Using the logging LIBC reveals that it thinks that the exe wasn't built with -Zfork. Strange. Need to dig into that.

[dmik]

With the above commit linking protoc.exe and other tools in release mode adds fork support back and all works.

Now I get a bunch of unresolved externals when linking the final DLL and also a message that worries me most:

Error! E2021: size of segment text exceeds 64k by 151119 bytes

I don't really like the smell of it... But I will try to fix the missing externals first.

[dmik]

With the above commits all unresolved externals are gone but I'm still seeing that E2021 about the size of segment text from wlink. I wonder how to debug to find a guilty object file (though I have a couple of guesses).

[StevenLevine]

Something generated a large 16-bit segment in an object file. The error cannot occur for 32-bit objects:

bld\wl\c\salloc.c:71 static offset BumpUp( offset ptr, offset size ) /*****/ { ptr += size; if( CurrentSeg != NULL && !(CurrentSeg->info & USE_32) && !(FmtData.type & MK_RAW) && ptr > 0x10000 ) { LnkMsg( ERR+MSG_SEG_TOO_BIG, "sl", CurrentSeg->segname, (unsigned long)(ptr-0x10000) ); } return( ptr ); }

The message is defined as:

bld\wl\h\lnkerror.msg:39 pick( MSG_SEG_TOO_BIG, "size of segment %s exceeds 64k by %l bytes" , "%s âZâOâüâôâgé╠âTâCâYé¬ %lâoâCâgò¬éUéSéjé­Æ┤éªé─éóé▄éÀ" )

which means the segment name is "text" which is somewhat odd, but it does tell you want to look for in the object files. Perhaps the object file was generated by nasm or the equivalent. I've never seen gcc or emxomf generate this segment name. A typical set of gcc generated segment names is:

dmpobj -q -rec=lnames testme.o LNAMES(96) recnum:4, offset:00000024h, len:0050h, chksum:73h(73) 1 - '' 2 - 'TEXT32' 3 - 'DATA32' 4 - 'BSS32' 5 - '$$SYMBOLS' 6 - '$$TYPES' 7 - 'CODE' 8 - 'DATA' 9 - 'BSS' 10 - 'DEBSYM' 11 - 'DEBTYP' 12 - 'FLAT' 13 - 'DGROUP'

Some combination of the above dmpobj command and grep should help you track down the errant object file.

[dmik]

@StevenLevine thanks for the dmpobj hint! Useful. And for the 16-bit segment reference as well — you are right. My memory didn't betray me either, I already faced the very same issue back then indeed: bitwiseworks/qtwebengine-chromium-os2#15. Easy to fix.

[dmik]

With the above commits, everything seems to build and link successfully so I'm closing this large ticket. The rest will be done via separate ones.