build: use Ubuntu Focal for depends, ci and gitian releases

patricklodder commented 1 year ago

Ubuntu Bionic will go end-of-life in April 2023, so everything that relies on it needs to be updated. This encompasses the:

continuous integration processes
gitian release system
supported OS for the depends system

This PR combines and enhances prior work done by @xanimo in #3133 and #3185 into an end-to-end upgrade to Ubuntu Focal for all these items, at once, because these turn out to all be interdependent on each other after extensive testing. Most of the prerequisites that came from the earlier PRs have already been merged separately, with the only remaining open requirement being #3204: that PR blocks this one as we need a portable set of check tooling to be able to compare binaries across host OS's. The commits from that PR are included in this one - this will need a rebase once that is merged. Commits that belong to this PR are 58553dba1525b71e9fa363af9afe20ca514e2de4 and onward.

what this does

wrap functions with an explicit glibc version selector for linux builds to ensure that the binary remains usable on systems with old glibc (this is enabled with the --enable-glibc-back-compat configure flag):
- clock_gettime, exp, log and pow for all target Linux architectures
- fcntl and fcntl64 for 32-bit Linux architectures (i686-pc-linux-gnu and arm-linux-gnuabihf targets)
update the clang and cctools packages in depends to respectively versions 6.0.1 and 921 while retaining minimum SDK compatibility. This is needed to be able to compile with our current macOS SDK on focal.
Upgrade the CI to use Ubuntu Focal and gcc 9 for all targets
Upgrade gitian scripts to use Ubuntu Focal and gcc 9 for all descriptors
Update the documentation of requirements for the depends system

what needs to be done now

[x] Wait until #3204 is merged or cross-platform binary checking is attained in any other way

---- once that is done ----

[x] Review, and apologies for the big change (will get a lot smaller after the previous item is checked)
[x] A gitian comparison to ensure that the release build is still deterministic
[x] Tests that binaries from the gitian builds are capable of sync on as old platforms as possible. Think: 32-bit armhf RPi, 32-bit intel with WinXP, Ubuntu Trusty, Centos 7, MacOS Mavericks/Yosemite... and so on.

acknowledgements

This work is the final outcome of months of work and reworking the approach between @xanimo and myself. I couldn't have done this alone. Thank you, @xanimo ❤️

patricklodder commented 1 year ago

force pushed to fix a dirty commit, replaced by ed87230

patricklodder commented 1 year ago

Rebased after merge of #3204

chromatic commented 1 year ago

Concept and code ACK after rebase. Let's move to the gitian step.

patricklodder commented 1 year ago

I've found 2 things that need rework on this:

macOs `-mlinker-version`

I think we need to bring back -mlinker-version per bitcoin/bitcoin@f61019f5 - needs some further investigation

pip install doesn't work in offline gitian builds

I could change it to add the pypi precompiled .whl as an input requirement as we did for bionic (but we don't need to self-compile this) - but I'm not 100% sure. Does anyone have an opinion on this?

xanimo commented 1 year ago

Good catch on the -mlinker-version. Re: pip, I'd say adding precompiled .whl sounds sensible.

chromatic commented 1 year ago

I'd rather get pip install working in offline builds if we can -- it's safer! -- but can live with the .whl if we're careful. Maybe we should build and host it ourselves?

xanimo commented 1 year ago

edit: disregard mlinker comment, have just reviewed src for clang 6.0.1 and the comments in ref link are applicable.

Anyway, wondering if there are any additional steps needed that I can help out on/status update?

michilumin commented 1 year ago

(Note I'm in the process of reviewing this, hopefully by EOW 24 Apr, but I'm trying to be meticulous so taking time. Will delete/edit this comment when complete.)

patricklodder commented 1 year ago

Thanks Michi. I'm still testing the macOS linker differences and we still have a gitian comparison to do (I'll bring back the lief wheel as we had for bionic gitian, so that it will work on offline builds too.) ETA Friday May 5th for both.

patricklodder commented 1 year ago

edit: disregard mlinker comment, have just reviewed src for clang 6.0.1 and the comments in ref link are applicable.

I've found that our -mlinker-version does not add anything to the default, and tested this thoroughly with otool. However, this can change when we:

Upgrade Qt, and/or
Upgrade clang, and/or
Update the build system to better support MacOS symbol checks from #3204

Therefore we would introduce tech debt by removing it, so let's prevent that. I have this commit staged, am waiting on the results from an offline gitian build now - if that succeeds I will amend this PR.

patricklodder commented 1 year ago

I have fixed the 2 issues I found myself and have pushed them on top - but these should be squashed. Please let me know if 7665bf6 and be2a61f are okay, then I rebase+squash these before a final gitian check

xanimo commented 1 year ago

I have fixed the 2 issues I found myself and have pushed them on top - but these should be squashed. Please let me know if 7665bf6 and be2a61f are okay, then I rebase+squash these before a final gitian check

These look good but are untested from me. Outside of that am wondering if you'd like to bump CodeQL to focal as well? Just noticing it's hanging atm and will never complete and since it fits within the purview of this PR it couldn't hurt.

michilumin commented 1 year ago

Will re-review with these two commits and certainly gitian check once we're ready to do so. (which i'm guessing is in the next day or so)..

xanimo commented 1 year ago

Disregard CodeQL comment although if you'd like to cherry-pick or have me open a PR or if you want to do it let me know: https://github.com/xanimo/dogecoin/actions/runs/4900179242 and local test: https://github.com/xanimo/dogecoin/actions/runs/4900214623

patricklodder commented 1 year ago

if you'd like to cherry-pick

I'll do that together with the rebase.

michilumin commented 1 year ago

re-on this.

patricklodder commented 1 year ago

This has now been rebased. I have included the version bump for CodeQL in b65c56b.

I think all we need now is a couple of gitian comparisons to make sure all these tooling changes didn't kill deterministic building, and then test those binaries on as many platforms we think is feasible.

I'd recommend prominent old OS testing targets beyond the usual new ones we all run to include, in order of what has seen the most changes:

Older macOs (perhaps Catalina?)
RPI 2 for arm32-hf
WinXP (preferably 32-bit)
Ubuntu Trusty (pref 32-bit)

Open to ideas for anything else

xanimo commented 1 year ago

Will try to get my hands on some older platforms to test binaries. In the meantime here is output from last nights build using your gitian-build.sh script with docker:

a3bfc6aea038793f15ef1928350f74d316b4b4cc4a4209607304551551ba14b8  dogecoin-1.14.7-osx-unsigned.dmg
f28aee147818440a23a40caea5a57e82db8620f8e23cf79245285892bf9281c4  dogecoin-1.14.7-osx-unsigned.tar.gz
e934cefbebd9e08802fe827a4cbcd45e4ce656f1840d6a9a6f61c4fabb539af0  dogecoin-1.14.7-osx64.tar.gz
394a46e8fd125a4bcefb3cf6223705b74b10c9c8379d1ca7b5ac2cd86bc329c6  dogecoin-1.14.7-win-unsigned.tar.gz
defe10e5a66f39ef33f33b1dbe1c0b5cd29c7bede0b3981e7bbc7f1a4738502c  dogecoin-1.14.7-win32-debug.zip
2d31c31910c0279fe5ba49a3f1f7dcffa9d8c032125f1ba32900d1d8633267fb  dogecoin-1.14.7-win32-setup-unsigned.exe
bd96ceaad8bd5ebe591b43e1f54c79e4f132c518e23c1d24c0ffa9f57c60aedf  dogecoin-1.14.7-win32.zip
8b6a0909446d83a95c285458bcce8414eb66f3a49e0e6066cb6a28a4e6310c36  dogecoin-1.14.7-win64-debug.zip
1c2719f3d2065b4a735d5b6fc359cd38d448ce4499ecb4bf7eb4ebb1b303584d  dogecoin-1.14.7-win64-setup-unsigned.exe
9616b5f82813c9e258e8bdcc01dd07fce06155a2d062a2a0063ad80db77f747f  dogecoin-1.14.7-win64.zip
c87712433744b4f63c9ce2e1c123c252829371e1ebbd9d6e2191bf9ea64dbafe  dogecoin-1.14.7-aarch64-linux-gnu-debug.tar.gz
705f6d56991d04b3bda1202b54e09889813bebdb374ddf1556ec77fc23d9b899  dogecoin-1.14.7-aarch64-linux-gnu.tar.gz
bc5372b1df4299ed778e3b1bddb621e1a444e25e5b656f821b18ba09a7ac8bb7  dogecoin-1.14.7-arm-linux-gnueabihf-debug.tar.gz
e40c63694b1e72fe2df517703ec8ee59f5787e150b3c356146ec4d39895f0255  dogecoin-1.14.7-arm-linux-gnueabihf.tar.gz
14b0ab0eed791e6ee47af34d447a90c6de2097da8635b19d78fced627f03eb56  dogecoin-1.14.7-i686-pc-linux-gnu-debug.tar.gz
4859b560a5edede1ca6dd355d3a71db2df46713add5f8d8eb0f33de0be8d324d  dogecoin-1.14.7-i686-pc-linux-gnu.tar.gz
814acead867d2bd36450c31ca7fca9daf118a4c1e54e29ba0af622e7febcfa4c  dogecoin-1.14.7-x86_64-linux-gnu-debug.tar.gz
d387c010e03f122147aba9aa7ec6d9c3c569462b3d65fbc387aa7adeb659699f  dogecoin-1.14.7-x86_64-linux-gnu.tar.gz

xanimo commented 1 year ago

Forgot one:

0d4f054718e2d097c4aa614ddee5cbe863c55e4bab0ea2cb1d49c4bc15a3295c  src/dogecoin-1.14.7.tar.gz

michilumin commented 1 year ago

Working on gitian build, slow going (overnight runs)... New (but with updates) Focal install, getting a lot of (not at same point) download failures in the gitian-build.sh process. (Not a hard error or anything.) Wiggling resources around to see if it gets past it.

patricklodder commented 1 year ago

getting a lot of (not at same point) download failures in the gitian-build.sh process

I get a 404 for expat-2.2.0.tar.bz2, I'll create an issue for this.

michilumin commented 1 year ago

Finally. Apologies for the delay.

    394a46e8fd125a4bcefb3cf6223705b74b10c9c8379d1ca7b5ac2cd86bc329c6  dogecoin-1.14.7-win-unsigned.tar.gz
    defe10e5a66f39ef33f33b1dbe1c0b5cd29c7bede0b3981e7bbc7f1a4738502c  dogecoin-1.14.7-win32-debug.zip
    2d31c31910c0279fe5ba49a3f1f7dcffa9d8c032125f1ba32900d1d8633267fb  dogecoin-1.14.7-win32-setup-unsigned.exe
    bd96ceaad8bd5ebe591b43e1f54c79e4f132c518e23c1d24c0ffa9f57c60aedf  dogecoin-1.14.7-win32.zip
    8b6a0909446d83a95c285458bcce8414eb66f3a49e0e6066cb6a28a4e6310c36  dogecoin-1.14.7-win64-debug.zip
    1c2719f3d2065b4a735d5b6fc359cd38d448ce4499ecb4bf7eb4ebb1b303584d  dogecoin-1.14.7-win64-setup-unsigned.exe
    9616b5f82813c9e258e8bdcc01dd07fce06155a2d062a2a0063ad80db77f747f  dogecoin-1.14.7-win64.zip

    a3bfc6aea038793f15ef1928350f74d316b4b4cc4a4209607304551551ba14b8  dogecoin-1.14.7-osx-unsigned.dmg
    f28aee147818440a23a40caea5a57e82db8620f8e23cf79245285892bf9281c4  dogecoin-1.14.7-osx-unsigned.tar.gz
    e934cefbebd9e08802fe827a4cbcd45e4ce656f1840d6a9a6f61c4fabb539af0  dogecoin-1.14.7-osx64.tar.gz

    c87712433744b4f63c9ce2e1c123c252829371e1ebbd9d6e2191bf9ea64dbafe  dogecoin-1.14.7-aarch64-linux-gnu-debug.tar.gz
    705f6d56991d04b3bda1202b54e09889813bebdb374ddf1556ec77fc23d9b899  dogecoin-1.14.7-aarch64-linux-gnu.tar.gz
    bc5372b1df4299ed778e3b1bddb621e1a444e25e5b656f821b18ba09a7ac8bb7  dogecoin-1.14.7-arm-linux-gnueabihf-debug.tar.gz
    e40c63694b1e72fe2df517703ec8ee59f5787e150b3c356146ec4d39895f0255  dogecoin-1.14.7-arm-linux-gnueabihf.tar.gz
    14b0ab0eed791e6ee47af34d447a90c6de2097da8635b19d78fced627f03eb56  dogecoin-1.14.7-i686-pc-linux-gnu-debug.tar.gz
    4859b560a5edede1ca6dd355d3a71db2df46713add5f8d8eb0f33de0be8d324d  dogecoin-1.14.7-i686-pc-linux-gnu.tar.gz
    814acead867d2bd36450c31ca7fca9daf118a4c1e54e29ba0af622e7febcfa4c  dogecoin-1.14.7-x86_64-linux-gnu-debug.tar.gz
    d387c010e03f122147aba9aa7ec6d9c3c569462b3d65fbc387aa7adeb659699f  dogecoin-1.14.7-x86_64-linux-gnu.tar.gz

edtubbs commented 11 months ago

Running test_dogecoin.exe from dogecoin-1.14.7-win32.zip generates this error on Windows XP Pro (Version 2002, Service Pack 3) with a Pentium 4 @ 2.80 GHz: "The procedure entry point if_nametoindex could not be located in the dynamic link library IPHLPAPI.DLL."

According to this discussion (https://github.com/libevent/libevent/pull/1111), if_nametoindex function is only supported from Windows Vista onwards. We can probably remove this dependency to make core compatible with XP. I can dig deeper if needed.

patricklodder commented 11 months ago

According to this discussion (libevent/libevent#1111), if_nametoindex function is only supported from Windows Vista onwards. We can probably remove this dependency to make core compatible with XP. I can dig deeper if needed.

Thanks, Ed! That's a good find.

To make sure that this is introduced with switching the gcc version and that this was not a regression from the libevent update in 1.14.6, could you let us know what happens when you run the test_dogecoin.exe that is included in the win32 zip from the 1.14.6 release?

patricklodder commented 11 months ago

PS: I double checked and this was one of the libraries I had to add when testing 1.14.6 (64-bit) with the recent symbol check upgrade.

This would mean that 1.14.6 would throw the same error - if this gets confirmed per question above, then this is a separate regression (on 1.14.6) and I'd suggest to give Win32/XP some TLC after we merge this (because then we'd have a working CI)

edtubbs commented 11 months ago

Thanks, Ed! That's a good find.

To make sure that this is introduced with switching the gcc version and that this was not a regression from the libevent update in 1.14.6, could you let us know what happens when you run the test_dogecoin.exe that is included in the win32 zip from the 1.14.6 release?

Sure thing! Getting the same error from 1.14.6 test_dogecoin.exe.

This would mean that 1.14.6 would throw the same error - if this gets confirmed per question above, then this is a separate regression (on 1.14.6) and I'd suggest to give Win32/XP some TLC after we merge this (because then we'd have a working CI)

Ok, agreed. Sounds good!

patricklodder commented 11 months ago

Getting the same error from 1.14.6 test_dogecoin.exe.

Thanks for verifying this!

Ok, agreed. Sounds good!

I've created issue #3276 from your finding, can be solved after this PR merges.

edtubbs commented 11 months ago

BeagleBone Black (Ubuntu 16.04.7 LTS)

Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
Model name:            ARMv7 Processor rev 2 (v7l)
CPU max MHz:           1000.0000
CPU min MHz:           300.0000

ubuntu@arm:~$ ldd --version
ldd (Ubuntu GLIBC 2.23-0ubuntu11.3) 2.23
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

ubuntu@arm:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/arm-linux-gnueabihf/5/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-libitm --disable-libquadmath --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-armhf/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-armhf --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-armhf --with-arch-directory=arm --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --enable-multilib --disable-sjlj-exceptions --with-arch=armv7-a --with-fpu=vfpv3-d16 --with-float=hard --with-mode=thumb --disable-werror --enable-multilib --enable-checking=release --build=arm-linux-gnueabihf --host=arm-linux-gnueabihf --target=arm-linux-gnueabihf
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu/Linaro 5.4.0-6ubuntu1~16.04.12)

ubuntu@arm:~$ ./dogecoin/dogecoin-1.14.7/bin/test_dogecoin 
./dogecoin/dogecoin-1.14.7/bin/test_dogecoin: relocation error: ./dogecoin/dogecoin-1.14.7/bin/test_dogecoin: symbol clock_gettime, version GLIBC_2.4 not defined in file libc.so.6 with link time reference

ubuntu@arm:~$ time ./dogecoin/dogecoin-1.14.6/bin/test_dogecoin 
Running 252 test cases...

*** No errors detected

real    14m30.454s
user    10m55.684s
sys 3m28.836s
ubuntu@arm:~$

test_dogecoin from dogecoin-1.14.7-arm-linux-gnueabihf.tar.gz generated an error on this armv7 target, but 1.14.6 did not.

patricklodder commented 11 months ago

./dogecoin/dogecoin-1.14.7/bin/test_dogecoin: relocation error:
./dogecoin/dogecoin-1.14.7/bin/test_dogecoin: symbol clock_gettime, version 
GLIBC_2.4 not defined in file libc.so.6 with link time reference

Have been able to reproduce this on xenial x86_64 too, it has something to do with librt not being required (readelf -d)

patricklodder commented 11 months ago

I did a bunch of comparisons using this configuration:

./configure --prefix=`pwd`/depends/x86_64-linux-gnu --enable-reduce-exports --enable-glibc-back-compat --enable-zmq LDFLAGS="-static-libstdc++"

After building 1.14.7-dev with bionic / gcc-7:

$ readelf -d src/dogecoind

Dynamic section at offset 0xb82b48 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
...

After building #3222 with focal / gcc-9:

$ readelf -d src/dogecoind

Dynamic section at offset 0xb31af8 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
...

The trick is to force include librt, but only that and nothing else, because we WANT to have a minimum set of shared lib dependencies. I applied this trick from an ubuntu list to change the way we set -lrt to LDFLAGS:

unset "as-needed": -Wl,-no-as-needed
link librt: -Wl,-lrt
re-set "as-needed": -Wl,-as-needed

This only adds librt.so.1 as a needed shared lib, and not any other stuff we don't need. In configure.ac I changed:

   #glibc absorbed clock_gettime in 2.17. librt (its previous location) is safe to link
   #in anyway for back-compat.
-  AC_CHECK_LIB([rt],[clock_gettime],, AC_MSG_ERROR(lib missing))
+  AX_CHECK_LINK_FLAG([[-Wl,-no-as-needed -Wl,-lrt -Wl,-as-needed]],[COMPAT_LDFLAGS="$COMPAT_LDFLAGS -Wl,-no-as-needed -Wl,-lrt -Wl,-as-needed"])

After building #3222 with focal / gcc-9 with this patch:

 $ readelf -d src/dogecoind

Dynamic section at offset 0xb31af8 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
...

I've tested the resulting binary on jammy, xenial and trusty x86_64, and those work for me. Will need to redo gitian check for linux.

xanimo commented 11 months ago

e1c688dfd2df9377b1228735e0f2f0e0e164e53e is v nice sir. booting up gitian vm and will start a build as soon as ed confirms the changes work on his older machines.

edtubbs commented 11 months ago

Thanks @patricklodder! Built your changes for armv7 and test_dogecoin passes on the target.

ubuntu@arm:~/dogecoin$ time ./test_dogecoin 
Running 253 test cases...

*** No errors detected

real    13m46.996s
user    10m15.776s
sys 3m20.888s
ubuntu@arm:~/dogecoin$ lscpu
Architecture:          armv7l
Byte Order:            Little Endian
CPU(s):                1
On-line CPU(s) list:   0
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             1
Model name:            ARMv7 Processor rev 2 (v7l)
CPU max MHz:           1000.0000
CPU min MHz:           300.0000
ubuntu@arm:~/dogecoin$

chromatic commented 11 months ago

Failure for me on 64-bit x86 Linux (Ubuntu 22.04.2 LTS):

============================================================================
Testsuite summary for Dogecoin Core 1.14.7
============================================================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See src/test-suite.log
Please report to https://github.com/dogecoin/dogecoin/issues
============================================================================
make[4]: *** [Makefile:9960: test-suite.log] Error 1
make[4]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[3]: *** [Makefile:10068: check-TESTS] Error 2
make[3]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[2]: *** [Makefile:10173: check-am] Error 2
make[2]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[1]: *** [Makefile:9852: check-recursive] Error 1
make[1]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make: *** [Makefile:694: check-recursive] Error 1

Contents of src/test-suite.log:

********* Start testing of URITests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : URITests::initTestCase()
PASS   : URITests::uriTests()
PASS   : URITests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of URITests *********
********* Start testing of PaymentServerTests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : PaymentServerTests::initTestCase()
FAIL!  : PaymentServerTests::paymentServerTests() Caught unhandled exception
   Loc: [qtestcase.cpp(1939)]
Totals: 1 passed, 1 failed, 0 skipped, 0 blacklisted, 24ms
********* Finished testing of PaymentServerTests *********
terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
  what():  boost::filesystem::create_directory: File exists: "/home/chromatic/.dogecoin"
FAIL qt/test/test_dogecoin-qt (exit status: 134)

xanimo commented 11 months ago

e28b7be2d835ed09e1bd11105a0df8591972918a31d141cb83d601aece96fc36  dogecoin-1.14.7-osx-unsigned.dmg
1016154956c80663f813fa279ee56796e8d9de3a9439d72aeae24c19b6c48fe4  dogecoin-1.14.7-osx-unsigned.tar.gz
adf08a3926939dd01802a7adacdf47c987b67c8b5a42ba4aa371150587554d46  dogecoin-1.14.7-osx64.tar.gz

79a7f345a5f6d4b2ebfed8b1247eb7d63300287a9fb1746d00557f85069b2232  dogecoin-1.14.7-win-unsigned.tar.gz
a64791c085a9a8e3e6f832e699fadd4a97139e4ee2510c2da54cc0d33edc1ca0  dogecoin-1.14.7-win32-debug.zip
69dc364611ee001623454bae81d9c9d2e55daacfba17c1f12ceebe18eb79e727  dogecoin-1.14.7-win32-setup-unsigned.exe
e0ebed4132e190d264f1b8d2952b39d63eadefa44763f25e14eaf22edf712985  dogecoin-1.14.7-win32.zip
65ceaf2df96c6bf4a2aa00e3daab3a0bb40d2671d9af1f8508fe53420cb058df  dogecoin-1.14.7-win64-debug.zip
2cdcfc497b327cf340b88528a09ac784b2961dff6c6ba7b360ee3cc4cba08e12  dogecoin-1.14.7-win64-setup-unsigned.exe
30b9604b0503703d9779cd295f3564cd8eb4b5508ad898c749717dc6c1422908  dogecoin-1.14.7-win64.zip

f22f79fe7e1ece3ee268e1999212b761f94194c155764a260e5a5794e521ef57  dogecoin-1.14.7-aarch64-linux-gnu-debug.tar.gz
05d3ae1662e15c8ffcfeb0ce6da3e10734ee5d91497ca08f2a84e11b7cfb457e  dogecoin-1.14.7-aarch64-linux-gnu.tar.gz
b03423ba9c70168b29452781c5e838b7c1152b78d04a4bc9a222f70f0b438635  dogecoin-1.14.7-arm-linux-gnueabihf-debug.tar.gz
eb4bb79bd3b0cd9f36050e74b995975910c4b5c1613b06a7f642c97d19424d02  dogecoin-1.14.7-arm-linux-gnueabihf.tar.gz
1d5b34a4e40ea28a6a7bd39a925599650c16a32038a31b9b245e5f38563588de  dogecoin-1.14.7-i686-pc-linux-gnu-debug.tar.gz
95ba2b5c3b427e4de902897184df800084346b5d633119888ff4a64738a77852  dogecoin-1.14.7-i686-pc-linux-gnu.tar.gz
be46009a2c1e29e8fbd2076dd188b82a5581f6a2a6a53d5fd5320f6106f27a76  dogecoin-1.14.7-x86_64-linux-gnu-debug.tar.gz
a56b84bfe3fbca99b8017b1dacea14fb0923d44a062b71cdaedf274e26357e51  dogecoin-1.14.7-x86_64-linux-gnu.tar.gz

6f7190189f109765e29f49557b57a105881d38c6b51493b6964a858be76e8a52  src/dogecoin-1.14.7.tar.gz

xanimo commented 11 months ago

new test target link and binaries available to test: https://gist.github.com/xanimo/dec5f0f1a91cbc6638eea482cf8890b8

@chromatic going to compile locally on my 64-bit jammy. will post results shortly.

patricklodder commented 11 months ago

Failure for me on 64-bit x86 Linux (Ubuntu 22.04.2 LTS)

Cannot repro with a ./configure --with-gui=qt5. @chromatic what's your configuration?

********* Start testing of URITests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : URITests::initTestCase()
PASS   : URITests::uriTests()
PASS   : URITests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 32ms
********* Finished testing of URITests *********
********* Start testing of PaymentServerTests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : PaymentServerTests::initTestCase()
QDEBUG : PaymentServerTests::paymentServerTests() PaymentServer::initNetManager: No active proxy server found.
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyNetwork: Payment request network \"test\" doesn't match client network \"1a91e3dace36e2be3bf030a65679fe821aa1d6ef92e7c9902eb318182c355691\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyExpired: Payment request expired \"1970-01-01 00:00:01\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifyExpired: Payment request expired \"1970-01-01 00:00:00\"."
QWARN  : PaymentServerTests::paymentServerTests() "PaymentServer::verifySize: Payment request too large (50001 bytes, allowed 50000 bytes)."
PASS   : PaymentServerTests::paymentServerTests()
PASS   : PaymentServerTests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 548ms
********* Finished testing of PaymentServerTests *********
********* Start testing of RPCNestedTests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : RPCNestedTests::initTestCase()
PASS   : RPCNestedTests::rpcNestedTests()
PASS   : RPCNestedTests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 156ms
********* Finished testing of RPCNestedTests *********
********* Start testing of CompatTests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : CompatTests::initTestCase()
PASS   : CompatTests::bswapTests()
PASS   : CompatTests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 1ms
********* Finished testing of CompatTests *********

xanimo commented 11 months ago

Failure for me on 64-bit x86 Linux (Ubuntu 22.04.2 LTS):

============================================================================
Testsuite summary for Dogecoin Core 1.14.7
============================================================================
# TOTAL: 2
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See src/test-suite.log
Please report to https://github.com/dogecoin/dogecoin/issues
============================================================================
make[4]: *** [Makefile:9960: test-suite.log] Error 1
make[4]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[3]: *** [Makefile:10068: check-TESTS] Error 2
make[3]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[2]: *** [Makefile:10173: check-am] Error 2
make[2]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make[1]: *** [Makefile:9852: check-recursive] Error 1
make[1]: Leaving directory '/home/chromatic/dev/dogecoin/src'
make: *** [Makefile:694: check-recursive] Error 1

Contents of src/test-suite.log:

********* Start testing of URITests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : URITests::initTestCase()
PASS   : URITests::uriTests()
PASS   : URITests::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped, 0 blacklisted, 0ms
********* Finished testing of URITests *********
********* Start testing of PaymentServerTests *********
Config: Using QtTest library 5.15.3, Qt 5.15.3 (x86_64-little_endian-lp64 shared (dynamic) release build; by GCC 11.3.0), ubuntu 22.04
PASS   : PaymentServerTests::initTestCase()
FAIL!  : PaymentServerTests::paymentServerTests() Caught unhandled exception
   Loc: [qtestcase.cpp(1939)]
Totals: 1 passed, 1 failed, 0 skipped, 0 blacklisted, 24ms
********* Finished testing of PaymentServerTests *********
terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
  what():  boost::filesystem::create_directory: File exists: "/home/chromatic/.dogecoin"
FAIL qt/test/test_dogecoin-qt (exit status: 134)

running:

make -C depends HOST=x86_64-pc-linux-gnu
./autogen.sh
./configure --prefix=`pwd`/depends/x86_64-pc-linux-gnu --enable-reduce-exports --enable-glibc-back-compat --enable-zmq LDFLAGS="-static-libstdc++"
make check

yielded:

PASS: test/test_dogecoin
PASS: qt/test/test_dogecoin-qt
============================================================================
Testsuite summary for Dogecoin Core 1.14.7
============================================================================
# TOTAL: 2
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

so can't repro either. perhaps you're using wrong gcc? mine is:

bluezr@bluezr:/mnt/source/repos/dogecoin$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.5.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-5Q4PKF/gcc-9-9.5.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-mutex
Thread model: posix
gcc version 9.5.0 (Ubuntu 9.5.0-1ubuntu1~22.04)

patricklodder commented 11 months ago

@xanimo: on Jammy with gcc 9? It packs 11 out of the box:

gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04.1)

Also --enable-glibc-back-compat is adapted for focal only (see PR title), and we bumped depends to support focal here, not jammy, so all that would exceed the goal of this PR 😕

xanimo commented 11 months ago

yeah i know it's beyond scope. i even had just run again with --with-gui=qt5 and get same results but haven't confirmed symbol-check, just wanted to see if i got the same results as @chromatic.

patricklodder commented 11 months ago

Here's a Dockerfile to compare apples to apples (make sure you distclean wherever you run it from(!)):

FROM ubuntu:jammy

# tz setting to prevent interactive crap
ENV TZ=Etc/UTC
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# build system and system deps per doc/build-unix.md
RUN apt-get update && \
    apt-get install -y curl git build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-program-options-dev libboost-test-dev libboost-thread-dev libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libprotobuf-dev protobuf-compiler libqrencode-dev libdb5.3++-dev libdb5.3++ libdb5.3-dev libzmq3-dev libminiupnpc-dev

# build it
WORKDIR /build
COPY . .

RUN ./autogen.sh && \
    ./configure --enable-gui=qt5 --disable-bench && \
    make -j12 -C src all

# test it
RUN make check

Please let me know what I need to change to make it fail on this PR but pass on 1.14.7-dev (i.e. it's a regression of THIS pull request)

Edit: to make it totally unambiguous how to test 😂

cd /tmp
# store above Dockerfile in /tmp/awesomedockerfile
git clone https://github.com/patricklodder/dogecoin my-3222-test
cd my-3222-test
git checkout 1.14.7-focal
docker build -f /tmp/awesomedockerfile .

xanimo commented 11 months ago

so just after initial attempt to build (for fun...) i'm getting resolution errors :'( and not sure if i should take any more time or space away from reviewers...

 => ERROR [3/7] RUN apt-get update &&     apt-get install -y curl git build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-  9.5s
------                                                                                                                                                                                                                           
 > [3/7] RUN apt-get update &&     apt-get install -y curl git build-essential libtool autotools-dev automake pkg-config libssl-dev libevent-dev bsdmainutils libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-program-options-dev libboost-test-dev libboost-thread-dev libqt5gui5 libqt5core5a libqt5dbus5 qttools5-dev qttools5-dev-tools libprotobuf-dev protobuf-compiler libqrencode-dev libdb5.3++-dev libdb5.3++ libdb5.3-dev libzmq3-dev libminiupnpc-dev:                                                                                                                                                                                                        
#0 0.372 Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease                                                                                                                                                                  
#0 0.372 Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease                                                                                                                                                        
#0 0.372 Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#0 0.372 Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#0 1.374 Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
#0 1.374 Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
#0 1.374 Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#0 1.374 Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#0 3.373 Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
#0 3.374 Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
#0 3.374 Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#0 3.374 Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#0 7.382 Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
#0 7.382   Temporary failure resolving 'archive.ubuntu.com'
#0 7.382 Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
#0 7.382   Temporary failure resolving 'archive.ubuntu.com'
#0 7.382 Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
#0 7.382   Temporary failure resolving 'security.ubuntu.com'
#0 7.382 Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
#0 7.382   Temporary failure resolving 'archive.ubuntu.com'

patricklodder commented 11 months ago

I'm sure that's an error in this PR and not in your DNS server 🤣

xanimo commented 11 months ago

retested f449518 after running make distclean and that passed and then ended up copying Dockerfile to /tmp as awesomedockerfile and running your commands @patricklodder for 1.14.7-focal and that passed.

EDIT: if anyone wants to audit the contents of build images they are here: https://hub.docker.com/layers/xanimo/dogecoin/f4495189/images/sha256-4cd71a572fcfe761efe046a14381eb339810b2bc7db34a08ce74c1448d30f77d?context=explore (dogecoin:1.14.7-dev) https://hub.docker.com/layers/xanimo/dogecoin/focal/images/sha256-b6cf7d6a4988394e6a1c244e5dbe44dc700b96d9fc34b18b57dee06b1aeca74a?context=explore (patricklodder:1.14.7-focal)

patricklodder commented 11 months ago

if anyone wants to audit the contents of build images

The 1.14.7-dev image is copying A LOT of data still on step 11 - unclean depends/? (shouldn't matter for this build tho)

xanimo commented 11 months ago

if anyone wants to audit the contents of build images

The 1.14.7-dev image is copying A LOT of data still on step 11 - unclean depends/? (shouldn't matter for this build tho)

yeh it's unclean depends/ dir. doh! 🤦

xanimo commented 11 months ago

if anyone wants to audit the contents of build images

The 1.14.7-dev image is copying A LOT of data still on step 11 - unclean depends/? (shouldn't matter for this build tho)

yeh it's unclean depends/ dir. doh! facepalm

updated image without depends build cache: https://hub.docker.com/layers/xanimo/dogecoin/f4495189/images/sha256-afedfe0d97d35c6ef8123a5b505f23bb878600006f02ba6f0d2359027c8a4ca0?context=explore

chromatic commented 11 months ago

I can reproduce my test error on 1.14.7-dev as well, so it's something specific to my environment and not this PR. I'll debug, but don't let it block further progress here.

edtubbs commented 11 months ago

new test target link and binaries available to test: https://gist.github.com/xanimo/dec5f0f1a91cbc6638eea482cf8890b8

arm-linux-gnueabihf
ubuntu@arm:~/dogecoin/dogecoin-1.14.7/bin$ time ./test_dogecoin
Running 253 test cases...

*** No errors detected

real 14m19.197s user 10m31.040s sys 3m31.264s ubuntu@arm:~/dogecoin/dogecoin-1.14.7/bin$ lscpu Architecture: armv7l Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 Model name: ARMv7 Processor rev 2 (v7l) CPU max MHz: 1000.0000 CPU min MHz: 300.0000 ubuntu@arm:~/dogecoin/dogecoin-1.14.7/bin$


Re-verified https://github.com/dogecoin/dogecoin/issues/3276 on Windows XP with win32 binary.

chromatic commented 11 months ago

:facepalm:

I fixed my error; my .dogecoin directory was not mounted properly. All tests pass for me on 64-bit x86 Linux (Ubuntu 22.04). I'll work on gitian over the weekend.

chromatic commented 11 months ago

My results match @xanimo 's:

e28b7be2d835ed09e1bd11105a0df8591972918a31d141cb83d601aece96fc36  dogecoin-1.14.7-osx-unsigned.dmg
1016154956c80663f813fa279ee56796e8d9de3a9439d72aeae24c19b6c48fe4  dogecoin-1.14.7-osx-unsigned.tar.gz
adf08a3926939dd01802a7adacdf47c987b67c8b5a42ba4aa371150587554d46  dogecoin-1.14.7-osx64.tar.gz

79a7f345a5f6d4b2ebfed8b1247eb7d63300287a9fb1746d00557f85069b2232  dogecoin-1.14.7-win-unsigned.tar.gz
a64791c085a9a8e3e6f832e699fadd4a97139e4ee2510c2da54cc0d33edc1ca0  dogecoin-1.14.7-win32-debug.zip
69dc364611ee001623454bae81d9c9d2e55daacfba17c1f12ceebe18eb79e727  dogecoin-1.14.7-win32-setup-unsigned.exe
e0ebed4132e190d264f1b8d2952b39d63eadefa44763f25e14eaf22edf712985  dogecoin-1.14.7-win32.zip
65ceaf2df96c6bf4a2aa00e3daab3a0bb40d2671d9af1f8508fe53420cb058df  dogecoin-1.14.7-win64-debug.zip
2cdcfc497b327cf340b88528a09ac784b2961dff6c6ba7b360ee3cc4cba08e12  dogecoin-1.14.7-win64-setup-unsigned.exe
30b9604b0503703d9779cd295f3564cd8eb4b5508ad898c749717dc6c1422908  dogecoin-1.14.7-win64.zip

f22f79fe7e1ece3ee268e1999212b761f94194c155764a260e5a5794e521ef57  dogecoin-1.14.7-aarch64-linux-gnu-debug.tar.gz
05d3ae1662e15c8ffcfeb0ce6da3e10734ee5d91497ca08f2a84e11b7cfb457e  dogecoin-1.14.7-aarch64-linux-gnu.tar.gz
1d5b34a4e40ea28a6a7bd39a925599650c16a32038a31b9b245e5f38563588de  dogecoin-1.14.7-i686-pc-linux-gnu-debug.tar.gz
b03423ba9c70168b29452781c5e838b7c1152b78d04a4bc9a222f70f0b438635  dogecoin-1.14.7-arm-linux-gnueabihf-debug.tar.gz
eb4bb79bd3b0cd9f36050e74b995975910c4b5c1613b06a7f642c97d19424d02  dogecoin-1.14.7-arm-linux-gnueabihf.tar.gz
95ba2b5c3b427e4de902897184df800084346b5d633119888ff4a64738a77852  dogecoin-1.14.7-i686-pc-linux-gnu.tar.gz
be46009a2c1e29e8fbd2076dd188b82a5581f6a2a6a53d5fd5320f6106f27a76  dogecoin-1.14.7-x86_64-linux-gnu-debug.tar.gz
a56b84bfe3fbca99b8017b1dacea14fb0923d44a062b71cdaedf274e26357e51  dogecoin-1.14.7-x86_64-linux-gnu.tar.gz

6f7190189f109765e29f49557b57a105881d38c6b51493b6964a858be76e8a52  src/dogecoin-1.14.7.tar.gz

61592c4cc346f46573a75e16f646f210c89d1c9fff498dc2685d39ab1e4da3c8  dogecoin-win-1.14-res.yml
4da0927ea5e8ba722237eaa8e0e2c842c37438bc626bdff6b7f956fe17554173  dogecoin-osx-1.14-res.yml
a2cfc7289e2df8ef80204e8e5cfe1564f32d215d34c714c5413414815468bd92  dogecoin-linux-1.14-res.yml

dogecoin / dogecoin