search

ElektraInitiative / libelektra

Elektra serves as a universal and secure framework to access configuration settings in a global, hierarchical key database.
https://www.libelektra.org
BSD 3-Clause "New" or "Revised" License
208 stars 123 forks source link

Build Server stuff #160

Closed markus2330 closed 2 months ago

markus2330 commented 9 years ago

This issue gives up-to-date information about the health of our build system.

Report here any permanent problems (that cannot be fixed by rerunning the build job). Temporary problems should be reported in #2967.

Current open issues for our Jenkins build system:

Fixed issues:

manuelm commented 9 years ago

@markus2330

Just pushed a few build system related fixes. But you need to fix some packages on your stable debian-stable machine as well:

markus2330 commented 9 years ago

Looks good, thanks for fixing those issues.

I also did those steps on debian-stable agent.

For other machines installing qtdeclarative5-dev was not possible, because it conflicts with qdbus which is needed by kde4. So I restored the previous script configure-debian-wheezy as configure-debian-wheezy-local.

I also added the installation steps you mentioned as notes in the README.md because they might be of interest by others.

markus2330 commented 9 years ago

Thanks for upgrading the agents!

Stuff that is missing on stable

1.) latex (+ I think texlive-latex-recommended is needed, too) see http://build.libelektra.org:8080/job/elektra-doc/495/console

-- Found Doxygen: /usr/bin/doxygen (found version "1.8.8") 
CMake Warning at doc/CMakeLists.txt:46 (message):
  Latex not found, PDF Manual can't be created.

-- Found Perl: /usr/bin/perl (found version "5.20.2") 
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_EXAMPLES

2.) Can you install clang (for elektra-clang, clang of wheezy won't work)? 3.) Can you install mingw+wine for elektra-gcc-configure-mingw?

manuelm commented 9 years ago

apt install --no-install-recommends doxygen-latex + clang + mingw done

Why do you need wine?

Btw, you should change i586-mingw32msvc-X to i686-w64-mingw32-X in Toolchain-mingw32.cmake. Right now this won't work on unstable.

markus2330 commented 9 years ago

Thank you for docu!

wine is needed to execute the cross-compiled windows binaries (e.g. exporterrors.exe)

I think you installed the mingw that builds for w64. In the mingw32 package, there is still a /usr/bin/i586-mingw32msvc-c++.

A new toolchain file for w64 is nevertheless appreciated.

manuelm commented 9 years ago

I installed gcc-mingw-w64-i686 which is the x64 build of mingw with i686 as target. The package mingw32-binutils is deprecated and not available on unstable any more.

manuelm commented 9 years ago

Wine installed on both containers.

markus2330 commented 9 years ago

Actually, the mingw build is bound to stable, so that should not be an issue.

MinGW-w64 is a fork on mingw and is quite a different target. Nobody tested it up to now.

markus2330 commented 9 years ago

thanks for installing wine

manuelm commented 9 years ago

Mingw-w64 looks superior. Maybe it's time to move on :-)

markus2330 commented 9 years ago

Contributions welcome ;) I do not have any machine to test it.

markus2330 commented 9 years ago

I am afraid you got the wrong wine, it should be apt-get install wine32

see also http://build.libelektra.org:8080/job/elektra-gcc-configure-mingw/218/console

manuelm commented 9 years ago

Nope.

root@debian-stable:~# apt-get install wine32
....
E: Package 'wine32' has no installation candidate
manuelm commented 9 years ago

ok, dpkg --add-architecture i386 will solve this. But can't you just pin the mingw/wine job to your build machine? The mingw setup is rather special.

Edit: I'll see if I can get elektra build with mingw-w64 so I don't need to install tons of i686 libs.

markus2330 commented 9 years ago

The problem is I do not have a spare jessie machine and wheezy's mingw does not know C++11.

manuelm commented 8 years ago

I managed to get mingw-w64 working. However std::mutex is not available because there's no glibc on windows and std::mutex depends on pthreads. Any ideas?

markus2330 commented 8 years ago

Wow, thanks!

Does it lead to a compilation error? The std::mutex is not used for internal functionality, but only in a header file to be included by a user. It is used in test cases though.

One solution for compiliation problems is to provide a std::mutex in the mingw case throwing system errors on every attempt to lock/unlock. Actually, I would expect the mingw people to at least provide something like that (e.g. when some macro is set, similar to -D_GLIBCXX_USE_NANOSLEEP)

https://github.com/meganz/mingw-std-threads might be another way. But that is most likely only useful if all test cases except the ones involving std::mutex already run.

Basically, this is only one instance of C++11 not properly available.

manuelm commented 8 years ago

mingw status right now:

Right now I'm building with -static-libgcc + -static-libstdc++. Otherwise wine is unable to find the dlls. Additional mutex doesn't compile either. I tried mingw-std-threads. just got more compile errors :-)

If I switch from x86_64-w64-mingw32-X to x86_64-w64-mingw32-X-posix std::mutex compiles fine, because pthread stuff is defined. However I get an additional dependency to libwinpthread-1.dll, which wine is unable to find.

I think our best bet is using x86_64-w64-mingw32-X-posix though.

markus2330 commented 8 years ago

Again, I am surprised that you even have this problem. Up to now we were happy when we got a libelektra.dll.

I cannot say anything about this x86_64-w64-mingw32-X-posix decision, because I do not use it and do not know the implications. I am wondering that such a posix-lib even exists, I thought that the posix-layer approach is cygwin and not mingw.

Does this decision even has an effect on libelektra.dll? If its only for the test cases, no one will care (as long as the build server is able to run it). If the test cases run, it will be a huge benefit. (See #270 where the unit tests unveiled some strange bugs on Mac OS X)

It seems like that libwinpthread-1.dll can be downloaded, I do not know if they work with wine though? Can you also add it as external project like done with dlfcn-win32 (so that all dlls are handled in the same way)? Otherwise, if you need to download 1 or 3 dlls for the tests might not really matter (again, I am no user, and do not understand the deployment concept, if there is any, of windows dlls).

@beku What do you think? Do you have time to test our latest 0.8.13 mingw-w64 build on Windows together with oyranos?

manuelm commented 8 years ago

Are tests usually enabled for the mingw build job? Yesterday all of them were disabled.

markus2330 commented 8 years ago

Yes, they were disabled. But afaik examples/benchmarks like cpp11_benchmark_thread were disabled, too. So I thought you changed it and compile more than it was done previously.

manuelm commented 8 years ago

I compiled the whole repo with C++11 enabled. Nothing more.

But executables like bin/basename.exe built with -posix run fine as long as you copy the required dlls to the bin directory (thank you windows for not having RPATH). I haven't found a way to a) let cmake find the dll directory + b) point wine to the dll directory. I thought static linking would work but then the build fails with duplicate symbols during linking the elektra dll. Because the dll already has the symbols included.

manuelm commented 8 years ago

@markus2330 I managed to get elektra to compile with mingw + running with wine without copying any dlls. The trick is to always enable static linking for both executable AND shared objects (CMAKE_SHARED_LINKER_FLAGS/CMAKE_EXE_LINKER_FLAGS => "-static").

To work around duplicated symbols I've added version-scripts for libelektra and libelektratools. This way only our symbols get exported.

This works really fine. e.g.

$ wine64 ./bin/kdb-static.exe
Usage: Z:\home\manuel\build\bin\kdb-static.exe <command> [args]

Z:\home\manuel\build\bin\kdb-static.exe is a program to manage elektra's key database.
Run a command with -H or --help as args to show a help text for
a specific command.

Known commands are:
check   Do some basic checks on a plugin.
convert Convert configuration.
cp      Copy keys within the key database.
export  Export configuration from the key database.
file    Prints the file where a key is located.
fstab   Create a new fstab entry.
get     Get the value of an individual key.
[...]

$ wine64 bin/cpp_example_iter.exe
user/key3/1
user/key3/2
user/key3/3

Even bin/cpp11_benchmark_thread.exe works.

Other things just crash:

$ wine64 ./bin/kdb-static.exe get
wine: Unhandled page fault on read access to 0x00000000 at address 0x7fd0e8b62c8a (thread 0009), starting debugger...
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Unhandled exception: page fault on read access to 0x00000000 in 64-bit code (0x00007fd0e8b62c8a).
Register dump:
 rip:00007fd0e8b62c8a rsp:000000000033f428 rbp:0000000000000000 eflags:00010293 (  R- --  I S -A- -C)
 rax:0000000000000000 rbx:000000000033f700 rcx:0000000000000000 rdx:000000000033f5b0
 rsi:0000000000000000 rdi:0000000000000000  r8:0000000000000000  r9:0000000000000072 r10:0000000000000000
 r11:000000000003f615 r12:000000000033f5b0 r13:00000000000373b0 r14:0000000000000000 r15:000000000033f930
Stack dump:
0x000000000033f428:  00007fd0e748ea93 0000000000000000
0x000000000033f438:  0000000000000000 0000000000000000
0x000000000033f448:  0000000000000028 0000000000010020
0x000000000033f458:  8d98315017c96400 6f46746547485300
0x000000000033f468:  687461507265646c 0000000000000000
0x000000000033f478:  0000000000000000 0000000000000000
0x000000000033f488:  000000000003fab0 0000000000030000
0x000000000033f498:  8d98315017c96400 6f46746547485300
0x000000000033f4a8:  687461507265646c 0000000000000000
0x000000000033f4b8:  0000000000000000 0000000000000000
0x000000000033f4c8:  0000000000000000 0000000000000000
0x000000000033f4d8:  0000000000000000 0000000000000000
Backtrace:
=>0 0x00007fd0e8b62c8a strlen+0x2a() in libc.so.6 (0x0000000000000000)
  1 0x00007fd0e748ea93 MSVCRT_stat64+0x92() in msvcrt (0x0000000000000000)
  2 0x00000000004744af in kdb-static (+0x744ae) (0x000000000003f9d0)
  3 0x000000000043bda5 in kdb-static (+0x3bda4) (0x000000000003f9d0)
  4 0x0000000000431d76 in kdb-static (+0x31d75) (0x00000000000360a0)
[...]

Right now I've simply added the version-script stuff without thinking about other compilers. Shall I continue my work or not interested?

manuelm commented 8 years ago

crashes in src/plugins/wresolver/wresolver.c because pk->filename is NULL

manuelm commented 8 years ago

pk is of type resolverHandles.user

I tried to take a look at the plugin but I fail to understand the for-loop in elektraWresolverOpen. The loop calls elektraWresolveFileName --> elektraResolve{Spec,Dir,User,System} which all malloc resolverHandle->filename and therefor leak memory.

markus2330 commented 8 years ago

Thanks for pointing that out! The code is obviously broken since the introduction in c87ae8e87a716b02b2c7ed790ad56a89d95547a9 During looping only and always the system handle was initialized. This lead to crashes when another namespace was used.

I fixed it in edb4d50856bb5331749220de5a83fa2062624a9d

About continuing work: On the one hand, it would be nice if the compiled stuff also runs. On the other hand, the release should happen this weekend, so a pull request would be important soon (there should be at least a chance of a short feedback circle, e.g. what the version-scripts actually does)

But imho its enough if only one variant (the static compilation) works. Great to see the kdb-tool running!

manuelm commented 8 years ago

Where can I find edb4d50856bb5331749220de5a83fa2062624a9d?

markus2330 commented 8 years ago

edb4d50856bb5331749220de5a83fa2062624a9d was pushed a bit later.

markus2330 commented 8 years ago

Which gcc versions are installed on debian-unstable-mm?

http://build.libelektra.org:8080/job/elektra-multiconfig-gcc-unstable/build_type=Release,gcc_version=5.2,plugins=ALL/56/console

says there is no gcc-5.2

Can you install as much compilers as possible, please?

manuelm commented 8 years ago

In some issue or PR I've said that I've removed all compilers except the latest. Edit: gcc 4.9 on stable, 4.9 + 5.x (default) on unstable

Please do these kind of tests (I find them highly unnecessary anyway) on your own containers. Mine won't stay forever anyway.

markus2330 commented 8 years ago

I have not read that. They have maybe 50MB each. Could you please install them again and answer the first question?

manuelm commented 8 years ago

Maybe I told you in our meeting. But I've definitely told you.

debian-unstable:~ # gcc -v 2>&1 | tail -n 1
gcc version 5.2.1 20150903 (Debian 5.2.1-16)

The version specific binary is called gcc-5. No separate package for minor versions anymore. So your multiconfig-gcc with this level of detail is kind of obsolete. I recommend removing gcc 4.7 and replacing gcc-5.2 with gcc-5 and be done.

The only additional compiler available I haven't installed is gcc-4.8. And gcc-4.8 has already been tagged for removal.

markus2330 commented 8 years ago

Thanks for the info! Seems like the glory days of many available compilers is over.

I fixed multiconfig-unstable.

markus2330 commented 8 years ago

I will close for now, thanks for the excellent agent setup.

markus2330 commented 8 years ago

Hello, jessie (stable) needs some more packages. Could you please install:

manuelm commented 8 years ago

fakeroot installed, gpg + repropro is already installed. Can you mail me your already existing gpg key? So both build machines have the same

markus2330 commented 8 years ago

Its ok to have different gpg keys. I am not sure if the current setup uses them at all, so first wait if http://build.libelektra.org:8080/job/elektra-git-buildpackage-jessie/2/ fails.

manuelm commented 8 years ago
markus2330 commented 8 years ago

Thanks for installation!

python-dev is available for Jessie, and python-support, too. Please install them.

I tested it locally, when these packages are installed, it builds for jessie.

manuelm commented 8 years ago

Sure, it's available but it's a wrong dependency. python-dev depends on python2.7-dev which is not sufficient. Instead python2.7-dev + python3-dev is required.

python-support isn't required at all imho.

markus2330 commented 8 years ago

I do not know why the dependencies were chosen this way, most of the packaging was done by @iandonnelly during gsoc.

markus2330 commented 8 years ago

Yes, the packages should be updated to build python3 bindings, too. Currently, its simply not done. Nevertheless, you can install python3-dev, so that the build won't break (when python3 bindings+plugins are added to the debian package).

manuelm commented 8 years ago

That doesn't mean they are correct :-) - I'm fairly sure about the python-dev deps. Can you please replace them and remove the python-support dep?

python3-dev and python2.7-dev is already installed. Otherwise no binding would build.

markus2330 commented 8 years ago

Btw. the official debian package from @pinotree builds python3-only. It would be a waste of time to fix whats in our "debian" branch, the work of @pinotree is superior anyway.

When I find time, I will update our "debian" branch to what @pinotree has done in the official package. He already allowed us to do so. I will wait for the qt-gui update, currently there is no hurry to change. And having python2 support would be important for one installation (where cheetah is used, which won't work with python3).

manuelm commented 8 years ago

I've never said I'll remove the python2 packages. All I'm saying is python-dev is an inaccurate dependency. We require explicit versions. So pythonX-dev is the correct dep to use.

Hopefully pinotree worked out the dependency correctly.

Btw, cheetah is dead. Don't use it.

markus2330 commented 8 years ago

Ok, replaced it. Please revert b7c266b36b0ab0fad9120e67a457b580c7c44690 and install python-support if it is needed after all.

I am sure pinotree did it correctly ;)

markus2330 commented 8 years ago

And it says: dpkg-checkbuilddeps: Unmet build dependencies: build-essential:native http://community.markus-raab.org:8080/job/elektra-git-buildpackage-jessie/3/console

manuelm commented 8 years ago

installed

pinotree commented 8 years ago

python-dev is a wrong dependency. it should be python2.7-dev or python3-dev or both

  • python-dev installs the development package for the default Python 2 version; since Wheezy, this is Python 2.7
  • python3-dev installs the development package for the default Python 3 version; Python 3.2 in Wheezy, 3.4 in Jessie, and so far still 3.4 in Stretch (I guess soon will be 3.5)

So, if you want to build against the default Python 2/3 version, use respectively python-dev/python3-dev, not the pythonX.Y-dev versions (which you need to use when you explicitly want a precise Python version installed, even if not the only one installed on the system, and not the default one). Using either is what I recommend.

manuelm commented 8 years ago

from python-dev description: This package is a dependency package, which depends on Debian's default Python version (currently v2.7).

According to this text python-dev can surely depend on python3 sometime soon