search

ElektraInitiative / libelektra

Elektra serves as a universal and secure framework to access configuration settings in a global, hierarchical key database.
https://www.libelektra.org
BSD 3-Clause "New" or "Revised" License
207 stars 123 forks source link

Build Server stuff #160

Closed markus2330 closed 2 months ago

markus2330 commented 9 years ago

This issue gives up-to-date information about the health of our build system.

Report here any permanent problems (that cannot be fixed by rerunning the build job). Temporary problems should be reported in #2967.

Current open issues for our Jenkins build system:

Fixed issues:

manuelm commented 8 years ago

Further more: There never will be another python2 version. So python2.7-dev will be the last python2 dev package ever.

Depending on python3-dev is what I said.

markus2330 commented 8 years ago

Now only the key is missing:

gpg: new configuration file `/home/jenkins/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/jenkins/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/jenkins/.gnupg/secring.gpg' created
gpg: keyring `/home/jenkins/.gnupg/pubring.gpg' created
gpg: skipped "Autobuilder <autobuilder@libelektra.org>": secret key not available
gpg: /tmp/debsign.DlSdnFtB/elektra_0.8.13-1.41.dsc: clearsign failed: secret key not available
debsign: gpg error occurred!  Aborting....
manuelm commented 8 years ago 
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/08C91995 2015-09-30
      Key fingerprint = BA4C 688E 9071 FD3F 57ED  E9D6 D0A9 EDB9 08C9 1995
uid                  Autobuilder <autobuilder@libelektra.org>
sub   2048R/E69F110A 2015-09-30

done

markus2330 commented 8 years ago

Thank you!

Please export /home/jenkins/repository via http.

manuelm commented 8 years ago

cannot access /home/jenkins/repository: No such file or directory ?

manuelm commented 8 years ago

→ http://194.117.254.29/elektra-stable/

markus2330 commented 8 years ago

@manuelm Could you please install ronn on the agents? (needed for generating man pages)

apt-get install ruby-ronn

manuelm commented 8 years ago

done

markus2330 commented 8 years ago

Thanks, jessie packages build again, and man pages are now included!

markus2330 commented 8 years ago

Please install musl, i.e.

apt-get install musl musl-dev musl-tools

Thank you!

markus2330 commented 8 years ago

musl installed and agends upgraded

markus2330 commented 8 years ago

Two important things about the build server:

  1. Do not create new empty jobs, but rather duplicate, they have correct settings (except whats mentioned in number 2.).
  2. We should use reference clones (in /home/jenkins/libelektra) or prefer shallow clones for every build-job (currently done only for some, e.g. elektra-clang). Currently the traffic is >300MB on commits because of the many unnecessary reclones.

@mpranj It would be great if you can fix 2.

mpranj commented 8 years ago

@markus2330 just to make sure: I should just apply the same clone behaviour to all build jobs like it is in elektra-clang?

mpranj commented 8 years ago

Shallow clones applied to all build-jobs except:

These jobs check out to some sub-directory. Wasn't sure what you want there so I'll leave them as they are for now.

markus2330 commented 8 years ago

Thank you! Yes they need complete history and branches, shallow clones make no sense but the reference clone repository would be useful.

markus2330 commented 8 years ago

Jenkins was updated to 1.651.2. Also all plugins were updated.

I will keep the issue open for the reference clone repos. We should also have "cron jobs" which update the repos from time to time, ideally using jenkins itself.

mpranj commented 8 years ago

Jenkins stopped building some jobs (since the update apparently). It fails with ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

markus2330 commented 8 years ago

Thanks for the info. I try to downgrade github request builder from 1.31 to 1.14.

mpranj commented 8 years ago

Now it seems stuck when setting the build status for the Github commit. It does warn that this is deprecated in the config.

markus2330 commented 8 years ago

I also tried to downgrade every plugin with *git* in its name, but then there were still errors (strange error related to Mailer Plugin, downgrade of Mailer Plugin did not help). So I updated everything to recent versions again. The problem seems to be a known issue upstream:

https://github.com/janinko/ghprb/issues/347

I hope they will fix it soon.

Another question: Does someone know how to run multiple jobs for every PR? (I would like to run both elektra-mergerequests-stable and elektra-mergerequests-unstable)

mpranj commented 8 years ago

The elektra-test-bindings job is working fine with parameterized builds (as also described in the upstream ticket). Couldn't we just switch it to parameterized builds? The bug has been reported upstream for a while, I don't see it fixed soon.

markus2330 commented 8 years ago

Good idea, we could change all PR jobs to parameterized builds, it actually has only advantages. It allows us to run the jobs manually by specifing a branch, too. And it also can be used for regular build jobs.

Ideally, every job could be executed by github PRs, too. (Except those specifically for non-PR tasks that update docu or coverage of the master branch)

A disadvantage of elektra-test-bindings' config is that it only does polling and takes quite long until it starts building (up to 5 min). I do not want to activate "Use github hooks for build triggering", however, to not break the build job.

Btw. are you sure that the "shallow clone" option is okay for the github pullrequest builder jobs?

I wonder how github picks the build job it uses for new PRs. Why is the elektra-test-bindings and elektra-ini-mergerequests never selected for a new PR? Why is it sometimes elektra-mergerequests-unstable and somtimes elektra-mergerequests(-stable)?

@manuelm do you have any idea?

markus2330 commented 8 years ago

Btw. somehow the communication of finished build jobs and github is severely impaired (even for elektra-test-bindings). It now says on nearly every build "Some checks haven’t completed yet".

manuelm commented 8 years ago

A disadvantage of elektra-test-bindings' config is that it only does polling and takes quite long until it starts building (up to 5 min).

And this is a problem because? Testing takes more than 5 minutes anyway.

Why is the elektra-test-bindings and elektra-ini-mergerequests never selected for a new PR?

Why should it? elektra-test-bindings gets triggered by the "Trigger phrase" only. No idea what elektra-ini-mergerequests is.

Why is it sometimes elektra-mergerequests-unstable and somtimes elektra-mergerequests-stable?

The -stable/-unstable are new? I'm not sure triggering multiple jobs per new PR is possible. I would do subjobs.

Btw I've said this a few times already but I think the amount of jobs is getting ridiculous and a sign of a messed up config. But criticizing is always easier than solving something.

markus2330 commented 8 years ago

The 5 min is a problem when you want to debug the build server. And I still hope that we get a quick first test sometime, taking about 5 min.

Ahh, ok, I missed the option "Only use trigger phrase for build triggering". The config for the github request builder is really a mess.

Someone talked about github projects where they have multiple jobs running for every PR. (Displayed individually)

What is a subjob? Do you mean multijob?

manuelm commented 8 years ago

Someone talked about github projects where they have multiple jobs running for every PR. (Displayed individually)

You'll have to add two services on github.

What is a subjob? Do you mean multijob?

Yeah multijob.

manuelm commented 8 years ago

btw, what about https://docs.travis-ci.com/ ? Travis has support for OSX.

I know it won't replace jenkins but might replace the PR/on every commit builds. Jenkins can still do the multiple compiler/etc.. testing. Edit: Travis even has gcc + clang.

mpranj commented 8 years ago

Agreed, it would be interesting to use their CPU power/electricity for free as elektra is open source.

markus2330 commented 8 years ago

It is likely that the connection between github and jenkins actually is 1:1. In the github service I entered http://build.libelektra.org:8080/github-webhook/ and I did not find a way to create another URL in jenkins. (I only found a way how to specify a override, but this did not create a new URL).

In https://github.com/janinko/ghprb/issues/142 they discuss that it should "just work"? (Without adding multiple services)

The sha1 problem, however, should be solved now. It was broken because Jenkins introduced a new security measurement which prunes unknown environment variables. I fixed it as suggested (added -Dhudson.model.ParametersAction.safeParameters=ghprbActualCommit,ghprbActualCommitAuthor,ghprbActualCommitAuthorEmail,ghprbAuthorRepoGitUrl,ghprbCommentBody,ghprbCredentialsId,ghprbGhRepository,ghprbPullAuthorEmail,ghprbPullAuthorLogin,ghprbPullAuthorLoginMention,ghprbPullDescription,ghprbPullId,ghprbPullLink,ghprbPullLongDescription,ghprbPullTitle,ghprbSourceBranch,ghprbTargetBranch,ghprbTriggerAuthor,ghprbTriggerAuthorEmail,ghprbTriggerAuthorLogin,ghprbTriggerAuthorLoginMention,GIT_BRANCH,sha1 to /etc/default/jenkins).

About usage of additional build servers: Yes, go ahead. It also solves the issue of multiple build jobs for a single PR ;) I never used travis-ci, so cannot say anything about it. I gave the permission that travis has access to ElektraInitiative.

markus2330 commented 8 years ago

First travis build: https://travis-ci.org/ElektraInitiative/libelektra/builds/130425147 I think we need some yaml file so that travis knows what to do.

And I figured out how to do multiple jenkins jobs per PR, a different context for every build-job was needed. In the next meeting we discuss what the "fast" and other build jobs should do.

manuelm commented 8 years ago

I'm working on travis (or checking some things out)

markus2330 commented 8 years ago

Have fun. Travis also added a github service, so I guess that a PR will be built with travis, too.

manuelm commented 8 years ago

I'm already swearing loudly

manuelm commented 8 years ago

-- Could NOT find JNI (missing: JAVA_AWT_LIBRARY JAVA_JVM_LIBRARY JAVA_INCLUDE_PATH JAVA_INCLUDE_PATH2 JAVA_AWT_INCLUDE_PATH) -- Exclude Plugin jni because jni not found

I'm unable to get the java plugin configure correctly. However Java bindings work. On debian unstable. Any ideas? Looking at the cmake module doesn't help much.

Edit: /usr/lib/jvm/java-8-openjdk-amd64/include/linux/jni_md.h, /usr/lib/jvm/java-8-openjdk-amd64/include/jawt.h and /usr/lib/jvm/java-8-openjdk-amd64/include/jni.h is in place

Edit 2: Got it. JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/ ....

manuelm commented 8 years ago

https://travis-ci.org/manuelm/libelektra/builds/130638376

Debian unstable built within a docker container. But building takes ages. Any good ideas?

rautesamtr commented 8 years ago

clang is often faster regarding build time, but i think the installation of the dependencies is what takes a large amount of time

rautesamtr commented 8 years ago

Isn't there a more minimal debian docker image then the one used? It seems a lot of packages get installed that should not be needed.

manuelm commented 8 years ago

@sirblackheart What makes you think that? https://github.com/manuelm/libelektra/blob/a3d4b9212e738f9b8f2c13318b8a661522dff4a6/scripts/docker/Dockerfile.debian_unstable#L6

rautesamtr commented 8 years ago

@manuelm probably the dist-upgrade. a lot of packages get updated that are desktop specific like wayland

manuelm commented 8 years ago

No. dist upgrade is short. maybe a minute. about 50% of the time is taken by installing the build deps.

manuelm commented 8 years ago

I'm pushing the build image to hub.docker.com right now. Hope that will speed things up. But the image has 1.9gb

manuelm commented 8 years ago

Elapsed time 14 min 8 sec

Not sure if we can do much better

rautesamtr commented 8 years ago

like i said, clang maybe gets us 2-3 mins. at least it does for the aseprite project https://travis-ci.org/aseprite/aseprite

markus2330 commented 8 years ago

It would be useful to have both compilers anyway.

manuelm commented 8 years ago

Just had an idea while preparing work stuff: What if we extract the paths of all commits in the push request and build bindings/plugins only if they are affected? e.g.

We still have the daily/twice a day full build on jenkins.

markus2330 commented 8 years ago

@manuelm good idea, @tom-wa would write such a script, can you create a new issue for this?

markus2330 commented 8 years ago

@mpranj: Reminder: add Mac OS X builds to travis and add mingw builds to PR. (*BSD seems to be more effort)

rautesamtr commented 8 years ago

@markus2330 now i understand @manuelm docker approach, travis will not support ubuntu 16.04 till next year so docker is needed to get all dependencies that ubuntu 14.04 does not have = swig3.0 libsystemd-devel.

manuelm commented 8 years ago

I'm sorry I couldn't attend the meeting today. At work we're still preparing a big software rollout today, so I can't leave the office. But within a short delay I can answer e-mails.

I've started to add OS X builds for travis 2 days ago: https://github.com/manuelm/libelektra/blob/e41ac43a18e5e9f9640a4042a313cc43f2704f65/.travis.yml Build is here: https://travis-ci.org/manuelm/libelektra/builds/130898079 Open things here:

I'm happy if anyone will assume my work from here. I don't have OS X and waiting for travis to inspect the OSX system sums up very fast.

re: docker: Yeah, travis default ubuntu version doesn't work well. Even cmake is too old. Fetching the uploaded docker image takes only about 3mins. And adding more images is a no brainer. So I think that's a nice way to workaround any pitfalls the default travis linux environment has (or might have after an update).

I haven't figured out a good way to integrate the different build and test phases of libelektra (cmake + make + make test) with docker (build + run) + travis (before_install, before_script, script). Docker containers exit after the command completes. Since docker containers are meant to be throw-away, you cannot resume afterwards. So your disk/compile state is vanished, unless you mount a local directory into the container. Will continue to work on docker next week.

markus2330 commented 8 years ago

@manuelm Great, you got further then we thought. Mac OS X for PRs and per commit would be really great. A lot of people are using Mac OS X now and I do not want to break the build for them again and again. In the meeting today @mpranj said he would pick up your work. Do you want to create a PR with the travis file?