Closed dexX7 closed 9 years ago
msgilligan
commented
9 years ago I'm glad you're doing this, @dexX7. I need to look at Travis CI and compare to using Jenkins. I'd like to see if I can at least get the Spock RegTest tests running on Travis.
dexX7
commented
9 years ago @msgilligan: I have basically no practical experience with either one of both, but it seems as if Jenkins has a lot more support for plugins, customization etc., and - this can be an advantage or disadvantage - it's no hosted solution.
I basically just played with it to get a feeling for, but since http://ci.mastercoin.info/ is already up: is there a specific reason why you'd consider a switch?
It would be nice, if PRs are auto build and tested. In this context probably worth to mention: just a few days ago GitHub extended it's PR status check system, allowing more than one CI system, and there is also has an API to integrate "homebaked" services:
https://github.com/blog/1935-see-results-from-all-pull-request-status-checks
dexX7
commented
9 years ago Refined the doxydocs a bit and it includes now also cross-references. Call and caller graphs were not removed, but can be extended by click:
Before integrating doc building into automated building, it's probably easier to begin with a simple script like:
I also added you guys to http://scan.coverity.com/projects/3764 with all permissions, so feel free to play with. We have 2-4 builds per week for free and this really seems powerful, but equally complex.
dexX7
commented
9 years ago Over the last days I also tested Circle CI and I have to say, it's really nice, most notably because it's build output looks very clean: https://circleci.com/gh/dexX7/mastercore/14
However, and I'm waiting for a response from the support team, there seems to be an issue with new pull requests, which have no "circle.yml" file, while the branch-to-be-merged-into does. This results in the inability of Circle to build and test a PR properly and as consequence, a failing build status is reported, e.g.: https://github.com/dexX7/mastercore/pull/2.
I was furthermore able to integrate the "state of art" RPC regtests from https://github.com/TheBlueMatt/test-scripts/ and started to explore test coverage with gcov/lcov, which is pretty time consuming (nearly 44 min build time). I did not yet run this with the Master Core RPC tests, but as intermediate result, based on the standard tests and the external Core RPC tests: https://circle-artifacts.com/gh/dexX7/mastercore/11/artifacts/0/home/ubuntu/mastercore/total.coverage/home/ubuntu/mastercore/src/index.html
At this point I recommend to adopt Travis CI testing, which can very well run parallel to the Jenkins tests.
msgilligan
commented
9 years ago Hi @dexX7, sorry for not responding earlier. I want to learn Travis CI for educational purposes, but I know I'll probably be using it on one or more projects -- including Omni. At the very least I need to better understand the pros and cons of Travis vs. Jenkins vs. others.
I'm submitting some patches and tests to bitcoinj and would like to submit some of Spock Bitcoin RPC tests upstream. I spoke briefly with Matt Corallo about it in SF last month and he was encouraging.
Note that Jenkins does have the ability to test PRs on Github. To that requires an install/configure of a security plugin and it probably also requires some kind of "containerization" because running code "injecting" through a pull request seems dangerous. I'm pretty sure Travis is now using Docker for containerization and Jenkins can be configured to do the same.
msgilligan
commented
9 years ago I would also like to run some of these other tests on our Jenkins server. It shouldn't be hard. Let me know if you want an admin account for the Jenkins server.
Note that CloudBees (Jenkins commercial SASS and services provider) has a FOSS program that I'm certain Omni Foundation could be accepted into and they also offer a lower-end free service called BuildHive
Since we already have a Jenkins server those service may be of limited benefit, but on the other hand it's nice to have options for people who are staring new related/forked project and want to use similar technology.
dexX7
commented
9 years ago Since we already have a Jenkins server those service may be of limited benefit, ...
That's true, however I see no downside in adopting several systems. Setting up Travis or Circle as project maintainer takes only a few clicks, if build files .travis.yml and circle.yml are provided, and I submitted https://github.com/mastercoin-MSC/mastercore/pull/269 to get the ball rolling.
Special terms for open source projects are indeed appealing, and there are many service providers offering free products, whether it's Travis, Circle or JetBrains.
msgilligan
commented
9 years ago Since we already have a Jenkins server those service may be of limited benefit, ...
I was referring to the free, hosted Jenkins services being of limited benefit to the Omni project, that's why I said "it's nice to have options for people who are staring new related/forked project and want to use similar technology [to our Jenkins server]".
Sorry for being unclear.
dexX7
commented
9 years ago We now have Travis CI enabled over at https://github.com/OmniLayer/omnicore.
However, I started also to play around with Circle CI (again).
The new configuration has bitcoind x64 for Ubuntu as target, and builds the dependencies from the ground up (similar to Travis, but cached!), and then runs the OmniJ's RPC tests:
It is also possible to collect build artifacts:
Updating the status on GitHub is possible, also in combination with Travis.
On the one hand I think we should adopt this, too, because Omni Core RPC tests are extremely important, on the other hand, this may be create a distraction: instead of focusing on getting things done, one may be temped to focus on getting a green status report, which is only relevant at the very end.
Especially @zathras-crypto, since you're less familiar with the spock tests: what do you think? Would that help you?
@msgilligan: what is your opinion on this? Your experience with software testing, and likely with CI systems in general, is superior than mine. What do you recommend?
dexX7
commented
9 years ago Travis CI is used for a while over at OmniLayer/omnicore, and we run OmniLayer/OmniJ RPC tests in the 64-bit Ubuntu build. See:
We started to leverage caching via the Docker based infrastructure recently.
1) I generated a Doxygen documentation for Master Core, based on a recent commit: http://bitwatch.co/doxygen/mastercore/20141221_bbd64c3f6f3ecce814c9c1d0929a645700e30c7f/
Most valuable are the call and caller graphs imho. I'm looking forward to keep an updated version of this in some automated fashion.
2) Still far away from compiling and testing automated cross-plattform building, but it turned out https://travis-ci.org/ is actually easier to use than I expected. I tested it with some private repos and it's also possible to append RPC tests etc., but as bare minimum all that's needed, is to include such a file in the root dir and linking a GitHub account to it. It's free for open source projects and you get 100 builds or so as test for private repos: https://github.com/dexX7/mastercore/blob/travis-mscore-0.0.9/.travis.yml
It is automated and can be initiated when there is a new PR or commit. Output looks like this: https://travis-ci.org/dexX7/mastercore/builds/44767498
This can very likely be combined with generating Doxygen files.
3) Bitcoin Core v0.10 appears to be able to build for all plattforms with ease and the release seems to be announced rather sooner than later. Are there already migration plans? It looks like there would be merge conflicts at this point, but that's not unexpected. But since Master Core is pretty isolated from the rest, resolving them is probably feasible.
Just wanted you to know, and it's likely I'll update this topic over the next days, if I find something interesting.