rethinkdb / rethinkdb_rebirth

The open-source database for the realtime web.
https://rethinkdb.com
Other
999 stars 42 forks source link

Fix missing packaging dependency and add support for Ubuntu Bionic, Debian Stretch, Jessie and Centos 7 #25

Closed floydkots closed 6 years ago

floydkots commented 6 years ago

Description

  1. Added fakeroot to the list of packages installed in the docker images since it stopped being installed automatically after we added --no-install-recommends to the Dockerfiles.
  2. Added support for Ubuntu Bionic Beaver, Debian Stretch, Jessie and Centos 7.
  3. Updated the README to reflect the above changes.
  4. Fixed the coveralls code coverage reporting. Refactored .travis.yml to use the COVERAGE target in mk/tools.mk to compile and link the sources with the respective flags, and then used cpp-coveralls to upload the results from the generated coverage.info file to coveralls.io.

Known Issues

  1. Tests are failing intermittently due to network issues being experienced by the travis VMs. Seems like a recurrence of the incident that was reported on Friday the 10th of August.
  2. If job stages 1 and 2 run successfully, stage 3 (Packaging and deployment) will fail because we are yet to create the keys, apt and rpm repositories under our rebirthdb organization on bintray.

With the above two resolved, everything should work as expected.

asakatida commented 6 years ago

Codacy is just making noise here. That issue type should probably be turned off.

coveralls commented 6 years ago

Pull Request Test Coverage Report for Build 85


Files with Coverage Reduction New Missed Lines %
src/arch/runtime/thread_pool.hpp 1 93.33%
src/clustering/generic/raft_network.hpp 1 80.0%
src/clustering/administration/auth/scram_authenticator.hpp 1 0.0%
src/clustering/administration/real_reql_cluster_interface.hpp 1 0.0%
src/clustering/administration/artificial_reql_cluster_interface.hpp 1 50.0%
src/rdb_protocol/datum_stream/slice.hpp 1 0.0%
src/crypto/hmac.hpp 1 0.0%
src/containers/archive/varint.hpp 1 94.12%
src/clustering/administration/tables/table_common.hpp 1 0.0%
src/concurrency/coro_pool.hpp 1 97.3%
<!-- Total: 32490 -->
Totals Coverage Status
Change from base Build 78: 51.5%
Covered Lines: 34828
Relevant Lines: 67617

💛 - Coveralls
floydkots commented 6 years ago

Finally the tests passed! Thank you @gabor-boros ^_^

asakatida commented 6 years ago

I've been over this a few times and it looks good. Waiting to give others some time to review. I do find it unusual that the actual bin is built external to the Docker process.

floydkots commented 6 years ago

... I do find it unusual that the actual bin is built external to the Docker process.

I didn't quite get this. Would you explain it some more?

asakatida commented 6 years ago

I didn't quite get this. Would you explain it some more?

Certainly. If I'm reading this right then a plain docker build . will create an image without the rebirthdb bin only the working tree is copied. Looking a the TravisCI config the build sends the commands to configure and make a bin. Then the image is running bash by default instead of rebirthdb with exposed ports. I could have missed something and BinTray is handling images in this style to create a deliverable. It does still add a config file I need to reference to build docker locally.

floydkots commented 6 years ago

You are correct. Travis builds the rebirthdb-$DISTRIBUTION-base images with the required system packages and the source's working tree majorly:

  1. For portability between the build stages.
  2. To make it easy to troubleshoot the builds by providing us with a reproducible environment in the docker images. If something is failing, say the Centos7 build, and the travis logs aren't sufficient, I can simply pull it's base docker image and build locally on my Ubuntu or Mac machine, to see why it's failing.

We then use these images to run containers that build the rebirthdb binaries for testing (stage 2), and then for packaging (stage 3). Thereafter, we send the built rebirthdb packages (containing the binaries) from (stage 3) to bintray, for distribution.

We stop short of building images that purely contain rebirthdb installed and executable. Thought we could use the rebirthdb-dockerfiles repo for that. The docker files would simply grab the distribution specific package from bintray and install it, then set the entry point to run rebirthdb --bind all with the proper ports exposed.

If I recall correctly, @norpan was waiting for the packages to be available before he could go ahead and update the docker files in that repo.

floydkots commented 6 years ago

An alternative, would be to add a fourth stage to build and deploy the docker images with rebirthdb installed, but I don't know, wouldn't that make the travis build process too lengthy? So I thought, we could instead let docker automatically build the images off the docker files in the rebirthdb-dockerfiles repo.

asakatida commented 6 years ago

That all makes sense. I approve stoping short of a release build in Travis.

floydkots commented 6 years ago

Speaking of releases, we have two bintray packages (different from rebirthdb packages) in the bintray apt and rpm repositories, rebirthdb-dev and rebirthdb. Every triggered build (with the exception of builds triggered by Pull Requests) on travis, if successful, will deploy a rebirthdb package to rebirthdb-dev but only builds triggered by tagged commits, which would be Github releases and pre-releases, will deploy a rebirthdb package to the bintray rebirthdb package.

I feel this should be improved by the time we get to the major 2.4 release, so that only releases, and not pre-releases, get to the rebirthdb package in bintray. But am happy with it this way for the next 1 or 2 pre-releases (v2.4.0-beta.x), which I think @atris will make to allow for further testing (performance, integration).

Release builds to docker however, remain a separate process.