Remove the need of docker credentials on Travis CI

[sbellem]

• Move base dependencies into separate docker image

• Resolves #387

  Main points:

• Dependencies that do not rely on the code base are packed into one docker image that will be maintained in an external repository.
• For the time being this parent Dockerfile (for dependencies) is under docker-honeybadgermpc-deps on Github and the image is on Docker Hub at honeybadgermpc-deps.
• Either before or right after this PR is merged the Dockerfile and image should be moved under initc3 or dsluiuc.

Some of the dependencies for the external honeybadgermpc-deps image, have been packaged into their own image:

• FLINT
• NTL
• PBC
• Charm-crypto
• Rust nightly frozen

Some positive side effects:

• The problem with FLINT (https://github.com/wbhart/flint2/issues/217 / https://github.com/wbhart/flint2/issues/512) is resolved, as we can use the latest code that is on Github.
• The problem with building pairing (#400) is taken care of by providing a frozen nightly build.
• The problem with Charm mentioned in #209, appears to be resolved with a more recent version of Charm. The image uses the latest code of Charm and the code could build and tests pass.
• Precision: The proposed approach should allow for much control over dependencies with respect to which versions are being used. This may help with the "reproducible" research concern (#100).
• Speed: Once a required specific version of a dependency is packed into an image it is not needed to re-build it as one simply copies the necessary files (e.g.: .h and .so) from the pre-built image. This can be done in the image used for development.

TODO:

• [x] Explain the proposed workflow.
• [x] Explain how one would go about to experiment with a new dependency and eventually add the new dependency.
• [x] Explain how one would upgrade or downgrade a dependency.
• [x] Use initc3 or dsluiuc as base org for docker base image honeybadgermpc-deps. I have used my account for the time being.
• [x] Document how to work with the images.
• [x] (pip) Install python dependencies for tests only for the unit test job, and docs only for the documentation job. Currently both tests and docs are installed for both jobs.

Will remove the docker_*.sh scripts in a subsequent step as they may be useful for build jobs done under the initc3/HoneyBadgerMPC repo, i.e.: TRAVIS_REPO_SLUG == initc3/HoneyBadgerMPC.

[codecov[bot]]

Codecov Report

Merging #401 into dev will increase coverage by 0.03652%. The diff coverage is n/a.

@@                Coverage Diff                 @@
##                 dev       #401         +/-   ##
==================================================
+ Coverage   77.19137%   77.2279%   +0.03651%     
==================================================
  Files             49         49                 
  Lines           5476       5476                 
  Branches         846        846                 
==================================================
+ Hits            4227       4229          +2     
+ Misses          1078       1077          -1     
+ Partials         171        170          -1

[sanket1729]

Hi @sbellem, This looks super clean and thanks for the prompt work. We would like to merge this as soon as possible.

1) Can we still use Drake's worksflow for the second repo? That is, we should auto push to docker hub whenever the base image is updated. 2) Can you also remove the scripts docker_*.sh inside the scripts folder as part of this PR. 3) I suppose this will require an edit in the README for install instructions too.

[sbellem]

Can we still use Drake's workflow for the second repo? That is, we should auto push to docker hub whenever the base image is updated.

Docker Hub allows for automated builds to be configured such that any change on the linked Github repo will trigger a new build of the image on Docker Hub.

I will look into the optimal way to set this up for the team. It is also possible to set automated tests via what Docker calls Autotest and I also want to look into this.

[sanket1729]

I think before we merge this, we should at least update the install instructions

[sbellem]

I think before we merge this, we should at least update the install instructions

I'll be adding documentation early on Tuesday, now that the images have been moved under the initc3 org on both Github and Docker Hub.

[sbellem]

Will we have to manually change the digest HBMPC_DEPS_DIGEST every time there is a new build image.

Yes, every time we wish to use a different version of the parent image.

If yes, can we look into some automated way of doing this? It is not a priority thing, but just "good to have" thing

I would need to look into it. Perhaps with a custom build phase hook after the push (hooks/post_push).

If we change the digest the change should go through a CI phase, like on Travis CI, or perhaps using Docker Hub Autotest.

An alternative would be to use tags instead of digests. Something like:

FROM initc3/honeybadgermpc-deps:python3.7-slim-buster

instead of

FROM initc3/honeybadgermpc-deps@sha256:46902d869ea881d7b00b72ff6accf2558a5e15849da5fa5cc722b4ff82a507f8

Tags are mutable whereas digests are immutable. In other words, when you are pulling an image by its tag that image may change from one day to another whereas using the digest provides a guarantee that this image will not change.

So, if initc3/honeybadgermpc-deps:python3.7-slim-buster has a digest equal to 46902d869ea881d7b00b72ff6accf2558a5e15849da5fa5cc722b4ff82a507f8 right now, and tomorrow initc3/honeybadgermpc-deps:python3.7-slim-buster is updated its digest will be different, so one could get the latest image by using the tag.

The advantage with using the image digest is that it helps with reproducible results, such as benchmarks and tests.

You may find this blog post interesting to read.