Installation and integration tests with Docker

[jlapeyre]

@garrison Do you have ideas about how to do CI or local test scripts using Docker ? Do you make a bunch of containers and run them in a shell script? (I am trying this here.) How do you check output? Simplest would be to look at return code from building the container. Maybe running a container can print to stdout, and you collect that ?

Here are some Dockerfiles where I am moving in this direction.

• Should these live in the qiskit_alt repo ? For local testing, at least, this seems like a good idea. I don't know if they should be integrated with pytest or not. I guess not
• Installation and initialization has been a big dev chore. I find it easy to miss corner cases and to introduce regressions. So, I'd like to test different ways to install in different environments. Doing too many combinations would be costly. However, you can set up images with most of the heavy work done.
  • For example, I want to avoid using a linux-distribution installation of Julia; they are buggy. In one image, we can have it installed and test that it is skipped, even if it is returned by "which". A light-weight solution is the just put a bogus script in /usr/bin/julia.
  • Also, we need to skip the julia executable used by juliaup. It is actually a wrapper around julia, that caused a couple of bugs if used with qiskit_alt. So I need to exclude it and use the paths to the julia binaries that are installed by juliaup. It is, all but impossible to install juliaup in a docker container (I opened an issue). But, we could very easily mock ~/.juliaup/bin/julia.
  • The most common way to install will be in a user's home directory. So I am concentrating on this first. However, a systemwide or root install would be good if we have enough human cycles to spare.
  • I expect each of the possible ways to do a user install will see a fair fraction of attempts.
    • Installing in the user's directory (eg ~/.local) just by using pip, but with no virtual environment
    • Installing using python -m venv ./venv.
    • Installing using conda.
• Conda
  • It is a bit tricky to install a conda environment in a docker container, but stuff I found online works ok.
  • By default, conda uses a statically linked python executable, which does not work with pyjulia. qiskit_alt detects this and does a warnings.warn and switches to juliacall. It would be nice to test this.
  • There is a way to install a dynamically linked python within conda and via conda and use it instead. But, for lack of time I stopped trying to get that to work. I'd say that's low priority, unless we find pyjulia is indispensible.

The containers so far are following these ideas for containers as layers (I think "layers" is not the correct term here)

• Lowest layer
  • required fedora packages: gcc, python, etc.
  • Create a user "quser" with /home/quser.
  • Install one virtual enviroment with venv /home/quser/venv.
  • Install all python dependencies that are not part of the broad qiskit_alt project. This includes installing qiskit-terra, numpy etc.
• First layers. Install qiskit_alt. Use pip install -e . Other layers that install conda instead
• On top of these layers, have containers that do different calls to qiskit_alt.project.ensure_init(calljulia="pyjulia", depot=False, install_julia=True), etc. Currently, these layers do an automatically installation of julia. But, this is costly to do every time. So...
• We can do RUN source ./venv/bin/activate && echo "y" | jill install to install julia in a layer.

[jlapeyre]

I have added modified Dockerfiles to the main branch, and to a release ae78f22 . Currently you have to run a script to build the docker images and check the log files for errors (usually pretty obvious). But, I think this will help quite a bit controlling regressions.