Docker fixes - Githubissues

cavalab / srbench

A living benchmark framework for symbolic regression

https://cavalab.org/srbench/

GNU General Public License v3.0

203 stars 74 forks source link

Docker fixes #66

Closed MilesCranmer closed 2 years ago

MilesCranmer commented 2 years ago

This PR is #59 but with a few extra changes:

Fixes the platform to linux/amd64 only, which enables this docker container to be built on any system whatsoever. (necessary since many of the conda packages are architecture-specific).
The dockerfile itself has been simplified, and the NVIDIA/GPU-specific parts have been removed. I think those should be up to the user to configure in a Dockerfile which depends on this one, since many systems do not have a GPU, and NVIDIA files are different by system.
Builds with micromamba instead of conda, which significantly reduces build time.
Also includes conda so that the install.sh files can proceed without modification. mamba and conda use the same structure so they can both be used.
Adds a GitHub action to test the building of the container.

The CI right now just tests if the container is buildable, but does not run the tests within the docker yet. Those can be added later.

MilesCranmer commented 2 years ago

Can also use caching of the docker image:

https://dev.to/dtinth/caching-docker-builds-in-github-actions-which-approach-is-the-fastest-a-research-18ei

MilesCranmer commented 2 years ago

@foolnotion I also merged the commits from your PR on @yoshitomo-matsubara's repository. It looks like your changes cavalab/srbench:master supersede those changes? But let me know if anything looks off here.

MilesCranmer commented 2 years ago

Actually maybe the NVIDIA parts should be left in. @yoshitomo-matsubara do you know if they affect the build for systems not using GPUs?

yoshitomo-matsubara commented 2 years ago

@MilesCranmer I think I tested the original docker file with the NVIDIA parts on a local machine (with no GPUs), so it won't affect the build for those with no GPUs

MilesCranmer commented 2 years ago

No but it looks like changing bash $install_file to $install_file in install.sh didn't work as intended and results in the above error. I was trying to keep the same shell env when running the install files since it complains it can't find conda otherwise.

MilesCranmer commented 2 years ago

Argh, I really can't seem get this working after many hours. The conda install runs fine, but after this, the docker container to forget that I've run conda init bash and keeps complaining about this. @yoshitomo-matsubara or @foolnotion want to have a stab?

lacava commented 2 years ago

Argh, I really can't seem get this working after many hours. The conda install runs fine, but after this, the docker container to forget that I've run conda init bash and keeps complaining about this. @yoshitomo-matsubara or @foolnotion want to have a stab?

I should be able to make a pass in the next couple weeks...

foolnotion commented 2 years ago

Next week I will have a look as well.

lacava commented 2 years ago

The conda install runs fine, but after this, the docker container to forget that I've run conda init bash and keeps complaining about this.

The fix seems to be to specify the environment as a precursor to SHELL as

SHELL ["conda", "run", "-n", "srbench", "/bin/bash", "-c"]

I'm able to build the Docker Image fine, but I've moved on to attempting to use the image (and cache it) for the tests on the cavalab:docker-fix branch.

lacava commented 2 years ago

@MilesCranmer can you check the PySR build on docker here: https://github.com/cavalab/srbench/runs/5254753478?check_suite_focus=true#step:6:1766

I've removed the conda dependency everywhere and parallized the docker build; it looks like PySR / Julia is trying to install or use conda in some way.

lacava commented 2 years ago

moved to PR #75