ClearML build using docker

sillsdev / silnlp

A set of pipelines for performing experiments on various NLP tasks with a focus on resource-poor/minority languages.

Other

30 stars 3 forks source link

ClearML build using docker #7

Closed johnml1135 closed 3 years ago

johnml1135 commented 3 years ago

Should we use docker for the ClearML jobs? More generally, would it be helpful to use docker for development? We could then host it in other locations and remote in using VSC...

Some references for poetry and docker working together: https://github.com/python-poetry/poetry/discussions/1879 https://stackoverflow.com/questions/53835198/integrating-python-poetry-with-docker

johnml1135 commented 3 years ago

Or, use poetry virtualenv - ClearML needs to be one or the other. Would both together be overkill?

johnml1135 commented 3 years ago

Step1: cpu only - using:

FROM mcr.microsoft.com/dotnet/runtime-deps:5.0.5-focal-amd64
RUN apt-get update -y && apt-get install python3.8 -y

Next, Push that to docker hub. This is our image. Use that image for running our code. Next. Update the code to run dotnet tool restore before everything - to always pull the right stuff. ClearML should see the poetry file and pull everything automatically (hopefully).

johnml1135 commented 3 years ago

Steps to done: (cpu-only)

Convert Path objects to S3Path
Run silnlp.common.extract_corpora (uses machine)
copy configuration from another experiment, edit.
Run silnlp.nmt.preprocess (uses machine) (gpu)
edit configuration - maxsteps = 1000
Run silnlp.nml.train (uses gpu) (logging results to compare)
Reroute csv blue scores to ClearML in sil.nmt.test

johnml1135 commented 3 years ago

What to do about:

meteor_path = Path(os.getenv("METEOR_PATH", "."), "meteor-1.5.jar")
plugin_file_path=Path(os.getenv("BETA_INV_PLUGIN_PATH", ".")),
mgiza_path = Path(os.getenv("MGIZA_PATH", "."), "mgiza")
fast_align_path = Path(os.getenv("FAST_ALIGN_PATH", "."), "fast_align")

Are these extra things that should be installed into the docker image?

johnml1135 commented 3 years ago

Scrap everything and start again - machine needs paths. We need to mount it.

I looked for a solution and documented what I found here: https://stackoverflow.com/questions/67496760/mounting-an-s3-bucket-in-docker-in-a-clearml-agent/67497264#67497264

Only a small number of changes were needed - but I also added the files for mounting rclone and the clearml-agent.

johnml1135 commented 3 years ago

Everything is done and integrated into master. We can open a new issue if we need to.