Switch everyone to the Docker Jupyter container

[gpsaggese]

To make simpler to collaborate we want to have everyone use the official Docker Jupyter container. The beauty is that everyone has exactly the same set-up so that it's easy to collaborate instead of wasting a large amount of time in "it works for me", "it doesn't work for you"

Instructions for the container is: https://github.com/sorrentum/sorrentum/blob/master/sorrentum_sandbox/altdata_examples/tutorial_jupyter.md

Also learning how to use Docker is key to working in the real world.

Slides and tutorials are in the UMD DATA605 class https://github.com/gpsaggese/umd_data605/blob/main/lectures/02.2%20-%20Docker%20DevOps.pdf

@samarth9008 is the TA of that class so he can help with Docker questions (but not on how to install it on your laptop, that's your problem. He can help if he wants, but he is not his job to provide IT services for your laptop).

Once you are done with the switch pls mark this as done

• [x] @FirstSingularity
• [x] @okritvik
• [ ] @vinay06vinay
• [x] @Chandramani05

[Chandramani05]

I am getting an error ‘No such container airflow_cont’ when running doctor_bash.sh

[Chandramani05]

Getting the same error even after pulling

[gpsaggese]

I think you are in the wrong dir, since it works for me in the right dir (I'll improve the instructions https://github.com/sorrentum/sorrentum/blob/master/sorrentum_sandbox/altdata_examples/README.md)

> pwd
/Users/saggese/src/sorrentum1/sorrentum_sandbox/altdata_examples

saggese@wireless-10-104-222-208.umd.edu branch:'master' ~/src/sorrentum1/sorrentum_sandbox/altdata_examples
> docker_jupyter.sh
++ git rev-parse --show-toplevel
+ GIT_ROOT=/Users/saggese/src/sorrentum1
+ REPO_NAME=sorrentum
+ IMAGE_NAME=jupyter
+ FULL_IMAGE_NAME=sorrentum/jupyter
+ docker image ls sorrentum/jupyter
REPOSITORY          TAG       IMAGE ID       CREATED       SIZE
sorrentum/jupyter   latest    29eea5978db1   11 days ago   1e+03MB
+ CONTAINER_NAME=jupyter
+ docker run --rm -ti --name jupyter -p 8888:8888 -v /Users/saggese/src/sorrentum1:/data sorrentum/jupyter /bin/bash -c 'source /data/sorrentum_sandbox/altdata_examples/set_env.sh; /data/sorrentum_sandbox/altdata_examples/run_jupyter.sh'
PYTHONPATH=:/data
<module 'helpers.hdbg' from '/data/helpers/hdbg.py'>
+ jupyter-notebook --port=8888 --no-browser --ip=0.0.0.0 --allow-root --NotebookApp.token= --NotebookApp.password=
[I 23:15:02.672 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
[I 23:15:02.678 NotebookApp] Authentication of /metrics is OFF, since other authentication is disabled.
[W 23:15:05.431 NotebookApp] All authentication is disabled.  Anyone who can connect to this server will be able to run code.
[I 23:15:05.596 NotebookApp] [jupyter_nbextensions_configurator] enabled 0.6.1
[I 23:15:05.663 NotebookApp] Serving notebooks from local directory: /data
[I 23:15:05.663 NotebookApp] Jupyter Notebook 6.5.2 is running at:
[I 23:15:05.664 NotebookApp] http://7508b0b8a12d:8888/
[I 23:15:05.664 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[Chandramani05]

I was able to run jupyter from alt_example directory using run_jupyter.sh

[FirstSingularity]

I was also able to run the same notebook. Will there a separate container for each project? I was only able to see that directory's files through jupyter.

[gpsaggese]

We could do one container per project, but too many containers are a pain to maintain. For now let's have one container with all the dev packages for all the projects.

One problem is that Docker is mounting the dir under altdata_examples and so you don't see projects. I'll fix it tomorrow

[okritvik]

I was also able to run run_jupyter.sh from sorrentum_sandbox/examples. @Chandramani05 @FirstSingularity were you able to open jupyter from the URL given by docker_jupyter.sh?

[FirstSingularity]

@okritvik I don't think that one worked, but the one on docker desktop worked for me just fine.

[thejameszhang]

I am trying to switch to the docker container, but when I run the first command source dev_scripts/client_setup/build.sh, this is the output before the terminal crashes and I'm not sure why. I guess it crashes when trying to create the virtual environment. Any idea what this could be? @gpsaggese @samarth9008

[samarth9008]

I am not sure if it would work on windows. @gpsaggese Can you merge the #280 so that we start pointing windows people to different solution.

[samarth9008]

@thejameszhang You can use instructions on this page for windows OS.

[gpsaggese]

Done. If Windows contributors find something interesting, they should add to the markdown for whoever is coming next (e.g., which VM package to use, tutorials on how to install, tips-and-tricks, etc)

[thejameszhang]

Thanks @samarth9008 . Installed the virtual machine, so I was able to run the first two scripts, but now running into another issue.

I don't have permission to pull the latest cmamp image?

[samarth9008]

Ohh for this you have two options, since docker run as root user, either add docker user to your user group. You can get the command by Google search or run this command as a root user.

On Thu, 25 May, 2023, 6:22 pm Samarth KaPatel, @.***> wrote:

Can you show the exact error you are facing?

On Thu, 25 May, 2023, 6:20 pm James Zhang, @.***> wrote:

Thanks @samarth9008 https://github.com/samarth9008 . Installed the virtual machine, so I was able to run the first two scripts, but now running into another issue.

[image: image] https://user-images.githubusercontent.com/70459407/241073255-bdf9b924-0352-4f34-97a0-614c8f266c72.png

I don't have permission to pull the latest cmamp image?

— Reply to this email directly, view it on GitHub https://github.com/sorrentum/sorrentum/issues/45#issuecomment-1563580495, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOAYU74FZUE3VJAU2H2IOSTXH7LKNANCNFSM6AAAAAAVPX6UEY . You are receiving this because you were mentioned.Message ID: @.***>

[gpsaggese]

Yes after we fix this we need to add instructions on how to configure Ubuntu to run Docker (e.g., groups, Docker-in-Docker). @jsmerix is an expert on this too.

[gpsaggese]

Also @samarth9008 this thread would also go in the one-by-one on-boarding thread. Let's merge the instructions to the on-boarding checklist and close this one.

[thejameszhang]

Thanks @samarth9008 I was able to pull the latest cmamp image using the command sudo docker pull sorrentum/cmamp:latest, but now I can't merge into master or start the container.

Error is that s3fs is not defined. I tried i git_merge_master, the i command is not found. Then tried using invoke which led to the NameError.

[thejameszhang]

Update: I pip installed s3fs but still says the "One and only one set-up config should be true" and then this time a KeyError: CK_ECR_BASE_PATH

[gpsaggese]

This is the same problem as https://github.com/sorrentum/sorrentum/issues/277

The overarching issue is that we are trying to run cmamp outside of our server and so we are hitting all new problems.

@PomazkinG can you try to run cmamp on your laptop and see what needs to be fixed? Happy to help.

[PomazkinG]

Given the screenshot from https://github.com/sorrentum/sorrentum/issues/45#issuecomment-1563712193 I'd say the problem is that the virtual env is not activated.

E.g., (amp.client_venv) grisha@dev2:~/src/orange1/amp, note (amp.client_venv). Here it is activated.

So run source dev_scripts/setenv_amp.sh before running i git_merge_master, see here

I'll try to set up everything from scratch on my local PC on Mon.

[samarth9008]

Also @samarth9008 this thread would also go in the one-by-one on-boarding thread. Let's merge the instructions to the on-boarding checklist and close this one.

I think it is already there in the checklist.

[thejameszhang]

Pulled the latest container and merged with master successfully, but now unable to start the docker container with i docker_bash.

[samarth9008]

Run these commands

1. sudo groupadd docker
2. sudo usermod -aG docker [your username]

Then run the invoke commands.

On Fri, 26 May, 2023, 1:08 pm James Zhang, @.***> wrote:

[image: image] https://user-images.githubusercontent.com/70459407/241281037-62bc134d-49bc-4532-935e-9327e940555f.png

Pulled the latest container and merged with master successfully, but now unable to start the docker container with i docker_bash.

— Reply to this email directly, view it on GitHub https://github.com/sorrentum/sorrentum/issues/45#issuecomment-1564683629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOAYU773RX3AFCF2SPW5WUDXIDPPLANCNFSM6AAAAAAVPX6UEY . You are receiving this because you were mentioned.Message ID: @.***>

[thejameszhang]

[samarth9008]

Can you switch to root user and then try to build the thin environment and run the docker commands? Lets see if the problem is related to your ubuntu VM or something else.

[thejameszhang]

Got the same KeyError as before.

[samarth9008]

Your thin environment is not active.

https://github.com/sorrentum/sorrentum/issues/45#issuecomment-1564359345

[thejameszhang]

There seems to be an error building it. I'll try copying a src dir to root. Update: this worked, building the environment now.

[PomazkinG]

@thejameszhang so everything is good now?

[okritvik]

Encountered a fatal error while running i docker_jupyter

[PomazkinG]

Encountered a fatal error while running i docker_jupyter

simply rename sorrentum -> sorrentum1. by convention a repo name should end with a digit, this is useful when having multiple clones of a repo

[okritvik]

Encountered a fatal error while running i docker_jupyter

simply rename sorrentum -> sorrentum1. by convention a repo name should end with a digit, this is useful when having multiple clones of a repo

Worked. Thanks!

[thejameszhang]

@okritvik you were able to run all commands without switching to root user?

[okritvik]

@thejameszhang Yes, I was able to run without switching to root. I guess I added my user itself to the sudoers group. You can either google search depending on your ubuntu version or follow this link https://linuxize.com/post/how-to-add-user-to-sudoers-in-ubuntu/

[thejameszhang]

Thanks @okritvik . I don't have permission to access the daemon socket when running invoke docker_bash, but i was able to pull the latest image and merge with master. I also already tried adding myself to the list of sudoers.

[PomazkinG]

@thejameszhang you may find this discussion helpful. the problem is that when you run docker pull ... manually you add the sudo prefix and all works, but when running i docker_bash it also pulls the image but without the sudo prefix.

Try creating a docker group and adding your user to it (see the link above)

[thejameszhang]

Thanks @PomazkinG I believe that worked. Now I can pull the latest image without sudo.

Did i docker_bash run successfully in the below screenshot?

[samarth9008]

Not required anymore. So closing.