Closed amy-tabb closed 4 years ago
@jph00 I guess I need to deprecate the fastdotai
namespace, and add some GPU images with CUDA support etc, to the fastai
namespace, which I think you said can be fully accomplished with conda? I tried this morning to make an image from conda that has all the CUDA drivers etc installed but wasn't successful (likely I'm doing something dumb).
If you have any pointers, LMK or if you have an idea how to quickly make these containers on top of the fastai
ones that is great too. Either way, I would like to try to help clean this up. Let me know your thoughts.
@amy-tabb : If you want to use GPUs I would suggest sticking to the fastdotai
ones for now. However, we need to clean this up to make this repo less confusing, sorry about that! I also need to cleanup the README. The fastai
namespaced images are mostly used for CI at the moment AFAIK.
Just as some rough notes, the way I see people building CUDA containers currently in the wild is layering ontop of Nvidia's images, i.e.
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
The way I tried to circumvent this situation is by using conda's cudatoolkit
but I am not sure if that is the right way to go or not? I was following this article https://towardsdatascience.com/managing-cuda-dependencies-with-conda-89c5d817e7e1 but wasn't a smooth ride so I was wondering if I was going down the wrong road.
P.S. I had initially tried inheriting from fastai/miniconda
and simply installing pytorch like so
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
However, this did not seem to expose the GPU ( and yes, I made sure I ran docker with the --gpu
flag)
Hi @hamelsmu thanks for the info. I have a tutorial for the current FastAI course using the fastdotai
namespace -- uses a bind mount so everything from fastbook is on your local disk and you don't have to commit containers to save.
If you happen to deprecate fastdotai
, and remember -- tag me please? (here or Twitter) so I can update my stuff.
I have the pipeline for cuda too in the tutorial, install the drivers from NVIDIA and then NVIDIA-Docker. I'm working on another deadline, still have to check the tutorial + its links, but when I get back to it, will incorporate looking into cudnn.
But -- I am getting all the Jupyter notebooks to run for Chapters 1-9 (had to stop) and fastdotai
is working for me.
@amy-tabb no problem will do. You can safely depend on fastdotai
for now, I will not be making any changes, and if I do it will be very slow such that I would keep the old ones maintained for some time. And of course, I will notify you as well if anything changes!
We will tag you on this issue if there are updates.
BTW thanks for doing the tutorial, I think myself and @jph00 would love to hear more about what you are working on and what kind of tutorial it is!
Sure thing -- once I get everything checked I'll send it over.
The main idea behind the tutorial is for someone who wants to run the 2020 FastAI course from their local machine, doesn't know anything about Docker, but maybe has some idea about installing things. (I don't go into all the GPU problems, though I have had some, i.e., a bad power supply.)
The project was driven by my own constraints and preferences, and I documented it along the way -> tutorial.
@hamelsmu small note, you don't have install cudnn separately if you have the NVIDIA driver and NVIDIA-Docker installed, because in pytorch, the cuda libraries are now included (the docs I could find are here ... would love a better reference) when using the binaries. (Building from source, yes, you need to install all of the NVIDIA libs yourself.)
So with this testing program,
import torch
from fastai.vision import *
from fastai.metrics import error_rate
print("Is cuda available?", torch.cuda.is_available())
print("Is cuDNN version:", torch.backends.cudnn.version())
print("cuDNN enabled? ", torch.backends.cudnn.enabled)
x = torch.rand(5, 3)
print(x)
I get something like this, using the fastdotai\fastai
image as a base image, and not adding anything pytorch or NVIDIA-specific:
fastai-user@atabb-Precision-T7610:~/testing$ python3 test2.py
Is cuda available? True
Is cuDNN version: 7603
cuDNN enabled? True
tensor([[0.7559, 0.9504, 0.9759],
[0.7765, 0.6080, 0.1925],
[0.7885, 0.9641, 0.9562],
[0.4040, 0.7394, 0.5701],
[0.4912, 0.2765, 0.4441]])
Hi @hamelsmu here's the tutorial. Any bad links, bad info, type of feedback appreciated. I have redone it a few times now.
It is live now, but other parts of my website don't link to it at the moment.
๐
This looks great! Some questions - I see you have this:
RUN echo '#!/bin/bash\njupyter notebook --ip=0.0.0.0 --no-browser' >> run_jupyter.sh
However, the base image already comes with a script like this:
Why do you choose to overwrite it?
You might want to mention that you can run commands like a jupyter notebook without actually entering the container interactively (I find that many people do not realize this):
Also you do not have to clone the fastai book and jupyter notebooks. Per the README
Other random things:
Thanks
@amy-tabb left some questions above, one more comment is perhaps go through the README of this repo you might find something useful (hopefully). Also let me know if you have any questions! Welcome to the fastai community ๐
@hamelsmu
I don't overwrite fastdotai\fastai
's run_jupyter.sh
, which is residing in fastdotai\fastai
's WORKDIR = /workspace (from pytorch\pytorch
, which reminds me, I need to put that detail in, in page 2).
Because the WORKDIR=/home
at that point in the Dockerfile, that's where my run_jupyer.sh
is. I wanted people to be able to use different ports than 8888, which in a test, I now realize won't work with my script. You can still access the fastai
script, sh ../../workspace/run_jupyter.sh
.
I chose in this tutorial to use the fastdotai\fastai
image and then clone the course locally so that people can make notes, experiment, make copies of a chapter and then alter it, without having to then worry about committing containers to save. You have to save some keys in particular files in Chapters 2 & 9, and doing all of that is easier (I feel) in the local environment where you can look around in using graphics versus in a container. (notes at https://amytabb.com/ts/2020-10-04/)
I will add a detail about running not as bash, thanks.
Ok, I will look into gpustat. Thanks for taking a look! I have some things to fix.
That makes sense , thanks again for sharing!
Hi @hamelsmu , don't know the Github etiquette for this. Should I close this, leave the issue open, something else? I don't need anything else : )
Ahh no reason to stress :). We arenโt dogmatic about these things. Thanks for being so thoughtful :). Feel free to close the issue!
On Fri, Oct 9, 2020 at 10:40 AM Amy Tabb notifications@github.com wrote:
Hi @hamelsmu https://github.com/hamelsmu , don't know the Github etiquette for this. Should I close this, leave the issue open, something else? I don't need anything else : )
โ You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fastai/docker-containers/issues/22#issuecomment-706314654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALKJEVC762LLXVUNJDZZHDSJ5DHZANCNFSM4SCBTDEA .
I'm playing around with these Docker images, by mistake I pulled
fastai/fastai
instead offastdotai/fastai
.There are links to both from this repo -- to
fastdotai
in the README, and on the right, tofastai
, but more occurrences offastdotai
in the README.Which is the preferred username to pull from? Thanks.