Computational resources for CDI workshop

[rsignell-usgs]

@dbuscombe-usgs, I'd like to get the resources set up properly for the CDI workshop.

Do you know how much memory each user will need to execute the workflows?

Will each student need multiple Dask workers to work in parallel?
(and if so, how many would you like to use)?

[dbuscombe-usgs]

16 GB would be enough memory for all tasks

Not sure about dask workers. Would that only apply if data (i.e. images) were read in as dask arrays?

[rsignell-usgs]

Yes.

[dbuscombe-usgs]

Would be possible but would require a redesign - not sure if I have time ahead of the July 10 workshop. So let's go with serial. That may change ahead of the September workshop. Sound ok?

[rsignell-usgs]

Oh, okay, I think I need different instances for our kubernetes cluster then -- basically orthogonal to our current dask workflows, which use lots of processors and little RAM on each. You need let processors and more RAM.

[rsignell-usgs]

@dbuscombe-usgs , how many people will be using simultaneously? 30? 40? 50?

[dbuscombe-usgs]

Between 30 and 40

[dbuscombe-usgs]

Never more than 40

[dbuscombe-usgs]

One other thing. Participants are encouraged to bring and work on their own data, so ideally each person should have a bit of temporary storage space so they can drag files into their own server

[rsignell-usgs]

@jreadey , currently each user has 8GB persisted user space, and 16GB "scratch" space, right?

jovyan@jupyter-rsignell-2dusgs:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          39G   15G   25G  38% /
tmpfs           7.7G     0  7.7G   0% /dev
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
/dev/nvme1n1    9.8G  2.5G  6.8G  28% /home/jovyan
/dev/nvme0n1p1   39G   15G   25G  38% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs           7.7G   12K  7.7G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           7.7G     0  7.7G   0% /sys/firmware

[jreadey]

I don't think the scratch space is user writable. For the hdflab jupyterhub, I've kept the user space low since it's a ~$1.00/month cost for each user who has ever logged in. This might be a problem for users looking to upload larger files though.

[dbuscombe-usgs]

8GB persisted user space is plenty - we'll make it work without scratch space. Thanks

[rsignell-usgs]

@jreadey, it seemed writable to me because this worked:

jovyan@jupyter-rsignell-2dusgs:~$ echo "foo" > /tmp/foo
jovyan@jupyter-rsignell-2dusgs:~$ more /tmp/foo
foo

or is that not a correct test?

[jreadey]

@rsignell-usgs - yes, that's correct. I wasn't thinking that /tmp was on the overlay filesystem

[rsignell-usgs]

@dbuscombe-usgs and @jreadey , a bit of good news: AWS increased the number of instances we can ask for from 15 to 40:

[rsignell-usgs]

I have fired up 40 instances and they seem to be running fine. @dbuscombe-usgs when should we turn them back off? Thursday at ?? PDT?

[dbuscombe-usgs]

Thanks. Thursday at 6pm PDT. Class will finish around 5pm

[rsignell-usgs]

We now have autoscaling enabled on pangeo.esipfed.org, so we shouldn't have to worry about scaling up or down, although it may take as long at 10 minutes for new instances to spin up and the jupyterhub server to become available for new users. So maybe have folks do that as soon as they get in the classroom, or before the opening remarks each day.