hytest-org / hytest

https://hytest-org.github.io/hytest/
22 stars 10 forks source link

How to add python modules to the filesystem-dask2 kernel #134

Closed sformel-usgs closed 1 year ago

sformel-usgs commented 1 year ago

I'm interested in using the filesystem-dask2 kernel but it's lacking s3fs, a module I need, for calling data from AWS. I tried out spinning up a dask cluster on qhub, but I wasn't sure how I go about adding modules, or if I'm allowed to at all. I'm pretty new to working in jupyter, so it's possible I'm just misunderstanding how kernels are intended to be used. Can I install additional modules, or is there a better approach?

rsignell-usgs commented 1 year ago

@sformel-usgs , conda-store is a killer feature of ESIP qhub, IMHO. It allows anyone to make changes in environments shared with the groups they belong to. You can't change any environments in the filesystem group, but you can change environments in the users group. Check out: https://jupyter.qhub.esipfed.org/hub/user-redirect/lab/tree/shared/users/Welcome.ipynb

But also note that the users-pangeo environment already has s3fs and a lot of other packages commonly required for cloud-native geospatial workflows. Try that out first and see if has what you need!

sformel-usgs commented 1 year ago

Thank you! I couldn't find that notebook and I started to think it was a figment of my imagination. I was wrongly under the impression that you could only spin up a dask cluster from the dask2 kernel. Anyway, sorry for overlooking the welcome notebook, I guess I didn't scroll down far enough. I was trying to figure things out from this other notebook in the examples folder. IMO the welcome notebook would be more logical in the shared folder (or a new directory like shared/intro) to keep it from being buried at the bottom of the users. That being said, I think my chagrin will make sure I won't forget it going forward.