microsoft / PlanetaryComputer

Issues, discussions, and information about the Microsoft Planetary Computer
https://planetarycomputer.microsoft.com/
MIT License
185 stars 9 forks source link

Increase worker memory capacity #282

Closed mattsmiScientific closed 1 year ago

mattsmiScientific commented 1 year ago

I've set up my own Planetary Computer hub and I can't change the default memory size of my Worker to anything larger that 2GiB. I see options in the .yaml file that may allow me to increase it but doesn't seem to work. The old Jupyter hub image from the planetary computer I was previousy using seemed to have an 8GiB per worker limit. How do I increase the memory capacity of my workers? Thanks for any help?

TomAugspurger commented 1 year ago

It's definitely possible, but I don't recall offhand where that's set at. https://gateway.dask.org/install-kube.html probably discusses it.

mattsmiScientific commented 1 year ago

Thanks @Tom - cracked it by editing the .yaml config file (example given at bottom of https://planetarycomputer.microsoft.com/docs/concepts/hub-deployment/) to set the minimum worker size to be larger return Options( Float("worker_cores", 1, min=1, max=16, label="Worker Cores"), Float("worker_memory", 8, min=8, max=32, label="Worker Memory (GiB)"), String("image", default="pangeo/pangeo-notebook:latest", label="Image"), Mapping("environment", {}, label="Environment Variables"), handler=option_handler, )

mattsmiScientific commented 1 year ago

Now completed