pangeo-data / jupyter-earth

Jupyter meets the Earth: combining research use cases in geosciences with technical developments within the Jupyter and Pangeo ecosystems.
https://jupytearth.org
Creative Commons Zero v1.0 Universal
28 stars 6 forks source link

Can we make a pod use some allocated memory and mount it as a filesystem location to write to? #93

Closed consideRatio closed 1 year ago

consideRatio commented 2 years ago

/dev/shm is a trick one can apply to mount RAM memory to a disk location. Can we make a k8s pod take some of its allocated memory and mount for use within the pod?

This is recommended when using Ray, but can we do that in a k8s environment where we have Pods etc, and where we both have requested memory that we are guaranteed, and a memory limit, where we are not guarnateed but could potentially use?

consideRatio commented 1 year ago

We have done this now. Here is a snippet from the JupyterHub helm chart configuration related to this.

    singleuser:
      # /dev/shm is mounted as a filesystem path, where writing to it means to
      # write to memory.
      #
      # How to: https://stackoverflow.com/questions/46085748/define-size-for-dev-shm-on-container-engine/46434614#46434614
      # Request for this by Ellie: https://fperezgroup.slack.com/archives/C020XCEFPEH/p1658168872788389
      #
      storage:
        extraVolumes:
          - name: dev-shm
            emptyDir:
              medium: Memory
        extraVolumeMounts:
          - name: dev-shm
            mountPath: /dev/shm