search

skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.63k stars 486 forks source link

FR: Support custom S3-server #3713

Open chymian opened 3 months ago

chymian commented 3 months ago

Since skypilot supports onPrem clusters, it would also be logical & necessary to support custom/onprem S3-server.

  1. there can be a significant speedup instead of pulling the data from S3-cloudservices (AWS/GC/etc), every time a task ist started inhouse, if the S3-bucket is hosted/mirrored inhouse as well.
  2. thinking EU Data-Security Laws and company policies: Data has to be keept inhouse!
  3. So, it's a missing piece and without it, K8s/onPrem support is not complete and not truly onprem, when one has to store the sensitive company data on a CloudService - just do download it again into my DC.
romilbhardwaj commented 3 months ago

+1. Meanwhile, noting a workaround using local Kubernetes volumes to address perf and privacy considerations - in your ~/.sky/config.yaml, you can specify custom volumes like so:

kubernetes:
  pod_config:
    spec:
      volumes:
        - name: example-volume
          hostPath:
            path: /tmp
            type: Directory
Michaelvll commented 3 months ago

Related to #2687