Central coordination for multiple skypilot cli users

[MattCWheeler]

I'm interested in using SkyPilot, but I'd like each member of our team to be able to interact with each cluster regardless of who spun it up.

*Is there any plan to support storing state (the stuff in ~/.sky/.db, etc) in some central place such as an s3 bucket or postgres database?**

To my mind this would be akin to how Terraform is used with multiple team members. Keep state files centrally (s3 is a common choice) and use locks (often kept in dynamodb) to mutex the actions that need to be mutex.

[concretevitamin]

Hello @MattCWheeler, thanks for raising this. There are a few answers to this:

• Central storage of .sky: 100% agree this is akin to Terraform state files being stored in S3 / other storage, so it's in scope (no ETA).
• Moving towards a client-server architecture
  • This way, each team deploys a SkyPilot server where everyone sends requests to
  • This is medium+ term
• Decentralized: We do see many orgs using SkyPilot in a decentralized fashion; this works for them due to
  • Each dev cluster can be autostopped/autodowned
  • Little interference between users
  • Moving onto scaling-out jobs: Each (managed spot) job uses its own VMs and are torn down on exit
  • Bookkeeping/Alerts: Each VM is auto-tagged on the cloud with skypilot-user: <whoami>
  • Easy to have a Slack/Email bot to daily query the cloud and send an alert to the team to see what VMs are live

Happy to chat more about any of these. Feel free to ping the dev team on the Slack: https://slack.skypilot.co/

[fozziethebeat]

How crazy would it be if I were setup a fixed server with skypilot and any other important configs and use that to trigger jobs within my cloud? That way, as long as all users who log in have shared access to the sky directory, they'll all end up seeing the same state of resources.

This would be a pretty simple short term hack to approximate a client server architecture.

[concretevitamin]

@fozziethebeat That's actually a recommended workaround for now! We've seen quite a few deployments using this pattern. Can have users log into that node (zero lift) or build a lightweight http server to wrap commands.

[fozziethebeat]

Great! I'll probably give that a shot