skypilot-org / skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
https://skypilot.readthedocs.io
Apache License 2.0
6.81k stars 513 forks source link

Custom benchmark for inference #4306

Open tylerweitzman opened 1 week ago

tylerweitzman commented 1 week ago

It would be great to have the ability to define custom logs through SkyCallback for things OTHER than steps/training. For example, inference with different modes and params, warmup time, etc. Something like sky_callback.report(key: string, value: string) that would then propagate as a column in the sky benchmark report from each VM.

Is there any way to currently achieve this?

concretevitamin commented 1 week ago

@tylerweitzman We're very happy to accept a community contribution for this.