Open morelen17 opened 1 year ago
bump! curious if these concerns have been looked at. The results were recently presented at Ray Summit 2024 but these concerns were not discussed in the presentation, just the original performance numbers.
Thanks for the bumping this and thank you for the thoughtful initial post.
Addressing the concerns:
With these changes, the exact benchmark numbers will be different, but I don’t expect any major changes in the overall trends/takeaways from the benchmarks. For fully updated numbers, it would probably be best to re run all the benchmarks with more recent updates from the past year.
Thanks @amogkam !
I do think that the 17x sagemaker performance multiple might fall quite steeply if cluster startup and docker pull are included. Curious if plans come up for Anyscale to rerun these, would love to see the results.
Hey @amogkam !
Thanks for blog post! Although I found the Ray vs other services results impressive I decided to conduct my own experiments on AWS Sagemaker. After reviewing the source code and running my benchmarks, I am ready to share the results and my concerns regarding the comparison approach.
Concerns:
ml.g4dn.xlarge
are ~40% cheaper than 1 xml.g4dn.12xlarge
($0.736 vs $4.89 per hour for compute only).max_concurrent_transforms=2
,max_payload=50
, the rest as in the notebookWould love to discuss my results! And please feel free to point out if I missed something or if I'm wrong about anything. Also ping me if any clarifications from my side are required. Thank you!