search

enso-org / enso

Enso Analytics is a self-service data prep and analysis platform designed for data teams.
https://ensoanalytics.com
Apache License 2.0
7.39k stars 323 forks source link

Investigate how to upload bench results to AWS #7669

Closed Akirathan closed 7 months ago

Akirathan commented 1 year ago

Motivation

GH has 90 days retention policy for all the artifacts. So the artifacts (bench-results.xml) that were created during benchmark workflow runs will be deleted after 90 days. This makes it difficult to maintain the bench_downloader script that generates the bench results website.

Let's investigate how difficult it is to upload benchmark results in the workflow run directly to AWS.

Issue created from comment:

Issues a warning if the user wants to download benchmarks that are expired according to the GH policy.

  • The default retention policy for all the artifacts is 90 days. Cannot be changed.

If we need to, it is rather straightforward to add a step to the benchmark job that not only uploads the results as a GH artifact, but also as a file to S3 - we already have/had such functionality. On S3 we can store the results indefinitely, if we want to.

Upload to S3 would be very nice, that would solve a lot of problems. Do you know how to do that? Or refer me to the point when we used to do that?

I've been writing this using the Github Actions a long time ago. I think we are still doing this, but I'm less familiar with the new Rust CI structure - I think https://github.com/enso-org/enso/blob/b8edf6a3582b35573dbd824ca04d21b39d4be910/build/build/src/aws.rs#L88-L123 may be a starting point but you may need to ask @mwu-tow for more details.

Originally posted by @radeusgd in https://github.com/enso-org/enso/issues/7599#issuecomment-1695381993

JaroslavTulach commented 1 year ago

@PabloBuchu mentioned to me there is going to be a batch execution of Enso scripts including access to some DB in Enso Cloud. I'd be good to eat our own dog food.

mwu-tow commented 7 months ago

@Akirathan I think this was covered by https://github.com/enso-org/enso/pull/9075?

Akirathan commented 7 months ago

@Akirathan I think this was covered by #9075?

@mwu-tow Yes, you are right. This issue is no longer relevant.