search

ersilia-os / ersilia

The Ersilia Model Hub, a repository of AI/ML models for infectious and neglected disease research.
https://ersilia.io
GNU General Public License v3.0
203 stars 131 forks source link

🐕 Batch: GitHub Action to automatically test the `--track` flag in a model run #925

Open miquelduranfrigola opened 9 months ago

miquelduranfrigola commented 9 months ago

Summary

The ersilia run --track flag uploads a model digest to an S3 bucket that is eventually monitored by Splunk. It would be good to have a cron job that does a short run to ensure that things are up-and-running. Suggestions are welcome!

Objectives

Documentation

GemmaTuron commented 1 week ago

Hi @DhanshreeA

You marked this as Ready, but the issue is not closed. what is the status?

DhanshreeA commented 1 week ago

This is not ready and I marked that incorrectly. Currently the --track flag is not incorporated anywhere other than ersilia-self-service.

However, I think we should bundle this with the ersilia inspect command since that command actually runs the same model a couple of times, for example, with 1, 10, 100 inputs - and an approach like that would be better suited for tracking the computational requirements of an individual model; and collecting the similar data to run a comparison across models.

At present within tracking, we're just taking an average of the memory and CPU percentage consumed regardless of the input sizes - but with that information incorporated, our analyses would be more substantial. So to conclude, we should bundle track with ersilia inspect, and potentially run this within ersilia maintenance repository.

GemmaTuron commented 6 days ago

Hi @DhanshreeA

This is a Batch initiative. Can you detail here the tasks to be completed for this, so I can convert them to issues (maybe in the ersilia-maintenance repository).