eth-easl / modyn

Modyn is a research-platform for training ML models on growing datasets.
MIT License
25 stars 3 forks source link

feat: Pipeline run analytics visualisation tool #464

Closed robinholzi closed 2 months ago

robinholzi commented 3 months ago

dependent on #451 and #463

Motivation

It is useful to have a tool that allows us to analyze and compare pipeline runs fast. This enables efficient experimentation and discovering of patterns in the log data. The findings from using this tool enable us to create plots for papers with almost no further data processing.

Changes

Most of the changes are in /analytics and are completely greenfield. Some code duplication was needed for the plot components as the UI hooks have to be defined at the server start time (thus not enabling us to generate every component dynamically with all its UI)

How to Review

It's more important to review the dataframe wrangling and aggregation compared to the UI boilerplate.

github-actions[bot] commented 2 months ago

Line Coverage: -% ( % to main) Branch Coverage: -% ( % to main)

robinholzi commented 2 months ago

@MaxiBoether Thanks for skimming through the PR! I guess skipping the thorough review is reasonable as long as we keep don't completely forget about the fact, that there could be issues! Reviewing would be hard anyways, as the dataframes have no annotations for columns etc. unfortunately.

When building plots for a paper/thesis we have to reinspect the logic anyways. Then it's much easier to validate on a specific case with specific values for fields like 'metric', 'dataset_id', 'eval_handler'!

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 82.50%. Comparing base (f536903) to head (ff9a424).

Files Patch % Lines
...yn/supervisor/internal/pipeline_executor/models.py 83.33% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #464 +/- ## ======================================= Coverage 82.50% 82.50% ======================================= Files 214 214 Lines 9949 9955 +6 ======================================= + Hits 8208 8213 +5 - Misses 1741 1742 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.