search

probabl-ai / skore

Skore lets you "Own Your Data Science." It provides a user-friendly interface to track and visualize your modeling results, and perform evaluation of your machine learning models with scikit-learn.
https://probabl.ai
MIT License
46 stars 0 forks source link

feat: Record each value of an item over time #504

Open thomass-dev opened 23 hours ago

thomass-dev commented 23 hours ago

We have seen that for new features, such as cross-validation, we want to aggregate metrics over time. We must change the way items are saved. We should now saving items in a "appending" mode rather in "overriding" mode. In this way, we can retain all the values of an item over time.

It will also allow the user to track the evolution of an item through the library or the interface https://github.com/probabl-ai/skore/issues/403.

Proposition: 5 phases

  1. Change the way items are saved without breaking any API (project or UI).

  2. Add a new endpoint (2.1) to retrieve all the values of an item from back-end to front-end. Add a new selection button on widgets allowing to switch from one version to another ( reference @rouk1 ), using this last endpoint.

    We should think how to intelligibly represent all the versions of an item in this button: do we display only the date of the item, its content, etc.

    (2.1) /item/<key>?<latest=true|false>. This new endpoint has a positional <key> parameter and returns, depending on a optional parameter <latest>, the latest version of the item, or all its versions.

  3. Add a new developer API inside the project one, to browse programmatically all the versions of an item. I propose to add an new ItemProxy component, returned by project.get_item, which looks something like: 

class ItemProxy:
    def latest() -> Item: ...
    def all() -> list[Item]: ...

  1. Kill diskcache to efficiently browse the different versions of an item using SQL-like language.

  2. Add a pruning system.

rouk1 commented 21 hours ago

Without stepping on toes of @anasstarfaoui I would recommend something simple to start. Icon to be refined (used that one). history

tuscland commented 19 hours ago

@rouk1 please explain how this solution caters to the user's needs.

rouk1 commented 4 hours ago

It's a proposal for a first step that is easy to implement in the current design. It allows users to see every version of there item. Obviously it does not provide a way to compare stuffs. Aren't users in need to store and visualize multiple versions of there items ? Maybe I miss something.