neptune-ai / neptune-client

📘 The experiment tracker for foundation model training
https://neptune.ai
Apache License 2.0
580 stars 63 forks source link

Feature Request: fetch_runs_table retrieve MIN/MAX value in floatSeries #1858

Open tyler-rt opened 2 months ago

tyler-rt commented 2 months ago

Is your feature request related to a problem? Please describe.

Currently, fetch_runs_table returns just the LAST value of a floatSeries column, and there's seemingly no way for different behavior. Note how it's possible for query but not columns:

ml_projects = neptune.init_project(
    project="My/runs",
    mode="read-only",
)
ml_projects.fetch_runs_table(
    query="max(`val/accuracy`:floatSeries) > 0.9",
    columns=[
        "val/accuracy",
        "max(`val/accuracy`:floatSeries)", # returns nothing
        "val/accuracy|LAST", # returns nothing
    ],
).to_pandas()

Describe the solution you'd like

I would like to be able to use NQL or similar in columns

Describe alternatives you've considered

iterating through each run, one-by-one. very slow! 100x more queries.

Have I missed a way to do this?

Edit: the alternative is worse than I thought. I want to do this for about ~80 columns, and using the Run API I have to query each of the 80 floatSeries independently. So instead of 1 query per 100 runs as with fetch_runs_table to get the LAST values of each column I have to do 8000x more queries to get the MIN values of each column.

SiddhantSadangi commented 2 months ago

Hey @tyler-rt 👋

Thank you for writing to us and submitting this feature request. I see the utility behind this and have forwarded it to our product team for review ✅

Meanwhile, you can get the results from the Web App, by adding the max aggregation of the val/accuracy field to the table, and then downloading it as CSV. You can also save this view as a custom view, so that you can get this data for new runs later.

I understand that adding 80 such columns to the table can take a while, and a manual approach is not what you would have in mind, but this is the best way of accomplishing what you need currently.

Please let me know if this helps or if I can help you with anything else 🤗