daavoo
commented
1 year ago With the current endpoint for live metrics, an existing experiment could be shared with 3 REST API calls:
start:
json={
"type": "start",
"repo_url": "STUDIO_REPO_URL",
"baseline_sha": "BASELINE_SHA",
"name": "EXP_NAME",
"client": "vscode", # I think `client` is just ignored by studio
},
headers={
"Authorization": "token STUDIO_TOKEN",
"Content-type": "application/json",
}data
Include here metrics, params, and plots (only linear plots are accepted by the API).
The API was designed for sending incremental updates of the plots on each step, but it would still work if the full data is sent and step is set to the latest:
json={
"type": "data",
"repo_url": "STUDIO_REPO_URL",
"baseline_sha": "BASELINE_SHA",
"name": "EXP_NAME",
"step": 2,
"metrics": {"metrics.json": {"data": {"step": 2, "foo": 3}}},
"params": {"params.yaml": {"fooparam": 1}},
"plots": {"plots/foo.tsv": {"data":
[{"step": 0, "foo": 1.0}, {"step": 1, "foo": 2.0}, {"step": 2, "foo": 3.0}]}
},
"client": "vscode",
},
headers={
"Authorization": "token STUDIO_TOKEN",
"Content-type": "application/json",
},done:
json={
"type": "done",
"repo_url": "STUDIO_REPO_URL",
"baseline_sha": "BASELINE_SHA",
"name": "EXP_NAME",
"client": "vscode",
},
headers={
"Authorization": "token STUDIO_TOKEN",
"Content-type": "application/json",
}
mattseddon
dberenbaum
shcheklein
omesser
Related to https://github.com/iterative/vscode-dvc/issues/2855, the extension can make it easier to share experiments. Let's discuss what's needed here?
My initial thoughts on what's needed:
For 1, I think it makes sense to use Studio since it already has all this functionality. The extension can upload the params, metrics, and plots to Studio like dvclive is doing for live metrics (except for the "live" part). After selecting any number of experiments, there could be an option to post to Studio. The only user friction should be having a Studio token.
For 2, I think there are a lot of ways to do it in DVC already, so it's probably not as critical, but maybe VS Code can make it smoother. With one click, the extension could create a branch with the same name as the experiment, push that to GitHub, and show the URL to create a PR (like the git cli message
Create a pull request on GitHub by visiting...). Regardless of the decided UX, it might be better to choose one and not overwhelm the user with options/choices here.doneevent