aimhubio / aim

Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.
https://aimstack.io
Apache License 2.0
5.23k stars 321 forks source link

[feat] Support remote mass updates #3203

Closed peter-sk closed 3 months ago

peter-sk commented 3 months ago

For tracked runs with long sequences (think millions of steps), the performance when copying/moving/syncing runs over to a remote repository is prohibitive (think 20 minutes to several hours for two computers in different organization with 1 GBit/s to the backbone). The reason is the overhead incurred by the many RPC calls.

This PR adds a mass update function "update" that allows to set multiple (key/path, value) tuples in one (RPC) call.

The result is that copying a run with millions of steps now takes between a few seconds and a couple of minutes, i.e., a speed-up of approx. 100x when copying with a chunk size of 128.

When this PR gets accepted (or similar functionality integrated otherwise), we would be happy to open another PR that integrate it into the "aim runs cp", "aim runs mv" etc. commands AND adds a "aim runs sync" command, which copies only new data, allowing for efficient close-to-real-time replication of repositories.

CLAassistant commented 3 months ago

CLA assistant check
All committers have signed the CLA.

CLAassistant commented 3 months ago

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Prof. Peter Schneider-Kamp seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.