wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
245 stars 21 forks source link

Is it possible to sync old wandb local runs? #91

Open andreear-23 opened 1 year ago

andreear-23 commented 1 year ago

Hi, I just managed to setup the local wandb version (had some issues with not being able to login, fixed thanks to Issue #64 ) and I was wondering if it is possible to sync old runs that saved also wandb logs locally.

The problem that I seem to be having is that my previous runs all were using a different entity than what I have now and I couldn't find any information on whether it is possible to somehow change the logs of the old runs to make it compatible. Basically when I try to sync old offline runs I get into the "Permission denied issue", and I checked the code and the entity was different than what I am using now.

Is there any way to visualize old wandb runs that have not been synced so far?

vanpelt commented 1 year ago

Sure! You can either override the entity by passing the -e flag to wandb sync or create a team with the same name as the old runs used and add yourself to it.

MBakirWB commented 1 year ago

Hi @AndreeaRadoescu , to add to the above with an expanded example, you can use the following command:

wandb sync --project "<project-name>" --entity "<entity>" <path-to-run-folder>

Test this with the following basic run, extract the content and add the run folder to your working directory

run-20220829_191828-3cvxvrbj.zip

More on wandb sync can be found in our documentation, see here. Please let us know if you have any questions.

andreear-23 commented 1 year ago

Thanks so much guys, it works. Before your answers I thought about just creating a username with the same name as the entity used in the experiments and that seemed to work but being able to change the entity name is a much better solution.

One more question though: in my attempt to try and get it to work, I might have greped for where that entity name was used in the wandb files and I saw it was in one of the binaries, so I kinda changed the binary (using nano which apparently parses it nicely) to use a different entity but I think that broke the validation of the binary that happens at a later time. When I tried to sync it with a changed name, I got an error message that checksum failed. So I changed it back to what it was before but I still get the same error message, probably the hash of the binary changed or something.... So now comes the question, is it possible to still recover that experiment somehow and sync the files with the wandb backend?

vanpelt commented 1 year ago

If you change the file back to use the original entity name the checksum should succeed.

andreear-23 commented 1 year ago

Ok, I will try that then. Thanks so much for all your help!

exalate-issue-sync[bot] commented 1 year ago

WandB Internal User commented: AndreeaRadoescu commented: Ok, I will try that then. Thanks so much for all your help!