dsblank
commented
11 months ago Sorry for the delay in responding to your question.
Basically, the speed of an experiment creation is based on how many things are being auto-logged at the beginning. Autologging the git data is perhaps the most time-consuming operations. But turning most auto-logging off makes it very fast.
Most items are uploaded in separate threads.
Your complexity can be reduced with:
from comet_ml import OfflineExperiment as Experimentand nothing else need to change. You can programmatically upload a zip file, but it is perhaps easier to:
comet upload filename.zipHope that helps someone!
Describe the Bug
I'm not sure if this is a bug, or if all Comet users wait for 4-5 seconds every time an experiment is created.
It seems excessive though, I want to be able to quickly iterate on a model, having to wait 5 seconds every time I re-run a script is a pretty big 'con' to using Comet (I'm currently performing an assessment of Comet/MLFlow/TensorBoard, etc), especially since it's so unnecessary.
I see that I could use
comet_ml.OfflineExperimentat the start of my script, allow the training to start, and asynchronously create the online Experiment while the GPU is busy training, but that seems like quite a lot of complexity on my end.Can't Comet take an 'offline first' approach and let my training get started, collecting metrics while asynchronously creating the experiment online then uploading all the data?
As a second prize, perhaps easier to do, it would be good if the API had an 'upload' option (does it?) and even better if the
OfflineExperimentobject had an.upload()method, or at the very least the filename that represents it (just puttingzip_file_filenameonselfwhen logging it).It would be great to get a response on this, as it will help in my assessment, if Comet have plans to work on performance then that counts for something!