Add more info to Comet-ML experiment creation from MLflow autolog

[florianblume]

Describe the Bug

It's not directly a bug but it has been driving me nuts for 2 days. Comet-ML automatically creates an experiment when you call mlflow.start_run and comet.disable_auto_logging has not been set to true. The only information about this automatically created experiment I get is Created by MLflow autologger, not very informative, especially since it's not really the MLflow autologger but rather the Comet-ML autologger. I don't need this feature and start a run with a modified MLflow version. All the logging happens later and I don't want an offline experiment to be created at this point. It would be nice if there was a print statement saying that because autologging has not been disabled this experiment was created.

Expected behavior

Some infor printed how to disable autologging and that that's the reason why the experiment was created.

Where is the issue?

• [X] Comet Python SDK
• [ ] Comet UI
• [ ] Third Party Integrations (Huggingface, TensorboardX, Pytorch Lighting etc)

[DN6]

Hi @florianblume. Could you give me a little more context on how you are using Comet with MLFlow (if it's possible to share a code snippet, please do)? If the auto-logger is enabled then experiment data should automatically be captured for your run.

[florianblume]

Hi @DN6,

sure thanks for getting back to me! It's a bit too complicated I guess to put the code here. What I essentially do is use a combination of Pytorch Lightning, MLflow and CometML. I run experiments with mlflow run using my own modified version of MLflow (that doesn't start a new experiment when running this command). Then, Pytorch Lightning instantiates the MLflow logger for local logging (that I have the experiments on the hard drive) and the CometML logger to log to the online platform.

The comet_ml package wants to be imported before any MLflow or Pytorch Lightning imports (since it modifies some of these packages' functions as I found out). I manually call in my code mlflow.start_run to make sure the local MLflow run exists, and when I do this CometML creates a new offline experiment due to the inject functions which is offline since I haven't set things like COMET_API_KEY at this point.

So the issue is within loggers/mlflow_logger.py. As I understand it this logger replaces some of the default MLflow functions. I had to set os.environ['COMET_DISABLE_AUTO_LOGGING'] = "1" to disable this behavior (I don't need it since I'm logging manually in Pytorch Lightning).

A short printed statement telling me directly how to disable the autologging which automatically creates an experiment when using MLflow (that's the part that took me quite a while to understand) would have been very helpful.

[DN6]

Understood. We will look into how we can provide more context for this behavior.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.