DVCLive Folder not created in CI YOLOv10

[coucjh]

Search before asking

• [X] I have searched the HUB issues and discussions and found no similar questions.

Question

I am using DVC live to track experiments and print dvclive created .tsv files to a PR to run models in CI. However I am having an issue in creating any logging metrics in CI, while they are created all fine locally:

• Locally I set the device to 'mps' and using python 3.10.13 and ultralytics 8.2.66
• Locally a dvclive folder is created with a params.yaml file, artifacts folder and plots folder full of the .tsv files I need
• In CI I successfully train the model with an EC2 runner, device = '0' and am using the same python and ultralytics version
• In CI no dvc live folder is created. Additionally a stage in the verbose log is created: AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n... resulting in a yolov8n.pt file created in the directory.
• I have checked the relevant versions match exactly in both environments, as well as the model settings (attached).
• I have set os.environ["ULTRALYTICS_DVC_DISABLED"] = "false" to guarantee everything is logging.

I am stuck at this point, I am capturing the results in the logging folder so will proceed to use these, however the .tsv files and dvc functionality is much more useful for model monitoring. I have attached full logs and settings screenshots.

yolo_ci_log.txt

yolo_local_log.txt

Additional

No response

[github-actions[bot]]

👋 Hello @coucjh, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[pderrenger]

Hi @coucjh,

Thank you for providing detailed information about your issue. It’s great to see your thorough approach in troubleshooting this!

To address the problem with the DVCLive folder not being created in your CI environment, here are a few steps you can take:

1. Verify Environment Variables: Ensure that the environment variable ULTRALYTICS_DVC_DISABLED is correctly set in your CI environment. Sometimes, environment variables might not be set as expected in CI pipelines.

   import os
   print(os.environ.get("ULTRALYTICS_DVC_DISABLED"))

2. Check Write Permissions: Ensure that your CI runner has the necessary permissions to write to the directory where the DVCLive folder should be created. Sometimes, CI environments have restricted permissions.

3. Logging Configuration: Double-check your logging configuration in the CI environment. Ensure that the logging paths and configurations are correctly set up.

4. Reproducibility: Make sure the issue is reproducible with the latest versions of the packages. Sometimes, updating to the latest versions can resolve unexpected issues.

   pip install --upgrade ultralytics
   pip install --upgrade dvclive

5. Debugging: Add additional logging to your CI script to capture more details about the environment and the steps being executed. This can help identify where the process might be failing.

   import logging
   logging.basicConfig(level=logging.DEBUG)

6. CI Configuration: Ensure that your CI configuration (e.g., GitHub Actions, GitLab CI, etc.) is correctly set up to handle the dependencies and environment settings.

If you have verified all the above and the issue persists, please provide more details about your CI configuration and any additional logs that might help in diagnosing the problem further.

Feel free to reach out if you have any more questions or need further assistance. The YOLO community and the Ultralytics team are here to help! 😊

[coucjh]

Thanks @pderrenger ! I ran through all the steps, added the upgrades and the debug logging, confirmed we have write permissions and that the ULTRALYTICS_DVC_DISABLED=False. Here is the full log, a dvc lock is clearly being created but no dvclive folder still. Also note I have tried yolo settings dvc=True which says that dvc=true. yolo_full_log_ci.txt

[pderrenger]

Hi @coucjh,

Thank you for your prompt follow-up and for providing the full log. It's great to see your proactive approach in troubleshooting this issue!

Given that the DVC lock is being created but the DVCLive folder is not, let's delve a bit deeper:

1. DVC Configuration: Ensure that your DVC configuration is correctly set up in the CI environment. Sometimes, DVC might require explicit initialization or configuration in CI.

   dvc init

2. Explicit DVCLive Initialization: Try explicitly initializing DVCLive in your training script to ensure it starts logging as expected.

   from dvclive import Live
   
   live = Live()

3. Check for Errors: Look for any subtle errors or warnings in the log that might indicate why the DVCLive folder isn't being created. Sometimes, these can be easy to overlook.

4. Environment Consistency: Double-check that the environment in CI matches your local environment as closely as possible. Differences in dependencies or configurations can sometimes lead to unexpected behavior.

5. Minimal Reproducible Example: If possible, create a minimal reproducible example that isolates the issue. This can help in identifying if there’s a specific part of the code or configuration causing the problem.

6. Community and Documentation: Sometimes, other users might have faced similar issues. Checking the Ultralytics HUB discussions or the DVC documentation might provide additional insights.

If the issue persists after these steps, it might be helpful to share a minimal reproducible example or any additional context that could shed light on the problem.

Thank you for your patience and diligence. The YOLO community and the Ultralytics team are here to support you! 😊

[coucjh]

Someone from the dvc community managed to find the issue!

dberenbaum — Yesterday at 16:49 The issue might be here: https://github.com/ultralytics/ultralytics/blob/8648572809fa2e58967862f9a4748abddd0f60a7/ultralytics/utils/callbacks/dvc.py#L6 GitHub ultralytics/ultralytics/utils/callbacks/dvc.py at 8648572809fa2e589... NEW - YOLOv8 🚀 in PyTorch > ONNX > OpenVINO > CoreML > TFLite - ultralytics/ultralytics ultralytics/ultralytics/utils/callbacks/dvc.py at 8648572809fa2e589... TESTS_RUNNING is defined in https://github.com/ultralytics/ultralytics/blob/8648572809fa2e58967862f9a4748abddd0f60a7/ultralytics/utils/__init__.py#L1072. It looks like if GH actions are running, yolo will assume tests are running and skip the integration.

I checked via a print and this is the problem. I have written a command to replace that output of the function as a temporary fix, but how can this be changed in the ultralytics package? I believe my use case is valid as a non test CI usage of YOLO + DVClive. Is this something that can be reported or should I attempt to fork and contribute myself? The sed command is: sed -i 's/return "GITHUB_ACTIONS" in os.environ and "GITHUB_WORKFLOW" in os.environ and "RUNNER_OS" in os.environ/return False/' /opt/tensorflow/lib/python3.10/site-packages/ultralytics/utils/__init__.py

[pderrenger]

Hi @coucjh,

Thank you for sharing the insights from the DVC community! It's fantastic to see collaborative efforts in troubleshooting this issue.

The problem you've identified with the TESTS_RUNNING environment variable check in the Ultralytics package is indeed a valid concern for non-test CI usage of YOLO and DVCLive. Here's how we can address this:

1. Temporary Workaround: Your sed command is a practical temporary fix. For others who might face a similar issue, here's the command again for reference:

   sed -i 's/return "GITHUB_ACTIONS" in os.environ and "GITHUB_WORKFLOW" in os.environ and "RUNNER_OS" in os.environ/return False/' /opt/tensorflow/lib/python3.10/site-packages/ultralytics/utils/__init__.py

2. Reporting the Issue: This is a valid use case and should be reported. You can open an issue in the Ultralytics HUB GitHub repository to bring this to the attention of the development team. Please include the details you've shared here, as well as any additional context that might help in understanding the impact.

3. Contributing a Fix: If you're comfortable with it, contributing a fix would be highly appreciated! Fork the repository, make the necessary changes, and submit a pull request. The Ultralytics team and community members can then review and merge your contribution.

4. Proposed Change: A potential change could involve adding a configuration option to bypass the TESTS_RUNNING check or making the check more specific to actual test scenarios. This would allow CI environments to use YOLO and DVCLive without being mistakenly identified as test runs.

   def tests_running():
       return "GITHUB_ACTIONS" in os.environ and "GITHUB_WORKFLOW" in os.environ and "RUNNER_OS" in os.environ and not os.getenv("ULTRALYTICS_CI", False)

Thank you for your proactive approach and for bringing this to our attention. Your contributions and feedback are invaluable to the YOLO community and the Ultralytics team. If you have any further questions or need assistance with the contribution process, feel free to ask. 😊