DVCLive Folder not created in CI YOLOv10

[coucjh]

Search before asking

• [X] I have searched the HUB issues and discussions and found no similar questions.

Question

I am using DVC live to track experiments and print dvclive created .tsv files to a PR to run models in CI. However I am having an issue in creating any logging metrics in CI, while they are created all fine locally:

• Locally I set the device to 'mps' and using python 3.10.13 and ultralytics 8.2.66
• Locally a dvclive folder is created with a params.yaml file, artifacts folder and plots folder full of the .tsv files I need
• In CI I successfully train the model with an EC2 runner, device = '0' and am using the same python and ultralytics version
• In CI no dvc live folder is created. Additionally a stage in the verbose log is created: AMP: running Automatic Mixed Precision (AMP) checks with YOLOv8n... resulting in a yolov8n.pt file created in the directory.
• I have checked the relevant versions match exactly in both environments, as well as the model settings (attached).
• I have set os.environ["ULTRALYTICS_DVC_DISABLED"] = "false" to guarantee everything is logging.

I am stuck at this point, I am capturing the results in the logging folder so will proceed to use these, however the .tsv files and dvc functionality is much more useful for model monitoring. I have attached full logs and settings screenshots.

yolo_ci_log.txt

yolo_local_log.txt

Additional

No response

[github-actions[bot]]

👋 Hello @coucjh, thank you for raising an issue about Ultralytics HUB 🚀! Please visit our HUB Docs to learn more:

• Quickstart. Start training and deploying YOLO models with HUB in seconds.
• Datasets: Preparing and Uploading. Learn how to prepare and upload your datasets to HUB in YOLO format.
• Projects: Creating and Managing. Group your models into projects for improved organization.
• Models: Training and Exporting. Train YOLOv5 and YOLOv8 models on your custom datasets and export them to various formats for deployment.
• Integrations. Explore different integration options for your trained models, such as TensorFlow, ONNX, OpenVINO, CoreML, and PaddlePaddle.
• Ultralytics HUB App. Learn about the Ultralytics App for iOS and Android, which allows you to run models directly on your mobile device.
  • iOS. Learn about YOLO CoreML models accelerated on Apple's Neural Engine on iPhones and iPads.
  • Android. Explore TFLite acceleration on mobile devices.
• Inference API. Understand how to use the Inference API for running your trained models in the cloud to generate predictions.

If this is a 🐛 Bug Report, please provide screenshots and steps to reproduce your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

[coucjh]

Thanks @pderrenger ! I ran through all the steps, added the upgrades and the debug logging, confirmed we have write permissions and that the ULTRALYTICS_DVC_DISABLED=False. Here is the full log, a dvc lock is clearly being created but no dvclive folder still. Also note I have tried yolo settings dvc=True which says that dvc=true. yolo_full_log_ci.txt

[coucjh]

Someone from the dvc community managed to find the issue!

dberenbaum — Yesterday at 16:49 The issue might be here: https://github.com/ultralytics/ultralytics/blob/8648572809fa2e58967862f9a4748abddd0f60a7/ultralytics/utils/callbacks/dvc.py#L6 GitHub ultralytics/ultralytics/utils/callbacks/dvc.py at 8648572809fa2e589... NEW - YOLOv8 🚀 in PyTorch > ONNX > OpenVINO > CoreML > TFLite - ultralytics/ultralytics ultralytics/ultralytics/utils/callbacks/dvc.py at 8648572809fa2e589... TESTS_RUNNING is defined in https://github.com/ultralytics/ultralytics/blob/8648572809fa2e58967862f9a4748abddd0f60a7/ultralytics/utils/__init__.py#L1072. It looks like if GH actions are running, yolo will assume tests are running and skip the integration.

I checked via a print and this is the problem. I have written a command to replace that output of the function as a temporary fix, but how can this be changed in the ultralytics package? I believe my use case is valid as a non test CI usage of YOLO + DVClive. Is this something that can be reported or should I attempt to fork and contribute myself? The sed command is: sed -i 's/return "GITHUB_ACTIONS" in os.environ and "GITHUB_WORKFLOW" in os.environ and "RUNNER_OS" in os.environ/return False/' /opt/tensorflow/lib/python3.10/site-packages/ultralytics/utils/__init__.py