search

comet-ml / issue-tracking

Questions, Help, and Issues for Comet ML
https://www.comet.ml
85 stars 7 forks source link

Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. #444

Closed G-B-Y closed 11 months ago

G-B-Y commented 2 years ago

Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. The container has connected to the internet, but it can't upload the data. I can ping www.google.com. And in windows, the data can be uploaded, but in container can't. Ubuntu16.04, Python3.7, PyTorch1.6, cuda10.1, cudnn7, comet-ml3.21.0. image

DN6 commented 2 years ago

Hello @G-B-Y. Could you ssh into your container, run the following command inside the container, and share the output with me here?

comet check --debug
G-B-Y commented 2 years ago

Hi @DN6, thank you for answer. Here is the result. comet-error_页面_1 comet-error_页面_2 comet-error_页面_3

G-B-Y commented 2 years ago

In container it still stucks, and can't be uploaded. image

G-B-Y commented 2 years ago

COMET INFO: Uploads: COMET INFO: environment details : 1 COMET INFO: filename : 1 COMET INFO: installed packages : 1 COMET INFO: model graph : 1 COMET INFO: os packages : 1 COMET INFO: source_code : 2 (13.91 KB) COMET INFO: --------------------------- COMET ERROR: Error sending a notification, make sure you have opted-in for notifications COMET INFO: Uploading metrics, params, and assets to Comet before program termination (may take several seconds) COMET INFO: The Python SDK has 3600 seconds to finish before aborting... COMET INFO: Uploading 3176 metrics, params and output messages COMET INFO: Uploading 3176 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET ERROR: File could not be uploaded COMET ERROR: File could not be uploaded COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2911 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2719 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 2286 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET INFO: Uploading 1851 metrics, params and output messages COMET WARNING: Failed to send all messages, metrics and output will likely be incomplete. COMET ERROR: Comet failed to send all the data back (1851 messages and 0 uploads) COMET WARNING: Failed to log run in comet.ml

DN6 commented 2 years ago

Hi @G-B-Y. It might be that the internet connection on the machine is not stable throughout the duration of the experiment. Could you try using the OfflineExperiment?

This means that you would change your experiment creation step from 

experiment = comet_ml.Experiment()

to

experiment = comet_ml.OfflineExperiment(offline_directory='./')

This will save the experiment as a zip file to the current working directory in your container. You can then upload this experiment using 

comet upload ./<Your Experiment Key>.zip

Could you test to see if this is working for you?

G-B-Y commented 2 years ago

Hi, thank you for answer. I try the way you suggest, but it still can’t work. The result is as followed.

@.***:/workspace# comet upload ./comet_data/a119e1686f004de688873b4b0268e74d.zip COMET INFO: Attempting to upload './comet_data/a119e1686f004de688873b4b0268e74d.zip'... COMET INFO: Starting the upload of the experiment COMET ERROR: File could not be uploaded COMET ERROR: File could not be uploaded COMET WARNING: Error processing line 781 COMET WARNING: Error processing line 782 COMET WARNING: Error processing line 783

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月13日 1:51 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hi @G-B-Y. It might be that the internet connection on the machine is not stable throughout the duration of the experiment. Could you try using the OfflineExperiment? This means that you would change your experiment creation step from experiment = comet_ml.Experiment() to experiment = comet_ml.OfflineExperiment(offline_directory='./') This will save the experiment as a zip file to the current working directory in your container. You can then upload this experiment using comet upload ./.zip Could you test to see if this is working for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

DN6 commented 2 years ago

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine?

G-B-Y commented 2 years ago

Hello, thank you for answer. I will try the way that you suggest, but it isn’t flexible any more.😢

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月18日 0:37 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

G-B-Y commented 2 years ago

Hi, there is a question that may be important: could IP-“172.17.0.3” case the upload error?

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月18日 0:37 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hello @G-B-Y. Sorry that did not work for you. Could you try saving the resulting offline experiment zip file directly on your machine, instead of inside the container? And try uploading from your machine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

DN6 commented 2 years ago

Hi @G-B-Y. Were you able to upload the zip file from outside the container? I believe the IP you are referring to is one that is assigned to the containers on your machine's subnetwork.

G-B-Y commented 2 years ago

Hi, thank you for answer. I’m sorry that the docker outside the container still can’t upload. The container’s image is “1.9.0-cuda11.1-cudnn8-devel”.

从 Windows 版邮件发送

发件人: Dhruv Nair 发送时间: 2021年11月22日 23:34 收件人: comet-ml/issue-tracking 抄送: G-B-Y; Mention 主题: Re: [comet-ml/issue-tracking] Hi, I’m using comel-ml in docker container. The container is connected to the Internet, but it can’t upload and receive the response. (Issue #444)

Hi @G-B-Y. Were you able to upload the zip file from outside the container? I believe the IP you are referring to is one that is assigned to the containers on your machine's subnetwork. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

DN6 commented 2 years ago

Hi @G-B-Y. To clarify, you saved the zip file from the Offline Experiment outside the container on to your windows machine and then tried uploading it to Comet by running the following command outside the container on your windows machine? 

comet upload <Path to your Experiment Key.zip file>
G-B-Y commented 2 years ago

Sorry for ambiguous expression. The container is the sub environment of the host. The host system is Ubuntu LTX 16.04.7, the container is image of "pytorch/pytorch:1.9.0-cuda11.1-cudnn8-devel", and there is no windows machine.

github-actions[bot] commented 11 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 11 months ago

This issue was closed because it has been stalled for 5 days with no activity.