search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.69k stars 654 forks source link

_send method None return not handled by _query_tasks method #695

Open zwergon opened 2 years ago

zwergon commented 2 years ago

For some reason, when i try to load a dataset (Dataset.get), method _query task is called and this method try to call _send method of InterfaceBase class. This method may return None and this case is not handled by the _query_task method that tries to read directly result. In my case, result is None and my Task crash.

I'm inside a docker launched by clearml_agent.

jkhenning commented 2 years ago

Hi @zwergon,

I assume this happens in offline mode? What would you expect in this case?

zwergon commented 2 years ago

I don't know if i am in offline mode. My clearml-agent is running on a box and launch a docker when i submit a task to the queue it belongs. And some times ( i'm unable to track exactly when so far) the task fails due to this "null pointer execption". I send a quick look at the procedure and i saw indeed that the method return None if offline is on. But i don't know when this "offline" tag is activated.

DavidNativ commented 2 years ago

Hi @zwergon

You should be in offline mode if you have used Task.set_offline(True) You can check your mode using Task.is_offline(). By default, task._offline_mode = None (and you are not in offline mode)

Is there any other message being logged that might help? The error messages might include Action failed , Failed sending , Retrying, previous request failed , contains illegal schema Are you able to connect to the backend at all from the program you are trying to get the dataset?