ValueError in Pipeline step

[Make42]

When switching use_current_task from False to True in

dataset_raw = Dataset.create(
    dataset_name="my_dataset_name",
    dataset_project="my_dataset_project",
    use_current_task=False
)

I get the error message:

 File "...\AppData\LoLocal\Temp\tmpt_28vdwn.py", line 66, in <module>
    results = reformatting(**kwargs)
  File "...\AppData\Local\Temp\tmpt_28vdwn.py", line 33, in reformatting
    dataset_raw.upload()
  File "...\.venv\lib\site-packages\clearml\datasets\dataset.py", line 636, in upload
    max_workers = 1 if self._task.output_uri.startswith(tuple(cloud_driver_schemes)) else psutil.cpu_count()
AttributeError: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2022.2.1\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "C:\Program Files\JetBrains\PyCharm 2022.2.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2022.2.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "...\pipeline_with_ID_output_3.py", line 538, in <module>
    executing_pipeline(
  File "...\.venv\lib\site-packages\clearml\automation\controller.py", line 4010, in internal_decorator
    pipeline_result = func(**pipeline_kwargs)
  File "...\pipeline_with_ID_output_3.py", line 504, in executing_pipeline
    print(f"reformatting_task_ID: {reformatting_task_ID}")
  File "...\.venv\lib\site-packages\clearml\utilities\proxy_object.py", line 309, in method
    obj = cb()
  File "...\.venv\lib\site-packages\clearml\automation\controller.py", line 3734, in result_wrapper
    raise ValueError(
ValueError: Pipeline step "reformatting - Demodaten", Task ID=23ae951aac814d44b956f55dc94e6876 failed

This seems to be a bug.

The snipped is in a pipeline step - a function that is decorated with @PipelineDecorator.component and used by a @PipelineDecorator.pipeline.

I am using

clearml 1.10.3 clearml-agent 1.5.2

[ainoam]

Thanks for reporting @Make42. We'll look into it.

[Make42]

@ainoam : Any progress on this?

[ainoam]

Still in the pipeline...

Thanks for following up @Make42.

[Make42]

Hi @ainoam, in order to see if the issue prevails (like a regression test for the bug), I find that use_current_task=True currently produces a ValueError, but a different one, namely, the ValueError is caused by a RecursionError: maximum recursion depth exceeded in comparison.

[...]

  File "...\clearml\utilities\proxy_object.py", line 30, in _set_callback
    self._update_func(self._update_obj, self)

  File "...\clearml\task.py", line 3547, in _refresh_args_dict
    config_dict.update(nested_from_flat_dictionary(nested_dict, a_flat_dict))
  File "...\clearml\utilities\proxy_object.py", line 45, in update
    self._set_callback()
  File "...\clearml\utilities\proxy_object.py", line 30, in _set_callback
    self._update_func(self._update_obj, self)

[...]

  File "...\clearml\task.py", line 3547, in _refresh_args_dict
    config_dict.update(nested_from_flat_dictionary(nested_dict, a_flat_dict))
  File "...\clearml\utilities\proxy_object.py", line 45, in update
    self._set_callback()
  File "...\clearml\utilities\proxy_object.py", line 30, in _set_callback
    self._update_func(self._update_obj, self)

  File "...\clearml\task.py", line 3543, in _refresh_args_dict
    a_flat_dict = task._arguments.copy_to_dict(flatten_dictionary(config_dict), prefix=name)
  File "...\clearml\backend_interface\task\args.py", line 491, in copy_to_dict
    parameters = dict([(k[len(prefix):], v) for k, v in self._task.get_parameters().items()
  File "...\clearml\backend_interface\task\task.py", line 1108, in get_parameters
    if not Session.check_min_api_version('2.9'):
  File "...\clearml\backend_api\session\session.py", line 752, in check_min_api_version
    return cls._version_tuple(cls.api_version) >= cls._version_tuple(str(min_api_version))
  File "...\clearml\backend_api\session\session.py", line 777, in _version_tuple
    return v + (0,) * max(0, 3 - len(v))
RecursionError: maximum recursion depth exceeded in comparison

I am using Dataset.create inside a @PipelineDecorator.component-decorated function in a pipeline that I am optimizing with a HyperParameterOptimizer such that the pipeline/component runs on an agent (which is my local notebook).

I have installed:

• clearml 1.10.4
• clearml-agent 1.5.2

[ainoam]

Thanks for the additional information @Make42. Do note that at this time, the latest clearml version is 1.11.1