search

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k stars 2.39k forks source link

TypeError: <Target> object is not iterable when check_complete(dep) is False #3280

Open GianlucaFicarelli opened 6 months ago

GianlucaFicarelli commented 6 months ago

When a Task is run, and one of the dependencies is not fulfilled, and this dependency has a single Target as output (instead of a list of targets, or a dict of targets), then Luigi fails with:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/luigi/worker.py", line 188, in run
    nonexistent_outputs = [output for output in dep.output() if not output.exists()]
TypeError: 'RemoteTarget' object is not iterable

The failing code is at this line and it seems related to the changes in https://github.com/spotify/luigi/pull/3258.

I think that the code should be fixed to handle the case when dep.output() isn't a list of targets.

winniehell commented 3 months ago

this was was fixed in https://github.com/spotify/luigi/pull/3281 but not yet released.

@andresgomezfrr do you have a rough estimate on when the next version will be released?

andresgomezfrr commented 3 months ago

I released a new one: https://github.com/spotify/luigi/releases/tag/3.5.1 @winniehell

winniehell commented 3 months ago

@andresgomezfrr thank you! :heart: