search

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k stars 2.39k forks source link

Fix error message in case of unfulfilled dependencies with single output #3281

Closed GianlucaFicarelli closed 5 months ago

GianlucaFicarelli commented 6 months ago

Description

Use flatten() to handle the outputs of any unfulfilled dependencies when a TaskProcess is run.

Motivation and Context

When a Task is run, and one of the dependencies is not fulfilled, and this dependency has a single Target as output (instead of a list of targets, or a dict of targets), then Luigi fails and prints the wrong error TypeError: 'RemoteTarget' object is not iterable.

See https://github.com/spotify/luigi/issues/3280

Have you tested this? If so, how?

Included unit test for TaskProcess.run()

winniehell commented 5 months ago

@andresgomezfrr can you please take a look at this?