search

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.92k stars 2.4k forks source link

Luigi Passes List to json.loads When Using TOML Config and ListParameter #2812

Closed adonig closed 4 years ago

adonig commented 5 years ago

I'm using luigi 2.8.9 with toml 0.10.0 as extra. My luigi.toml contains a list. Accessing its value with a luigi.ListParameter causes the following error:

File "/Users/asd/Projects/smtc/pipeline/.venv/lib/python3.7/site-packages/luigi/parameter.py", line 1054, in parse return list(json.loads(x, object_pairs_hook=_FrozenOrderedDict)) File "/Users/asd/.pyenv/versions/3.7.4/lib/python3.7/json/init.py", line 341, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, '

The problem is that x already has type <class 'list'> while it should be a str instead.

adonig commented 5 years ago

NOTE: A workaround is just using a luigi.Parameter instead.

ihowell commented 4 years ago

Would you mind posting your .toml file and associated config? This is an interesting case.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.