Silence false positive UnconsumedParameterWarning

spotify / luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Apache License 2.0

17.85k stars 2.39k forks source link

Silence false positive UnconsumedParameterWarning #3235

Closed starhel closed 1 year ago

starhel commented 1 year ago

UnconsumedParameterWarning is emitted even if parameters are correctly consumed.

Description

There are two cases when UnconsumedParameterWarning is false positive:

camelCase or parameters with dashes in name are not correctly recognized when cfg parser is used.
Some parameters are defined in core section of config (autoload_range, no_configure_log), but are not needed in class instance (as must be used before class is initialized).

Motivation and Context

False positive UnconsumedParameterWarning forced me to completely disable this feature as any output to stderr is treated as error in pipelines.

Have you tested this? If so, how?

I've added unitests and tested this branch in my pipeline.

starhel commented 1 year ago

@andresgomezfrr Do you have any idea how to fix codecov?

andresgomezfrr commented 1 year ago

@andresgomezfrr Do you have any idea how to fix codecov?

@starhel Could you try to update this branch with master?

starhel commented 1 year ago

Branch was updated, I've forced CI to retest PR.

edit: codecov python package is no longer supported: https://github.com/codecov/codecov-python.

dlstadther commented 1 year ago

Apologies for not merging this sooner!

tashrifbillah commented 2 weeks ago

I am using Luigi 3.5.2, the latest. But I still have these warnings!

Edit: I think I see the issue. The PR ignored only these parameters:

    ignore_unconsumed = {
        'autoload_range',
        'no_configure_logging',
    }

but any other parameter still raises these warnings.

I ended up solving it as:

https://github.com/pnlbwh/luigi-pnlpipe/commit/4e5977ab73da25030637e50ea7f78005f11f5e40