ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Fix bug in passing multiple set config per task via cli #78

Closed amitschang closed 4 months ago

amitschang commented 4 months ago

Previously dict would be updated at the base-level meaning any parameters at level below task would be overridden. Here use the executor set_config method which already handles dict update at appropriate depth.

Resolves https://github.com/ssec-jhu/dplutils/issues/77

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (05aa2e0) to head (3d6ead8).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #78 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 11 11 Lines 578 570 -8 ========================================= - Hits 578 570 -8 ``` | [Flag](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/78/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/78/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | `100.00% <100.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

xiangchenjhu commented 4 months ago

Should the script also include tests to verify that updating or overriding individual keys within a nested dictionary configuration does not inadvertently affect adjacent keys?

amitschang commented 4 months ago

@xiangchenjhu, could you provide an example? Do you mean like:

given task function like:

def task(indf, dict_kwarg: dict, ...):

that this sets dict_kwarg['dict_elem'] and leaves other keys alone?

--set-config task.kwargs.dict_kwarg.dict_elem = 1

I don't think at the moment it descends that far. But maybe you mean something else?

xiangchenjhu commented 4 months ago

Your example captures my concern. It's understandable if supporting such deep configuration updates isn't the goal for this change

amitschang commented 4 months ago

@xiangchenjhu, cool thanks. You brought up an interesting prospect though, so I made issue to track that possibility: https://github.com/ssec-jhu/dplutils/issues/79