search

ssec-jhu / dplutils

Distributed(Data) Pipeline Uitilities
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Move remote setup in ray stream executor to enable configuration #68

Closed amitschang closed 4 months ago

amitschang commented 4 months ago

Setting up remotes in the initialization meant that task specific overrides like setting num_cpus via executor set_config did not have affect.

Resolves https://github.com/ssec-jhu/dplutils/issues/67

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (bed9fdb) to head (15b765b).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #68 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 11 11 Lines 574 576 +2 ========================================= + Hits 574 576 +2 ``` | [Flag](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/68/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/ssec-jhu/dplutils/pull/68/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu) | `100.00% <100.00%> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssec-jhu#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ryanhausen commented 4 months ago

@amitschang thanks, will review

amitschang commented 4 months ago

@ryanhausen, thanks, you are more than welcome to of course, but I did switch the automated request to Xiang since he's using this package.

xiangchenjhu commented 4 months ago

Setting up remotes in the initialization meant that task specific overrides like setting num_cpus via executor set_config did not have affect.

How to set num_cpus in this scenario?

amitschang commented 4 months ago

@xiangchenjhu , can you be more specific? The test shows how this could be done, and from the command line it would be like 

pl --set-config task.num_cpus=2