[ENH]: Allow for `local` queue using GNU parallel

fraimondo commented 6 months ago

Are you requiring a new dataset or marker?

[X] I understand this is not a marker or dataset request

Which feature do you want to include?

So far the queue section of the YAML allows to use HTCondor.

The feature requested is to include a local queue that uses GNU parallel to run, locally, using multicore architectures. This will benefit all users dealing with relatively small datasets on environments without HPC/HTC, like dual-socket and powerful workstations.

How do you imagine this integrated in junifer?

By allowing this:

queue:
  jobname: TestLocalQueue
  kind: local
  jobs: 4
  env:
    kind: conda
    name: junifer

Do you have a sample code that implements this outside of junifer?

No response

Anything else to say?

No response

github-actions[bot] commented 5 months ago

PR Preview Action v1.4.7 :---: Preview removed because the pull request was closed. 2024-03-28 09:31 UTC

codecov[bot] commented 5 months ago

Codecov Report

Attention: Patch coverage is 95.60440% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 88.15%. Comparing base (f957538) to head (9ec6835).

Additional details and impacted files

[![Impacted file tree graph](https://app.codecov.io/gh/juaml/junifer/pull/306/graphs/tree.svg?width=650&height=150&src=pr&token=5H21JuZXMw&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)](https://app.codecov.io/gh/juaml/junifer/pull/306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) ```diff @@ Coverage Diff @@ ## main #306 +/- ## ========================================== + Coverage 87.98% 88.15% +0.16% ========================================== Files 109 111 +2 Lines 4728 4820 +92 Branches 949 957 +8 ========================================== + Hits 4160 4249 +89 - Misses 415 417 +2 - Partials 153 154 +1 ``` | [Flag](https://app.codecov.io/gh/juaml/junifer/pull/306/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [docs](https://app.codecov.io/gh/juaml/junifer/pull/306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `100.00% <ø> (?)` | | | [junifer](https://app.codecov.io/gh/juaml/junifer/pull/306/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | `88.15% <95.60%> (+0.16%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/juaml/junifer/pull/306?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml) | Coverage Δ | | |---|---|---| | [junifer/api/queue\_context/\_\_init\_\_.py](https://app.codecov.io/gh/juaml/junifer/pull/306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvcXVldWVfY29udGV4dC9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | | | [junifer/api/queue\_context/htcondor\_adapter.py](https://app.codecov.io/gh/juaml/junifer/pull/306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvcXVldWVfY29udGV4dC9odGNvbmRvcl9hZGFwdGVyLnB5) | `99.10% <ø> (ø)` | | | [junifer/api/functions.py](https://app.codecov.io/gh/juaml/junifer/pull/306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvZnVuY3Rpb25zLnB5) | `93.75% <75.00%> (+0.09%)` | :arrow_up: | | [...er/api/queue\_context/gnu\_parallel\_local\_adapter.py](https://app.codecov.io/gh/juaml/junifer/pull/306?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml#diff-anVuaWZlci9hcGkvcXVldWVfY29udGV4dC9nbnVfcGFyYWxsZWxfbG9jYWxfYWRhcHRlci5weQ==) | `96.51% <96.51%> (ø)` | | ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/juaml/junifer/pull/306/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=juaml)

synchon commented 5 months ago

@fraimondo Do we merge after CI completion?

juaml / junifer