NatLibFi / Annif

Annif is a multi-algorithm automated subject indexing tool for libraries, archives and museums.
https://annif.org
Other
201 stars 41 forks source link

add support for spawn multiprocessing mode #638

Closed osma closed 1 year ago

osma commented 2 years ago

This (draft) PR attempts to add support for the spawn multiprocessing mode. This is the only mode supported on Windows, and the default mode on Mac OS. The fork mode, supported on Linux and some other *nix systems, is more efficient because it allows loading models only once and then reusing them in forked child processes, with most of the memory shared between the processes. But spawn mode is still better than not being able to perform multiprocessing at all.

This PR needs more testing, ideally on both Mac OS and Windows. Currently there is just a single unit test that exercises parallel evaluation using the spawn mode.

Fixes #637

codecov[bot] commented 2 years ago

Codecov Report

Base: 99.58% // Head: 99.58% // Increases project coverage by +0.00% :tada:

Coverage data is based on head (36c79c0) compared to base (63af34c). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #638 +/- ## ======================================= Coverage 99.58% 99.58% ======================================= Files 87 87 Lines 5972 5992 +20 ======================================= + Hits 5947 5967 +20 Misses 25 25 ``` | [Impacted Files](https://codecov.io/gh/NatLibFi/Annif/pull/638?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) | Coverage Δ | | |---|---|---| | [annif/parallel.py](https://codecov.io/gh/NatLibFi/Annif/pull/638/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-YW5uaWYvcGFyYWxsZWwucHk=) | `100.00% <100.00%> (ø)` | | | [annif/registry.py](https://codecov.io/gh/NatLibFi/Annif/pull/638/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-YW5uaWYvcmVnaXN0cnkucHk=) | `100.00% <100.00%> (ø)` | | | [tests/test\_cli.py](https://codecov.io/gh/NatLibFi/Annif/pull/638/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-dGVzdHMvdGVzdF9jbGkucHk=) | `100.00% <100.00%> (ø)` | | | [tests/test\_parallel.py](https://codecov.io/gh/NatLibFi/Annif/pull/638/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-dGVzdHMvdGVzdF9wYXJhbGxlbC5weQ==) | `100.00% <100.00%> (ø)` | | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

osma commented 2 years ago

Successfully tested parallel training of NN ensemble with this PR branch on a MacBook with an M1 processor (arm architecture) by @holehan, according to this report. Thanks a lot!

osma commented 1 year ago

Rebased on current master (with black & isort reformatting), fixed up and force-pushed.

sonarcloud[bot] commented 1 year ago

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

monalehtinen commented 1 year ago

Tested parallel processing in this PR branch using Windows 10 (Intel Core i5-8250U processor). Succesfully trained an nn-ensemble model, ran into no problems related to PR.