catalystneuro / dandi_s3_log_parser

S3 log parsing for the DANDI Archive.
BSD 3-Clause "New" or "Revised" License
1 stars 2 forks source link

Reduce memory overhead per worker #35

Closed CodyCBakerPhD closed 3 months ago

CodyCBakerPhD commented 3 months ago

During recent stress testing with parallelization, workers kept causing broken processes in the pool by using too much memory, even with relatively low buffer maximums set

Here are some further optimizations to minimize the memory overhead required per worker

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 80.30303% with 13 lines in your changes missing coverage. Please review.

Project coverage is 67.89%. Comparing base (0b1e03a) to head (53c5353). Report is 4 commits behind head on main.

Files Patch % Lines
src/dandi_s3_log_parser/_s3_log_line_parser.py 71.42% 12 Missing :warning:
...c/dandi_s3_log_parser/_dandi_s3_log_file_parser.py 87.50% 1 Missing :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35/graphs/tree.svg?width=650&height=150&src=pr&token=RIT59WJokn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro)](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro) ```diff @@ Coverage Diff @@ ## main #35 +/- ## ========================================== - Coverage 72.37% 67.89% -4.48% ========================================== Files 11 11 Lines 409 408 -1 ========================================== - Hits 296 277 -19 - Misses 113 131 +18 ``` | [Flag](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro) | `67.89% <80.30%> (-4.48%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro) | Coverage Δ | | |---|---|---| | [src/dandi\_s3\_log\_parser/\_s3\_log\_file\_parser.py](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35?src=pr&el=tree&filepath=src%2Fdandi_s3_log_parser%2F_s3_log_file_parser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro#diff-c3JjL2RhbmRpX3MzX2xvZ19wYXJzZXIvX3MzX2xvZ19maWxlX3BhcnNlci5weQ==) | `100.00% <100.00%> (+2.04%)` | :arrow_up: | | [...c/dandi\_s3\_log\_parser/\_dandi\_s3\_log\_file\_parser.py](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35?src=pr&el=tree&filepath=src%2Fdandi_s3_log_parser%2F_dandi_s3_log_file_parser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro#diff-c3JjL2RhbmRpX3MzX2xvZ19wYXJzZXIvX2RhbmRpX3MzX2xvZ19maWxlX3BhcnNlci5weQ==) | `81.11% <87.50%> (+0.11%)` | :arrow_up: | | [src/dandi\_s3\_log\_parser/\_s3\_log\_line\_parser.py](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35?src=pr&el=tree&filepath=src%2Fdandi_s3_log_parser%2F_s3_log_line_parser.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro#diff-c3JjL2RhbmRpX3MzX2xvZ19wYXJzZXIvX3MzX2xvZ19saW5lX3BhcnNlci5weQ==) | `82.02% <71.42%> (-6.71%)` | :arrow_down: | ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/catalystneuro/dandi_s3_log_parser/pull/35/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=catalystneuro)