search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

mmap()'d Ring Space #135

Open jaycedowell opened 4 years ago

jaycedowell commented 4 years ago

This PR adds a new mmap()'d space, named "mapped", for Bifrost rings. This will allow for larger disk-based rings for buffering data.

jaycedowell commented 4 years ago

I'm generally happy with how it works but the specification of the base directory in user.mk seems a little fiddly. I'm also not sure how to allow multiple base directories to be specified.

coveralls commented 4 years ago

Coverage Status

Coverage decreased (-0.03%) to 61.338% when pulling 4ff90d08c9a6bb2044f840ef64d0990f69c9ee7c on jaycedowell:disk-backed-ring into 1681fde6e643fcc03a3cea10e411b8411aeb31cc on ledatelescope:master.

codecov-io commented 4 years ago

Codecov Report

Merging #135 into master will decrease coverage by 0.21%. The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #135      +/-   ##
==========================================
- Coverage   58.78%   58.56%   -0.22%     
==========================================
  Files          63       64       +1     
  Lines        5095     5112      +17     
==========================================
- Hits         2995     2994       -1     
- Misses       2100     2118      +18
Impacted Files Coverage Δ
python/bifrost/pipeline.py 84.38% <ø> (ø) :arrow_up:
python/bifrost/libbifrost.py 55.07% <ø> (ø) :arrow_up:
python/bifrost/ndarray.py 77.23% <100%> (ø) :arrow_up:
python/bifrost/Space.py 61.11% <100%> (ø) :arrow_up:
python/bifrost/memory.py 69.64% <50%> (-4.44%) :arrow_down:
python/bifrost/romein.py 0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2af28de...1d755d8. Read the comment docs.

codecov-commenter commented 4 years ago

Codecov Report

Merging #135 into master will decrease coverage by 2.90%. The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #135      +/-   ##
==========================================
- Coverage   61.62%   58.71%   -2.91%     
==========================================
  Files          64       64              
  Lines        5300     5302       +2     
==========================================
- Hits         3266     3113     -153     
- Misses       2034     2189     +155
Impacted Files Coverage Δ
python/bifrost/libbifrost.py 52.70% <ø> (-25.68%) :arrow_down:
python/bifrost/pipeline.py 84.66% <ø> (ø)
python/bifrost/memory.py 70.17% <50.00%> (-6.19%) :arrow_down:
python/bifrost/Space.py 61.11% <100.00%> (ø)
python/bifrost/ndarray.py 76.92% <100.00%> (-5.99%) :arrow_down:
python/bifrost/fir.py 0.00% <0.00%> (-100.00%) :arrow_down:
python/bifrost/romein.py 0.00% <0.00%> (-100.00%) :arrow_down:
python/bifrost/blocks/scrunch.py 37.03% <0.00%> (-59.26%) :arrow_down:
python/bifrost/map.py 20.83% <0.00%> (-56.25%) :arrow_down:
python/bifrost/linalg.py 36.84% <0.00%> (-52.64%) :arrow_down:
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update a53b35b...3abcc4a. Read the comment docs.

jaycedowell commented 4 years ago

I think the environment variable approach is viable and takes care of my last concern.

codecov-io commented 3 years ago

Codecov Report

Merging #135 (e8714cf) into master (e1cea0b) will decrease coverage by 2.90%. The diff coverage is 66.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #135      +/-   ##
==========================================
- Coverage   61.62%   58.71%   -2.91%     
==========================================
  Files          64       64              
  Lines        5300     5302       +2     
==========================================
- Hits         3266     3113     -153     
- Misses       2034     2189     +155
Impacted Files Coverage Δ
python/bifrost/libbifrost.py 52.70% <ø> (-25.68%) :arrow_down:
python/bifrost/pipeline.py 84.66% <ø> (ø)
python/bifrost/memory.py 70.17% <50.00%> (-6.19%) :arrow_down:
python/bifrost/Space.py 61.11% <100.00%> (ø)
python/bifrost/ndarray.py 76.92% <100.00%> (-5.99%) :arrow_down:
python/bifrost/fir.py 0.00% <0.00%> (-100.00%) :arrow_down:
python/bifrost/romein.py 0.00% <0.00%> (-100.00%) :arrow_down:
python/bifrost/blocks/scrunch.py 37.03% <0.00%> (-59.26%) :arrow_down:
python/bifrost/map.py 20.83% <0.00%> (-56.25%) :arrow_down:
python/bifrost/linalg.py 36.84% <0.00%> (-52.64%) :arrow_down:
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e1cea0b...e8714cf. Read the comment docs.

codecov-commenter commented 3 years ago

Codecov Report

Patch coverage: 55.55% and project coverage change: -0.04 :warning:

Comparison is base (0bab382) 65.39% compared to head (d768ed5) 65.36%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #135 +/- ## ========================================== - Coverage 65.39% 65.36% -0.04% ========================================== Files 67 67 Lines 5840 5846 +6 ========================================== + Hits 3819 3821 +2 - Misses 2021 2025 +4 ``` | [Impacted Files](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration) | Coverage Δ | | |---|---|---| | [python/bifrost/libbifrost.py](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration#diff-cHl0aG9uL2JpZnJvc3QvbGliYmlmcm9zdC5weQ==) | `71.11% <ø> (ø)` | | | [python/bifrost/pipeline.py](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration#diff-cHl0aG9uL2JpZnJvc3QvcGlwZWxpbmUucHk=) | `84.70% <ø> (ø)` | | | [python/bifrost/version/\_\_main\_\_.py](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration#diff-cHl0aG9uL2JpZnJvc3QvdmVyc2lvbi9fX21haW5fXy5weQ==) | `0.00% <0.00%> (ø)` | | | [python/bifrost/Space.py](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration#diff-cHl0aG9uL2JpZnJvc3QvU3BhY2UucHk=) | `65.00% <100.00%> (ø)` | | | [python/bifrost/memory.py](https://app.codecov.io/gh/ledatelescope/bifrost/pull/135?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=LEDA+Collaboration#diff-cHl0aG9uL2JpZnJvc3QvbWVtb3J5LnB5) | `79.66% <100.00%> (+0.71%)` | :arrow_up: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

jaycedowell commented 2 years ago

Well, at least mapped compiles on MacOS now. That's something.

jaycedowell commented 2 years ago

Keep the logging directory and the mapped ring directory separate on mac seems to have helped with a lot of the testing failures but not there is a segfault when the test suite exits.

jaycedowell commented 2 years ago

Looks like there are some problems related to the recent changes to filesystem.[ch]pp.