launchflow / buildflow

BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
https://docs.launchflow.com/buildflow
Apache License 2.0
193 stars 7 forks source link

update file sink to shard per replica #252

Closed boetro closed 1 year ago

boetro commented 1 year ago

This will help us avoid file contention when writing from a bunch of replicas.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 96.15% and project coverage change: +0.17% :tada:

Comparison is base (e00b915) 79.65% compared to head (f871646) 79.82%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #252 +/- ## ========================================== + Coverage 79.65% 79.82% +0.17% ========================================== Files 146 146 Lines 5229 5284 +55 ========================================== + Hits 4165 4218 +53 - Misses 1064 1066 +2 ``` | [Files Changed](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow) | Coverage Δ | | |---|---|---| | [buildflow/io/local/providers/file\_providers.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2xvY2FsL3Byb3ZpZGVycy9maWxlX3Byb3ZpZGVycy5weQ==) | `100.00% <ø> (+8.33%)` | :arrow_up: | | [buildflow/io/local/strategies/file\_strategies.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2xvY2FsL3N0cmF0ZWdpZXMvZmlsZV9zdHJhdGVnaWVzLnB5) | `86.66% <75.00%> (-2.81%)` | :arrow_down: | | [end\_to\_end\_tests/file\_stream\_local\_test.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-ZW5kX3RvX2VuZF90ZXN0cy9maWxlX3N0cmVhbV9sb2NhbF90ZXN0LnB5) | `96.20% <97.05%> (+0.36%)` | :arrow_up: | | [.../actors/pipeline\_pattern/pull\_process\_push\_test.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2NvcmUvYXBwL3J1bnRpbWUvYWN0b3JzL3BpcGVsaW5lX3BhdHRlcm4vcHVsbF9wcm9jZXNzX3B1c2hfdGVzdC5weQ==) | `91.76% <100.00%> (+1.09%)` | :arrow_up: | | [buildflow/core/app/runtime/actors/runtime\_test.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2NvcmUvYXBwL3J1bnRpbWUvYWN0b3JzL3J1bnRpbWVfdGVzdC5weQ==) | `94.93% <100.00%> (+0.16%)` | :arrow_up: | | [buildflow/io/aws/strategies/s3\_strategies.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2F3cy9zdHJhdGVnaWVzL3MzX3N0cmF0ZWdpZXMucHk=) | `100.00% <100.00%> (ø)` | | | [buildflow/io/aws/strategies/s3\_strategies\_test.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2F3cy9zdHJhdGVnaWVzL3MzX3N0cmF0ZWdpZXNfdGVzdC5weQ==) | `100.00% <100.00%> (ø)` | | | [buildflow/io/gcp/strategies/storage\_strategies.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2djcC9zdHJhdGVnaWVzL3N0b3JhZ2Vfc3RyYXRlZ2llcy5weQ==) | `100.00% <100.00%> (ø)` | | | [buildflow/io/local/file\_test.py](https://app.codecov.io/gh/launchflow/buildflow/pull/252?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=launchflow#diff-YnVpbGRmbG93L2lvL2xvY2FsL2ZpbGVfdGVzdC5weQ==) | `100.00% <100.00%> (ø)` | |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.