Open MaxGroot opened 11 months ago
All modified and coverable lines are covered by tests :white_check_mark:
Comparison is base (
676d61c
) 80.06% compared to head (c3747b6
) 80.38%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Attention: Patch coverage is 96.34146%
with 3 lines
in your changes are missing coverage. Please review.
Project coverage is 81.94%. Comparing base (
4a47670
) to head (873ba91
).
Files | Patch % | Lines |
---|---|---|
flow/record/base.py | 89.65% | 3 Missing :warning: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@yunzheng To support transparent compression when writing to filelike objects, I've added wrap_in_compression
to base.py
. This wraps fp in a compressor if this seems sensible based on a given path. This is copied from open_path
, though changed to always instantiate the compressor using a filelike object. This should solve the problem of writing a gzipped file to a GCS bucket, though cannot test it for realsies.
An unfortunate consequence is that the RecordAdapter
method has become just a tad more complex than it already was. We've discussed refactoring it in the past, but I think it will remain complex due to how flexible it is expected to be. It has to account for a URL, a file-like object, and whether or not the adapter should be a writer or a reader, with some complexity mixed in for handling stdio. If you have ideas for refactoring it properly I'm all ears, though that may be something for a different PR.
Depends on https://github.com/fox-it/flow.record/pull/83.
This pull request adds a Google Cloud Storage adapter for flow.record. It supports both reading from and writing to Google Cloud Storage in a streaming manner.