The latest sunk file is overwritten locally when resuming a stream.

streamingfast / substreams-sink-files

Binary application to consume your Substreams and output it's data out to files format (JSON, CSV, etc.)

Apache License 2.0

2 stars 4 forks source link

The latest sunk file is overwritten locally when resuming a stream. #3

Open AngeloCa opened 1 year ago

AngeloCa commented 1 year ago

Let's assume a stream is sunk until block 100000, as stated in issue #2, the latest file is 80000-90000.jsonl.

When resuming the sinking process to get the newest data (giving a new stop range e.g 120000), the latest file from the previous extraction (80000-90000.jsonl) is overwritten by an empty file.

Note: this behavior only appears when the output file path is a local path. (Sinking data to S3 doesn't overwrite the latest file)

dkdocs commented 6 months ago

Any resolution on this issue?

ArnaudBger commented 5 months ago

Hey @dkdocs it should be fixed, on the develop branch !