parquet-ans-processor was OOM killed. The rootcause was that the version tracker in parquet gap detector keeping too many versions in there due to the fact that there is no structs to be processed until later txn_version in mainnet.
The workaround I adopted here is to return empty parquet upload result when the buffer is empty that we didn't actually upload to avoid parquet gap detector to use so much memory to keep the map size big until there is an actual upload.
returning empty parquet result meaning we invoke update_processor_status function so that processor_status also gets updated more frequently.
[Uploading ans-trace.pdf…]()
Test Plan
tested locally that it now updates processor status early enough.
Description
parquet-ans-processor was OOM killed. The rootcause was that the version tracker in parquet gap detector keeping too many versions in there due to the fact that there is no structs to be processed until later txn_version in mainnet.
The workaround I adopted here is to return empty parquet upload result when the buffer is empty that we didn't actually upload to avoid parquet gap detector to use so much memory to keep the map size big until there is an actual upload. returning empty parquet result meaning we invoke update_processor_status function so that processor_status also gets updated more frequently.
[Uploading ans-trace.pdf…]()
Test Plan
tested locally that it now updates processor status early enough.