search

ecohealthalliance / open-rvfcast

Wellcome Open RVFCast project repository
Other
0 stars 1 forks source link

Arrow error in forecast anomaly dataset #70

Closed emmamendelsohn closed 7 months ago

emmamendelsohn commented 7 months ago 
> forecasts <- arrow::open_dataset("data/forecast_anomalies/forecast_anomaly_2010-03-27.gz.parquet" ) |> collect()
Error in `compute.Dataset()`:
! IOError: Unexpected end of stream: Page was smaller (5077) than expected (5240)
Run `rlang::last_trace()` to see where the error occurred.

This was one of the files that I regenerated for #68. Seems to be stochastic as some of the regenerated files are now working. Looks like metadata isn't matching actual file size.

emmamendelsohn commented 7 months ago

this has been solved by updating to arrow 14.0.0.2 and regenerating these files