cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data
https://cagov.github.io/caldata-mdsa-caltrans-pems/
MIT License
7 stars 0 forks source link

Investigate+fix mart unloading failure #478

Open JamesSLogan opened 1 week ago

JamesSLogan commented 1 week ago

During a CI run, one of our marts failed to unload to s3 due to a large file size:

Database Error in model imputation__detector_summary (models/marts/imputation/imputation__detector_summary.sql)
  100083 (22000): Max file size (536870912) exceeded for unload single file mode.
    We recommend that you increase the max file size parameter, or disable single-file mode 
    in the unload command and combine the unloaded files into a single file after you download them.

It's not clear why this appears to only have happened once, but this should be investigated and fixed so that it doesn't eventually block CI/CD.