Use remote file URI instead of relative path for remote archived files

deployment-gap-model-education-fund / deployment-gap-model

ETL code for the Deployment Gap Model Education Fund

https://www.deploymentgap.fund/

MIT License

6 stars 2 forks source link

Use remote file URI instead of relative path for remote archived files #290

Closed bendnorman closed 10 months ago

bendnorman commented 11 months ago

I noticed archive paths for files stored in github were the absolute local path where the files stored on GCS were paths relative to /app/data/data_cache/. This felt awkward so I decided to use the full GCS URI instead for remote files.

There's for sure a more graceful way to handle this caching behavior (PUDL's layered cache or maybe fsspec) but I figured this was good enough for now.