search

developmentseed / tecnico-energy-app

https://dashboard-ds-peach.vercel.app/
0 stars 0 forks source link

Add checksum logic to avoid redundant ingestions #63

Closed alukach closed 2 months ago

alukach commented 2 months ago

To minimize the amount of data being sent to our database (and thus avoiding Vercel limits), this PR adds a new table that stores the checksum of the XLSX and geojson.gz files. Upon ingestion, we look to see if these checksums are stored in the DB, and if so, skip ingestion. Checksums are performed on a data stream to avoid holding the entire file in memory.

vercel[bot] commented 2 months ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment | Name | Status | Preview | Comments | Updated (UTC) | | :--- | :----- | :------ | :------- | :------ | | **dashboard-ds** | ⬜️ Ignored ([Inspect](https://vercel.com/dashboard-in/dashboard-ds/45gzJtBocXskJgKMKCekki81r7xb)) | | | May 17, 2024 1:56am |