Temp tables not always being cleaned up

internetofwater / nldi-crawler

Network Linked Data Index Crawler

https://labs.waterdata.usgs.gov/about-nldi/

Other

6 stars 9 forks source link

Temp tables not always being cleaned up #127

Closed EthanGrahn closed 2 years ago

EthanGrahn commented 2 years ago

The crawler creates a temporary table during ingestion and is supposed to drop the table once complete. I've noticed that there are temp tables in our deployed database that did not get cleaned up. (different issue from https://github.com/ACWI-SSWD/nldi-db/issues/36) There needs to be investigation on what causes the cleanup to fail.

EthanGrahn commented 2 years ago

As I was looking through the code today, I discovered that the crawler does not have any functionality that deletes the temp tables. It drops the _old table and shuffles around the name of the temp and original tables.

EthanGrahn commented 2 years ago

I'll remove the bug tag since this is not really a bug.

EthanGrahn commented 2 years ago

I discovered today that the temp table is properly removed during an initial ingestion, but sticks around if ingested a second time.