I'm not sure why the "insert into DB via batches in transactions" strategy wasn't working, but I tried many fixes and wasn't successful.
So, this takes another approach: insert them all at once via COPY rather than INSERTs, after manually CSV-ifying the records.
This seems to work pretty well, doesn't take longer than our previous strat, and isn't flaky.
This "ingest" strategy in general is temporary, so I'm not worried about the scalability of this solution.
I'm not sure why the "insert into DB via batches in transactions" strategy wasn't working, but I tried many fixes and wasn't successful.
So, this takes another approach: insert them all at once via COPY rather than INSERTs, after manually CSV-ifying the records. This seems to work pretty well, doesn't take longer than our previous strat, and isn't flaky.
This "ingest" strategy in general is temporary, so I'm not worried about the scalability of this solution.