SFDO-Tooling / Snowfakery

A tool for generating fake data that has relations between tables.
BSD 3-Clause "New" or "Revised" License
141 stars 32 forks source link

Clean up temporary directories after closing iterator #1029

Closed prescod closed 5 months ago

prescod commented 5 months ago

Fixes #901

What was happening on Windows is that that the temporary directory was being garbage collected before the open file handle temporary file inside of it. This was probably also handling on *ix, but it was harmless.

On Windows, the temporary directory object needs to live a bit longer than the local tempfile, which needs to live a bit longer than than the SQL Alchemy connection to the tempfile.

So I use an ExitStack to inject cleanup code which is run before the connection is closed and the file handle is released.

All of this is triggered by the garbage collector letting go of the iterator when the recipe is finishing up.

It might be better to do a deterministic, explicit, top-down cleanup, but that should probably be a different PR.