Automatically close prepared statements after execute

[kevinlang]

Right now we rely entirely on NIF destructor calling behavior which is apparently unreliable for cleaning up our queries/statements. One thing I see the myxql driver do is automatically close any unnamed queries after execute.

As part of this we can change our implementation of handle_close to actually close the query given.

[warmwaffles]

I think that is fine. It should not affect the streaming implementation since handle_close isn't called until it's done.

rely entirely on NIF destructor calling behavior which is apparently unreliable

Is there a thread or bug report to link to for this? I'm interested in it because if I make another NIF in the future, I'd like to know.

[kevinlang]

I think unreliable was the wrong word - I think "not immediate" is the better phrasing.

This link is a good one: http://erlang.org/pipermail/erlang-questions/2020-November/100150.html

As a general rule: never use finalizers for the primary resource reclamation path

(emphasis mine)

[warmwaffles]

Interesting. Good to know.

[warmwaffles]

I'm looking at the old https://github.com/mmzeeman/esqlite repo trying to figure out how they solved the prepared statement releasing, and I don't think they really do anything about it other than just letting the GC eventually take it.

Looking further in that pipemail this was brought up

http://erlang.org/pipermail/erlang-questions/2020-November/100157.html

I fully agree with Jesper about not relying on GC for resource reclamation. An additional risk is that the resource does not get GC’ed at all if its reference has “leaked” away onto the heap of another process, like for example a logger or an io-server process.

For the error case when a resource owning process crashes, a good alternative can be to use https://erlang.org/doc/man/erl_nif.html#enif_monitor_process and reclaim the resource in the down callback.

But then this was also brought up

http://erlang.org/pipermail/erlang-questions/2020-November/100134.html

Prior to OTP 22.0 resource destructors were called directly by the thread that happened to decrease the reference counter down to zero.

From OTP 22.0 resource destructors are scheduled. This in order to ensure a stable known execution context for the destructor callback. So it may not have executed when erlang:garbage_collect returns, but will very soon (unless you have a very overloaded system or a bug preventing ref counter from reaching zero).

To not get dependent on the garbage collector to release large memory blocks (or other limited OS resources) you can solve it (like a lot of other problems) with an extra level of indirection. Keep the resource object small and allocate the large block separately. Then you can deallocated the big block whenever you want and mark the resource as dead with a NULL pointer for example.

Here's the whole thread http://erlang.org/pipermail/erlang-questions/2020-November/thread.html#100131

I'm going to keep exploring this. Also this is a really hard thing to test and I'm not sure how I would go about it. @kevinlang if you have any ideas I'm all ears.