Derek and I did some investigation and we found that the problem arrises after a timeout occurs. rest.rb is using Timeout.timeout to kill the method call if it takes longer than 1 second. This works in ruby by making the thread raise an exception wherever it is in the stack. Most of our time was spent trying to reproduce this consistently, which we were finally able to do once we added a sleep to cause this timeout to fire.
After discovering this, we realized that the destruction of temporary databases was not happening in a begin/rescue, and that a lot of the prepared statements were not being closed. After adding code to close these connections in a begin/ensure, we were no longer able to reproduce the failure case any more no matter how many timeout errors we triggered.
We also reverted the previous change of creating a new Database object every time as it is not needed. This patch also seems to solve some of the memory leaks (but not all of them).
Derek and I did some investigation and we found that the problem arrises after a timeout occurs. rest.rb is using
Timeout.timeoutto kill the method call if it takes longer than 1 second. This works in ruby by making the thread raise an exception wherever it is in the stack. Most of our time was spent trying to reproduce this consistently, which we were finally able to do once we added a sleep to cause this timeout to fire.After discovering this, we realized that the destruction of temporary databases was not happening in a begin/rescue, and that a lot of the prepared statements were not being closed. After adding code to close these connections in a begin/ensure, we were no longer able to reproduce the failure case any more no matter how many timeout errors we triggered.
We also reverted the previous change of creating a new Database object every time as it is not needed. This patch also seems to solve some of the memory leaks (but not all of them).
In summary: TICKETS PLEASE!