search

ProjectSidewalk / SidewalkWebpage

Project Sidewalk web page
http://projectsidewalk.org
MIT License
83 stars 24 forks source link

sidewalk-chicago down, SQL error #3553

Closed yaomtc closed 4 months ago

yaomtc commented 4 months ago

onError Happened

Execution exception[[SQLException: Timed out waiting for a free available connection.]]

https://sidewalk-chicago.cs.washington.edu/

yaomtc commented 4 months ago

It's back now.

jonfroehlich commented 4 months ago

Thank you. I restarted the server and it should be back up now.

For the past two weeks, we have intermittent but serious server failures. We have invested significant resources in debugging but cannot seem to figure out the problem. Keeping this ticket open until we figure out the fix.

yaomtc commented 4 months ago

Oops. I'm getting the error with Explore now.

yaomtc commented 4 months ago

It appears to be working again, for now.

misaugstad commented 4 months ago

This has finally been fixed! Here's a quick summary of what happened...

The issue came out this commit: d23937d12105ffe84a44aebef658aab8040eecb8

The issue being addressed by that commit: In Seattle, sending data to the back-end from the Explore page was timing out before getting the response back to the front end. This was happening when users placed a label, because we were sending info to SciStarter on how long users spent to add a label, which required looking at the interaction logs. This query was taking 45+ seconds, and it was being done synchronously.

I thought that this query should be done async, since the timing of the data being sent to SciStarter doesn't matter to our front-end. So I made it async! But apparently that didn't play well with db queries in Slick 2...

This is no longer a problem though, since I later added a table with a much smaller subset of the interactions data that we can use to make these calculations. So the queries work fine synchronously as they take less than half a second instead of 45 seconds.

We should also be able to do something like what I tried to do once we upgrade Slick from 2 to 3, which natively supports Futures.