search

sqlitebrowser / dbhub.io

A "Cloud" for SQLite databases. Collaborative development for your data. 😊
https://dbhub.io
GNU Affero General Public License v3.0
372 stars 39 forks source link

Weird error to investigate #40

Closed MKleusberg closed 7 years ago

MKleusberg commented 7 years ago

I've just tired to access https://dev1.dbhub.io/slawomir.budzynski/NewDB.db (at around 14:00 UTC+2, as a guest user) and got a database not available error. After reloading it worked fine. Any chance to find out what went wrong there?

justinclift commented 7 years ago

Interesting. Just did a grep for "slaw" through the "request.log", generated by our current request logging system.

For that time range, it shows:

62.210.129.246:60620 - - [2017-05-05T11:57:59.635329616Z] "GET /slawomir.budzynski HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
62.210.129.246:60620 - - [2017-05-05T11:58:54.483635158Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
62.210.129.246:60620 - - [2017-05-05T11:58:54.48372408Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
62.210.129.246:60620 - - [2017-05-05T11:59:03.26838308Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"

On the webUI console output (a different thing), the only output shown matching the above times is this:

2017/05/05 11:58:54 Error: http2: stream closed

That http2: stream closed error comes from the in-built Golang libraries, and isn't something I've studied to great depth yet. So far I've just assumed things like this meant some kind of change client side (closed browser tab, etc), but it sounds like that's not the case here.

Let's mark this as needing follow up, and we can circle back to it later as it definitely sounds important.

MKleusberg commented 7 years ago

Hmm... strange. This error is probably not related to that particular issue because I got at least the error page. I'll update this issue if I can reproduce the error.

justinclift commented 7 years ago

Yeah. It sounds like we need more logging code on the page for viewing databases for things that don't go correctly.

If you hit it again, please screenshot it. I'm kind of wondering if the "database not available" thing is coming from the PostgreSQL side or the SQLite-code-reading side. It might be meaning we need to adjust our PG config to allow more simultaneous operations, but I'd be kind of surprised. It's a dev server and we don't get thousands of hits per seconds (yet :wink:).

MKleusberg commented 7 years ago

Will do :smiley:

While playing around with this I've just stumbled upon a different error message but the steps to reproduce it were close to the ones for the other message. What I've tried now is opening a number of browser tabs for different databases and users and then reload them all with one mouse click. Here's what I got for 6 of the 11 tabs: screenshot_20170505_160711

justinclift commented 7 years ago

Interestingly, and probably related, this is showing up in the console from just a few minutes ago:

2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'kales2019/wigle'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'kaushal.rajkotia/sevakfc.sqlite'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'vika08/Product.db'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'eng_awlaqi/mydb.db'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'cikal/demo2.db'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'jccultima123/database.db'. Error: conn is busy
2017/05/05 13:59:15 Error looking up star count for database. User: '' DB: 'slawomir.budzynski/NewDB.db'. Error: conn is busy

Looking at the request log for the matching time range shows a single IP address doing those requests. Doing a grep of the request log for that IP address (which seems like a Tor exit node according to nslookup):

178.32.53.94:48652 - - [2017-05-05T13:50:15.244042085Z] "GET / HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:34.553677403Z] "GET /justinclift HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:35.123138991Z] "GET /vika08 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:35.569295175Z] "GET /eng_awlaqi HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:36.207789524Z] "GET /koladiyagoral HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:36.788476219Z] "GET /test4 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:37.392415867Z] "GET /test2 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:37.874767657Z] "GET /test1 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:38.399652609Z] "GET /test0 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:39.146405776Z] "GET /kaushal.rajkotia HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:41.021635318Z] "GET /kales2019 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:41.931286128Z] "GET /younes201 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:42.130102121Z] "GET /test3 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:42.808441861Z] "GET /slawomir.budzynski HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:43.251033064Z] "GET /cikal HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:43.931207704Z] "GET /martintest1 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:44.55384912Z] "GET /jccultima123 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:51:45.299738448Z] "GET /mkleusberg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:02.838099213Z] "GET /justinclift/Bike%20Counts%20In%20Edinburgh.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:03.312718512Z] "GET /justinclift/Assembly%20Election%202017.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:04.02020468Z] "GET /justinclift/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:04.349084679Z] "GET /justinclift/DB4S%20download%20stats.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:04.888755111Z] "GET /justinclift/spatial56day.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:09.271613319Z] "GET /vika08/Product.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:10.760260645Z] "GET /eng_awlaqi/mydb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:12.026118115Z] "GET /koladiyagoral/db_recipes HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:13.440081987Z] "GET /test4/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:15.090681508Z] "GET /test2/DB4S%20download%20stats.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:18.762556795Z] "GET /test1/DB4S%20download%20stats.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:19.356981181Z] "GET /test1/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:23.474820236Z] "GET /test0/DB4S%20download%20stats.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:23.930448154Z] "GET /test0/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:25.351068004Z] "GET /test0/demo2.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:48.453361127Z] "GET /kaushal.rajkotia/sevakfc.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:49.724035289Z] "GET /kales2019/wigle HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:53.22049822Z] "GET /younes201/msgDb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:54.699947154Z] "GET /test3/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:52:56.199669495Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:53:12.76509391Z] "GET /martintest1/Belfast%20Bikes%20Docking%20Stations.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:53:16.36335537Z] "GET /jccultima123/database.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:48652 - - [2017-05-05T13:53:17.619807109Z] "GET /mkleusberg/issue946.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:22.633250628Z] "GET / HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:25.736782481Z] "GET /vika08 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:26.206196792Z] "GET /eng_awlaqi HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:27.144742263Z] "GET /koladiyagoral HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:28.519163785Z] "GET /vika08/Product.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:30.386023356Z] "GET /eng_awlaqi/mydb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:31.371051109Z] "GET /koladiyagoral/db_recipes HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:39.945331274Z] "GET /vika08/Product.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:40.239730953Z] "GET /eng_awlaqi/mydb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:40.582667057Z] "GET /koladiyagoral/db_recipes HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:43.355989366Z] "GET /kaushal.rajkotia HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:43.597000981Z] "GET /kales2019 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:44.149311396Z] "GET /younes201 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:44.240053017Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:44.240059183Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:44.240105459Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:45.928003202Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:46.113932247Z] "GET /slawomir.budzynski HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:46.448695533Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:46.536784224Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:48.551238166Z] "GET /jccultima123 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:49.623615232Z] "GET /cikal HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:52.40592102Z] "GET /kaushal.rajkotia/sevakfc.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:53.915051678Z] "GET /kales2019/wigle HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:56.32610892Z] "GET /younes201/msgDb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:57.669668283Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:58:59.261966468Z] "GET /jccultima123/database.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:00.583078653Z] "GET /cikal/demo2.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.256682211Z] "GET /kaushal.rajkotia HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347506225Z] "GET /kales2019 HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347541729Z] "GET /kales2019/wigle HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347502636Z] "GET /kaushal.rajkotia/sevakfc.sqlite HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347947378Z] "GET /vika08/Product.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347951945Z] "GET /eng_awlaqi/mydb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347761676Z] "GET /cikal/demo2.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.34832502Z] "GET /jccultima123/database.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.347702533Z] "GET /younes201/msgDb.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.348236307Z] "GET /slawomir.budzynski/NewDB.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:15.436766867Z] "GET /koladiyagoral/db_recipes HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:18.58464256Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:18.673153218Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:18.673181656Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:22.715478045Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:22.803883261Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:22.803907898Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:22.923204371Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.380048321Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.468283612Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.468295315Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.578354818Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.892903032Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.98115071Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:23.981167956Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:25.080285241Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:25.168729644Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:25.168719286Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:37.549524925Z] "GET /jccultima123/database.db HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:38.191252762Z] "GET /images/sqlitebrowser.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:38.281682262Z] "GET /images/rackspace.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
178.32.53.94:56992 - - [2017-05-05T13:59:38.28169562Z] "GET /images/auth0.svg HTTP/2.0" "-" "-" "" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
chrisjlocke commented 7 years ago

When it displays that webpage, does it display a template first, then 'javascript' it out? When I view that page, for about a second it displays a template?

image

MKleusberg commented 7 years ago

Just reloaded all of the 11 tabs again and here we go: 3 of them work fine. 1 prints "Database query failed". 7 print "The requested database doesn't exist".

The last one is the message I got earlier.

justinclift commented 7 years ago

@MKleusberg Ahhh, I guess that was you then. :smile:

That error about the social stats is definitely helpful. The social stats are stored in PostgreSQL, so it looks like a problem in the way we're interacting with that. Some potential things to look into spring to mind:

Gimme 2 minutes, and try again. 😁

MKleusberg commented 7 years ago

@chrisjlocke I get that too. But it's the same for some other AngularJS pages too...

@justinclift Yep, that was me :wink:

justinclift commented 7 years ago

@chrisjlocke Yep, it loads a template with some of the values written in, then AngularJS fires and updates things browser side (using Javascript).

The "{{ stuff }}" tags are AngularJS placeholders in a template. eg:

    https://github.com/sqlitebrowser/dbhub.io/blob/master/webui/templates/database.html#L105

justinclift commented 7 years ago

Hmm, postgresql.conf file on dev1 has:

max_connections = 100

So it's probably either us leaking connections or doing something else not quite right in our driver connection to it. PG itself would consider 100 connections a fairly no real effort required situation (depending on the SQL being run through them of course).

Ahhh well, this is what a dev server is for. Finding things out and fixing them. 😬

justinclift commented 7 years ago

For getting this fixed, it might be a weekend thing. I'm not feeling too great atm so crawled into bed... but I have a laptop nearby and keep checking email. Doh. Can anyone say "internet addiction"? 😈

justinclift commented 7 years ago

Ahhh, I suspect the problem is more that we're just opening a single database connection, instead of several of them + multiplexing (eg using connection pooling).

That should be pretty solvable (but not today). I'll need to read more on the pgx driver we're using.

justinclift commented 7 years ago

Yep, looks like this is the thing to investigate:

    https://godoc.org/github.com/jackc/pgx#ConnPool

My brain isn't up for it now, but hopefully will be tomorrow. :smile:

justinclift commented 7 years ago

... and it looks really simple to do a basic implementation: https://github.com/sqlitebrowser/dbhub.io/commit/d0b7e8bcbb7000df9ed5d56f8b2479120d6e5b84

So, just applied that to the dev1 server.

@MKleusberg Want to try the simultaenous connection thing again? I'm suspecting we'll need to raise the (default 5) connections higher, but I'd like to see if your previous approach still breaks it at 5. :wink:

justinclift commented 7 years ago

As a test, I just did the "Reload all browser tabs" thing on dev1 with 20 open database connections. No errors showed up in the console, and all the databases loaded.

That's sounding positive... :smile:

justinclift commented 7 years ago

And again, this time after clearing out memcache just to make sure that's not masking things. Still all ok.

Hopefully that means we're on the right track.

justinclift commented 7 years ago

This seems useful for testing with: https://en.wikipedia.org/wiki/Siege_(software)

It can take a list of URLs to run against, which meant passing it all of dev1's public DB's seems useful:

https://dev1.dbhub.io/mkleusberg/pr921.db
https://dev1.dbhub.io/justinclift/spatial56day.db
https://dev1.dbhub.io/justinclift/Bike%20Counts%20In%20Edinburgh.sqlite
https://dev1.dbhub.io/justinclift/Assembly%20Election%202017.sqlite
https://dev1.dbhub.io/justinclift/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/justinclift/DB4S%20download%20stats.sqlite
https://dev1.dbhub.io/vika08/Product.db
https://dev1.dbhub.io/eng_awlaqi/mydb.db
https://dev1.dbhub.io/koladiyagoral/db_recipes
https://dev1.dbhub.io/test4/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/test2/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/test2/DB4S%20download%20stats.sqlite
https://dev1.dbhub.io/test1/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/test1/DB4S%20download%20stats.sqlite
https://dev1.dbhub.io/test0/demo2.db
https://dev1.dbhub.io/test0/DB4S%20download%20stats.sqlite
https://dev1.dbhub.io/test0/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/kaushal.rajkotia/sevakfc.sqlite
https://dev1.dbhub.io/kales2019/wigle
https://dev1.dbhub.io/younes201/msgDb.db
https://dev1.dbhub.io/test3/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/slawomir.budzynski/NewDB.db
https://dev1.dbhub.io/cikal/demo2.db
https://dev1.dbhub.io/martintest1/Belfast%20Bikes%20Docking%20Stations.sqlite
https://dev1.dbhub.io/jccultima123/database.db
https://dev1.dbhub.io/mkleusberg/issue946.db

Then ran it for a while after clearing memcache on dev1:

$ siege -b -i -f siege_urls.txt

No errors at all showed up on dev1's webUI console, so I think we're ok for now. 😁

MKleusberg commented 7 years ago

Just tried it with my patent-pending 'Reload all browser tabs' denial of service routine and got no errors. I'll close this issue then :smiley: Thanks for fixing it!