since about 1 hour this pops up

[Manuel18071991]

2019-12-28 16:39:20.138652

Traceback: Traceback (most recent call last): File "/home/toconnell/kdm-manager/v3/session.py", line 85, in wrapper  return func(self, *args, **kwargs) File "/home/toconnell/kdm-manager/v3/session.py", line 199, in init  self.session = mdb.sessions.find_one({"_id": session_id}) File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 1269, in find_one  for result in cursor.limit(-1): File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1225, in next  if len(self.data) or self._refresh(): File "/usr/local/lib/python2.7/dist-packages/pymongo/cursor.py", line 1117, in _refresh  self.session = self.collection.database.client._ensure_session() File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 1598, in _ensure_session  return self.__start_session(True, causal_consistency=False) File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 1551, in start_session  server_session = self._get_server_session() File "/usr/local/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 1584, in _get_server_session  return self._topology.get_server_session() File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 429, in get_server_session  None) File "/usr/local/lib/python2.7/dist-packages/pymongo/topology.py", line 200, in _select_servers_loop  self._error_message(selector)) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused

[toconnell]

@Manuel18071991 Hey!

Thanks for checking in with this! I was out and couldn't get to it until just now. It was actually the API that was affected, so I've got a ticket open there for this issue: https://github.com/theLaborInVain/kdm-manager-api/issues/13

Given the nature of the failure, the webapp should have told you that the application was down for maintenance and alerted me, but that failed to work as intended, so I will leave this issue open to track the research on that (which will be separate from the API issue that was the root cause).

What a mess!

Anyway, yeah: thanks again for taking the time to write in. I am really sorry about this!

[toconnell]

So, instead of this:

Database failure should consider the user not logged in and basically send them back to the login screen.

[toconnell]

OK--I just simulated this in dev and ultimately, here's how it goes for in-progress sessions:

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/cgitb.py", line 271, in __call__
    self.handle((etype, evalue, etb))
  File "/usr/lib/python2.7/cgitb.py", line 291, in handle
    self.file.write(doc + '\n')
IOError: [Errno 32] Broken pipe

Original exception was:
Traceback (most recent call last):
  File "/home/toconnell/kdm-manager/v3/index", line 50, in <module>
    utils.record_response_time(current_view, stop-start)
  File "/home/toconnell/kdm-manager/v3/utils.py", line 289, in record_response_time
    mdb.response_times.insert({"created_on": datetime.now(),"view":view_name,"time":tdelta.total_seconds()})
  File "build/bdist.linux-x86_64/egg/pymongo/collection.py", line 2467, in insert
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "build/bdist.linux-x86_64/egg/pymongo/mongo_client.py", line 823, in _get_socket
  File "build/bdist.linux-x86_64/egg/pymongo/topology.py", line 214, in select_server
  File "build/bdist.linux-x86_64/egg/pymongo/topology.py", line 189, in select_servers
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
127.0.0.1 - - [28/Dec/2019 21:21:40] CGI script exit status 0x100

...which will show the spinner forever and then, eventually, kick the user back to the sign in (where, if they try to sign in, they will see the downtime warning).

[toconnell]

@Manuel18071991 In the future, these types of failures will be handled correctly by the webapp. The change will go out in the next release of the Manager (which I'm working on now and which will be out soon).

The root cause is still being worked on over here: https://github.com/theLaborInVain/kdm-manager-api/issues/13

Thanks again for writing in! I really appreciate it.

[toconnell]

toconnell@mona:~/kdm-manager/v3$ git diff session.py
diff --git a/v3/session.py b/v3/session.py
index 4d35590..d1fa156 100644
--- a/v3/session.py
+++ b/v3/session.py
@@ -14,6 +14,9 @@ from string import Template
 import sys
 import traceback

+# third party
+import pymongo
+
 import admin
 import api
 import assets
@@ -196,7 +199,11 @@ class Session:
                 self.set_cookie=True

         if session_id is not None:
-            self.session = mdb.sessions.find_one({"_id": session_id})
+            try:
+                self.session = mdb.sessions.find_one({"_id": session_id})
+            except pymongo.errors.ServerSelectionTimeoutError:
+                self.logger.error('The database is unavailable!')
+                self.session = None
             if self.session is None:
                 sign_in()
             else: