search

Ouranosinc / CanarieAPI

CanarieAPI: Self describing REST service for Canarie registry
https://canarieapi.readthedocs.io/
ISC License
0 stars 1 forks source link

[BUG] Monitoring and access logs values not updating #21

Open fmigneault opened 1 year ago

fmigneault commented 1 year ago

Describe the bug

In some situations, monitoring seems to incorrectly update its latest update time. https://github.com/bird-house/birdhouse-deploy/pull/284#issuecomment-1515370247

There also appears to be other cases were "access" logs (when monitoring paths of the services are accessed) are not updating the services and platform invocations and lastAccess values, although lastInvocationsUpdate and lastStatusUpdate do seem to udpate, indicating that the crons are properly running, but something is not updated/stored correctly. See /canarie/server/platform/stats and /canarie/node/service/stats , or any other service.

To Reproduce

  1. Start a birdhouse-deploy instance and wait for a minute or two.
  2. Go to /canarie/node/service/status endpoint of the instance to check lastUpdateStatus.

Expected behavior

lastUpdateStatus should update with the cron job set for every minute.

Screenshots n/a

Environment

fmigneault commented 2 months ago

Maybe related. Bug detected in one of the instance logs: Seen using version 1.0.0. 

canarie-api-cron             | [2024-07-25 00:05:02,324] [25] [INFO] canarieapi.app_object : Loading custom configuration from /config/docker_configuration.py
canarie-api-cron             | [2024-07-25 00:05:02,325] [25] [INFO] canarieapi.app_object : Cron job for parsing server log
canarie-api-cron             | [2024-07-25 00:05:02,325] [25] [INFO] canarieapi.app_object : Loading configuration
canarie-api-cron             | [2024-07-25 00:05:02,326] [25] [INFO] canarieapi.app_object : Database not defined. Establishing connection...
canarie-api-cron             | [2024-07-25 00:05:02,327] [25] [DEBUG] canarieapi.app_object : Using configured filename: [/data/stats.db]
canarie-api-cron             | [2024-07-25 00:05:02,327] [25] [DEBUG] canarieapi.app_object : Setup database connection with filename: [/data/stats.db]
canarie-api-cron             | [2024-07-25 00:05:02,327] [25] [DEBUG] canarieapi.app_object : Initialize database with filename: [/data/stats.db]
canarie-api-cron             | [2024-07-25 00:05:02,327] [25] [DEBUG] canarieapi.app_object : Skipping database initialization: [/data/stats.db] (already exists)
canarie-api-cron             | [2024-07-25 00:05:02,328] [25] [INFO] canarieapi.app_object : Loading log file : /logs/access_file.log
canarie-api-cron             | Traceback (most recent call last):
canarie-api-cron             |   File "<string>", line 1, in <module>
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/logparser.py", line 110, in cron_job
canarie-api-cron             |     update_db(parse_log(access_log_fn))
canarie-api-cron             |               ^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/logparser.py", line 60, in parse_log
canarie-api-cron             |     if last_access is None or dt_parse(records["datetime"]) > last_access:
canarie-api-cron             |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             | TypeError: can't compare offset-naive and offset-aware datetimes
canarie-api-cron             | [2024-07-25 00:05:07,261] [26] [ERROR] canarieapi.app_object : Exception occurs while trying to check status of node.Finch-public
canarie-api-cron             | Traceback (most recent call last):
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
canarie-api-cron             |     response = conn.getresponse()
canarie-api-cron             |                ^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 461, in getresponse
canarie-api-cron             |     httplib_response = super().getresponse()
canarie-api-cron             |                        ^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/http/client.py", line 1378, in getresponse
canarie-api-cron             |     response.begin()
canarie-api-cron             |   File "/usr/local/lib/python3.11/http/client.py", line 318, in begin
canarie-api-cron             |     version, status, reason = self._read_status()
canarie-api-cron             |                               ^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/http/client.py", line 279, in _read_status
canarie-api-cron             |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
canarie-api-cron             |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
canarie-api-cron             |     return self._sock.recv_into(b)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/ssl.py", line 1311, in recv_into
canarie-api-cron             |     return self.read(nbytes, buffer)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/ssl.py", line 1167, in read
canarie-api-cron             |     return self._sslobj.read(len, buffer)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             | TimeoutError: The read operation timed out
canarie-api-cron             | 
canarie-api-cron             | 
canarie-api-cron             | The above exception was the direct cause of the following exception:
canarie-api-cron             | 
canarie-api-cron             | 
canarie-api-cron             | Traceback (most recent call last):
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
canarie-api-cron             |     resp = conn.urlopen(
canarie-api-cron             |            ^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 844, in urlopen
canarie-api-cron             |     retries = retries.increment(
canarie-api-cron             |               ^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
canarie-api-cron             |     raise reraise(type(error), error, _stacktrace)
canarie-api-cron             |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
canarie-api-cron             |     raise value
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 790, in urlopen
canarie-api-cron             |     response = self._make_request(
canarie-api-cron             |                ^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 538, in _make_request
canarie-api-cron             |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
canarie-api-cron             |     raise ReadTimeoutError(
canarie-api-cron             | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='host-140-216.rdext.crim.ca', port=443): Read timed out. (read timeout=5)
canarie-api-cron             | 
canarie-api-cron             | 
canarie-api-cron             | During handling of the above exception, another exception occurred:
canarie-api-cron             | 
canarie-api-cron             | 
canarie-api-cron             | Traceback (most recent call last):
canarie-api-cron             |   File "<string>", line 1, in <module>
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/monitoring.py", line 129, in cron_job
canarie-api-cron             |     monitor()
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/utility_rest.py", line 302, in retry
canarie-api-cron             |     return func(*args, **kwargs)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/monitoring.py", line 62, in monitor
canarie-api-cron             |     status, message = check_service(request=test_dic["request"],
canarie-api-cron             |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/canarieapi/monitoring.py", line 99, in check_service
canarie-api-cron             |     resp = requests.request(**default_request)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
canarie-api-cron             |     return session.request(method=method, url=url, **kwargs)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
canarie-api-cron             |     resp = self.send(prep, **send_kwargs)
canarie-api-cron             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
canarie-api-cron             |     r = adapter.send(request, **kwargs)
canarie-api-cron             |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
canarie-api-cron             |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
canarie-api-cron             |     raise ReadTimeout(e, request=request)
canarie-api-cron             | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='host-140-216.rdext.crim.ca', port=443): Read timed out. (read timeout=5)
canarie-api-cron             | canarie-api: found version [2.4.0] release time [2024-07-09T16:13:07Z] from [github: Ouranosinc/cowbird]
canarie-api-cron             | canarie-api: found version [5.6.1] release time [2024-06-14T04:02:53Z] from [github: crim-ca/weaver]
canarie-api-cron             | canarie-api: found version [0.9.2] release time [2022-07-19T21:42:25Z] from [github: bird-house/finch]
canarie-api-cron             | canarie-api: found version [2.25.2--v2024.06.25-kartoza] release time [2024-07-08T15:47:45.053567Z] from [docker: pavics/geoserver]
canarie-api-cron             | canarie-api: found version [0.5_dev] release time [2020-01-21T20:20:21.953117Z] from [docker: pavics/hummingbird]
canarie-api-cron             | canarie-api: found version [4.1.5-20240426] release time [2024-04-26T19:27:03.530615Z] from [docker: pavics/jupyterhub]
canarie-api-cron             | canarie-api: found version [3.38.0] release time [2023-11-28T17:50:36Z] from [github: Ouranosinc/Magpie]
canarie-api-cron             | canarie-api: found version [0.18.1] release time [2023-05-25T18:46:31Z] from [github: Ouranosinc/raven]
canarie-api-cron             | canarie-api: found version [4.6.18-unidata-2022-01] release time [2023-04-11T18:46:40.742406Z] from [docker: pavics/thredds-docker]
canarie-api-cron             | canarie-api: found version [0.9.0] release time [2023-02-08T15:05:45Z] from [github: bird-house/twitcher]
canarie-api-cron             | canarie-api: found version [5.6.1] release time [2024-06-14T04:02:53Z] from [github: crim-ca/weaver]