Overview ticket - current bugs/warnings/errors on main

stephanie0x00 commented 3 months ago

The purpose of this ticket is to give an overview on the current bugs/errors/warning that are found in KAT on the current main. Based on this ticket we can determine if/which items need to be picked up (this sprint).

Current bugs in main (commit a613e82d966b83a13c7669124e2440c164f9851e (HEAD -> main, origin/main, origin/HEAD)):

The raw files for KAT FindingTypes seem to (always) contain the output of the KAT Findings database. Unsure if this is a bug or intented behaviour.
Sometimes RAW files for failed tasks cannot be downloaded. #2864
Setting clearance level on object is not shown in the UI. As a user you have to press the button twice before the chosen clearance level is shown back correctly to the user. #2703
ValueError for staticfiles with DEBUG=False prevents UI from loading: #2973
Some boefjes in the Katalogus do not have a clearance level (CWE Finding Types, CVE Finding Types, External Database, KAT finding types, RetireJS Finding Types, Snyk finding types)

Error reading bcrypt version: #2675

bytes-1  | [2024-05-15 14:19:50 +0000] [8] [INFO] [on] Application startup complete.
bytes-1  | [2024-05-15 14:20:01 +0000] [8] [INFO] [h11_impl] 172.30.0.13:43630 - "GET /health HTTP/1.1" 200
bytes-1  | [2024-05-15 14:21:19 +0000] [8] [WARNING] [bcrypt] (trapped) error reading bcrypt version
bytes-1  | Traceback (most recent call last):
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/passlib/handlers/bcrypt.py", line 620, in _load_backend_mixin
bytes-1  |     version = _bcrypt.__about__.__version__
bytes-1  |               ^^^^^^^^^^^^^^^^^
bytes-1  | AttributeError: module 'bcrypt' has no attribute '__about__'
bytes-1  | [2024-05-15 14:21:19 +0000] [8] [INFO] [h11_impl] 172.30.0.3:52854 - "POST /token HTTP/1.1" 200
bytes-1  | [2024-05-15 14:21:19 +0000] [8] [INFO] [db] Connecting to database postgresql://bytes_app:***@postgres:5432/bytes with pool size 16...

Connection error (these connection errors also show up on the octopoes_api_worker, the logs for the octopoes_api_worker are in the second half of this snippet) (#2213)


bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [file_raw_repository] Writing raw data with id b38c050c-7193-43b9-
9250-dbe272d972ee to disk
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [sql_meta_repository] Added raw data [id=b38c050c-7193-43b9-9250-dbe272d972ee]
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [ERROR] [io_services_utils] _AsyncBaseTransport._produce() failed, aborting connection: error=ConnectionResetError(104, 'Connection reset by peer'); sock=<socket.socket fd=16, family=10, type=1, proto=6, laddr=('fc42:ca7::8', 54274, 0, 0)>; Caller's stack:
bytes-1  | Traceback (most recent call last):
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable
bytes-1  |     self._produce()
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce
bytes-1  |     num_bytes_sent = self._sigint_safe_send(self._sock,
bytes-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap
bytes-1  |     return func(*args, **kwargs)
bytes-1  |            ^^^^^^^^^^^^^^^^^^^^^
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send
bytes-1  |     return sock.send(data)
bytes-1  |            ^^^^^^^^^^^^^^^
bytes-1  | ConnectionResetError: [Errno 104] Connection reset by peer
bytes-1  | Traceback (most recent call last):
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable
bytes-1  |     self._produce()
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce
bytes-1  |     num_bytes_sent = self._sigint_safe_send(self._sock,
bytes-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap
bytes-1  |     return func(*args, **kwargs)
bytes-1  |            ^^^^^^^^^^^^^^^^^^^^^
bytes-1  |   File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send
bytes-1  |     return sock.send(data)
bytes-1  |            ^^^^^^^^^^^^^^^
bytes-1  | ConnectionResetError: [Errno 104] Connection reset by peer
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [io_services_utils] _AsyncTransportBase._initate_abort(): Initiating abrupt asynchronous transport shutdown: state=1; error=ConnectionResetError(104, 'Connection reset by peer'); <socket.socket fd=16, family=10, type=1, proto=6, laddr=('fc42:ca7::8', 54274, 0, 0)>
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [io_services_utils] Deactivating transport: state=1; <socket.socket fd=16, family=10, type=1, proto=6, laddr=('fc42:ca7::8', 54274, 0, 0)>
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [ERROR] [base_connection] connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [connection] AMQP stack terminated, failed to connect, or aborted: opened=True, error-arg=StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",); pending-error=None
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [connection] Stack terminated due to StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [io_services_utils] Closing transport socket and unlinking: state=2; <socket.socket fd=16, family=10, type=1, proto=6, laddr=('fc42:ca7::8', 54274, 0, 0)>
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [ERROR] [blocking_connection] Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",)
bytes-1  | [2024-05-15 14:26:22 +0000] [8] [INFO] [rabbitmq] Channel error Stream connection lost: ConnectionResetError(104, 'Connection reset by peer'), recreating channel

octopoes_api_worker-1 | [2024-05-16 07:55:24,237: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.recalculate_scan_profiles[3079f3a9-9834-4f54-ab47-99897b9ceee8]: Finished scan profile recalculation [org=aa] [dur=0.37s] octopoes_api_worker-1 | [2024-05-16 07:56:23,859: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.schedule_scan_profile_recalculations[c1dc822c-58c1-4faf-ab68-89f56061a10c]: Scheduled scan profile recalculation [org=aa] octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [io_services_utils] _AsyncBaseTransport._produce() failed, aborting connection: error=ConnectionResetError(104, 'Connection reset by peer'); sock=<socket.socket fd=18, family=10, type=1, proto=6, laddr=('fc42:ca7::b', 50874, 0, 0)>; Caller's stack: octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable octopoes_api_worker-1 | self._produce() octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce octopoes_api_worker-1 | num_bytes_sent = self._sigint_safe_send(self._sock, octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap octopoes_api_worker-1 | return func(*args, *kwargs) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send octopoes_api_worker-1 | return sock.send(data) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^ octopoes_api_worker-1 | ConnectionResetError: [Errno 104] Connection reset by peer octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable octopoes_api_worker-1 | self._produce() octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce octopoes_api_worker-1 | num_bytes_sent = self._sigint_safe_send(self._sock, octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap octopoes_api_worker-1 | return func(args, *kwargs) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send octopoes_api_worker-1 | return sock.send(data) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^ octopoes_api_worker-1 | ConnectionResetError: [Errno 104] Connection reset by peer octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [base_connection] connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",) octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [blocking_connection] Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",) octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [manager] Failed connecting to rabbitmq, retrying... octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/app/octopoes/octopoes/events/manager.py", line 155, in _try_connect octopoes_api_worker-1 | self._connect() octopoes_api_worker-1 | File "/app/octopoes/octopoes/events/manager.py", line 167, in _connect octopoes_api_worker-1 | self.channel.queue_declare(queue=f"{self.client}__scan_profile_increments", durable=True) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 2524, in queue_declare octopoes_api_worker-1 | self._flush_output(declare_ok_result.is_ready) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 1353, in _flush_output octopoes_api_worker-1 | self._connection._flush_output(lambda: self.is_closed, waiters) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 523, in _flush_output octopoes_api_worker-1 | raise self._closed_result.value.error octopoes_api_worker-1 | pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer') octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [INFO] [manager] Connected to RabbitMQ octopoes_api_worker-1 | [2024-05-16 07:56:24 +0000] [10] [INFO] [service] Recalculated scan profiles octopoes_api_worker-1 | [2024-05-16 07:56:24,253: INFO/ForkPoolWorker-2] octopoes.tasks.tasks.recalculate_scan_profiles[f09331e1-590a-495a-989b-acd6aed82395]: Finished scan profile recalculation [org=aa] [dur=0.39s] octopoes_api_worker-1 | [2024-05-16 07:57:23,857: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.schedule_scan_profile_recalculations[64e0f628-6764-4595-8bc2-6c8fa4d929c9]: Scheduled scan profile recalculation [org=aa] octopoes_api_worker-1 | [2024-05-16 07:57:24 +0000] [12] [INFO] [service] Recalculated scan profiles


- Octopoes Object not found exception
'''
boefje-1  | [2024-05-16 09:28:20 +0000] [17] [INFO] [app] Set status to TaskStatus.COMPLETED in the scheduler for task[id=d277a98a-6e67-4beb-aa76-8ee3cfc24b49]
boefje-1  | [2024-05-16 09:28:20 +0000] [17] [INFO] [job_handler] Handling boefje ssl-certificates[task_id=105ade14-50d6-4c21-be96-6644a8e3f3f7]
boefje-1  | [2024-05-16 09:28:20 +0000] [17] [INFO] [_client] HTTP Request: GET http://octopoes_api/aa/object?reference=Website%7Cinternet%7C185.73.34.10%7Ctcp%7C80%7Chttp%7Cinternet%REDACTED.nl&valid_time=2024-05-16%2009%3A28%3A20.592323%2B00%3A00 "HTTP/1.1 404
 Not Found"
boefje-1  | [2024-05-16 09:28:20 +0000] [17] [ERROR] [app] An error occurred handling scheduler item[id=105ade14-50d6-4c21-be96-6644a8e3f3f7]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/octopoes/connector/octopoes.py", line 47, in _verify_response
boefje-1  |     response.raise_for_status()
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 761, in raise_for_status
boefje-1  |     raise HTTPStatusError(message, request=request, response=self)
boefje-1  | httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://octopoes_api/aa/object?reference=Website%7Cinternet%7C185.73.34.10%7Ctcp%7C80%7Chttp%7Cinternet%REDACTED.nl&valid_time=2024-05-16%2009%3A28%3A20.592323%2B00%3A00'
boefje-1  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
boefje-1  | 
boefje-1  | During handling of the above exception, another exception occurred:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 123, in handle
boefje-1  |     ooi = get_octopoes_api_connector(boefje_meta.organization).get(
boefje-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/octopoes/connector/octopoes.py", line 86, in get
boefje-1  |     res = self.session.get(
boefje-1  |           ^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1054, in get
boefje-1  |     return self.request(
boefje-1  |            ^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 827, in request
boefje-1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send
boefje-1  |     response = self._send_handling_auth(
boefje-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth
boefje-1  |     response = self._send_handling_redirects(
boefje-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects
boefje-1  |     raise exc
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects
boefje-1  |     hook(response)
boefje-1  |   File "/app/boefjes/octopoes/connector/octopoes.py", line 51, in _verify_response
boefje-1  |     raise ObjectNotFoundException(data["detail"])
boefje-1  | octopoes.models.exception.ObjectNotFoundException: Website|internet|185.73.34.10|tcp|80|http|internet|REDACTED.nl
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/app.py", line 247, in _start_working
boefje-1  |     handler.handle(p_item.data)
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 127, in handle
boefje-1  |     raise ObjectNotFoundException(f"Object {reference} not found in Octopoes") from e
boefje-1  | octopoes.models.exception.ObjectNotFoundException: Object Website|internet|185.73.34.10|tcp|80|http|internet|REDACTED.nl not found in Octopoes
boefje-1  | [2024-05-16 09:28:20 +0000] [17] [INFO] [_client] HTTP Request: PATCH http://schedul
'''

- Certificate search error 1:

boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [_client] HTTP Request: GET http://octopoes_api/aa/object?reference=Hostname%7Cin ternet%7C.com&valid_time=2024-05-15%2014%3A29%3A15.152641%2B00%3A00 "HTTP/1.1 200 OK" boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [job_handler] Starting boefje pdio-subfinder[4cc8252e-90bc-43ba-8e47-11ff253d8de3 ] boefje-1 | [2024-05-15 14:29:15 +0000] [18] [ERROR] [job_handler] Error running boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9] boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs boefje-1 | response.raise_for_status() boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status boefje-1 | raise HTTPError(http_error_msg, response=self) boefje-1 | requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://crt.sh/?Identity=.com&match=%3D&output=json&deduplicate=Y boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9] boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"


- Certificate search error 2:

boefje-1 | [2024-05-16 07:49:21 +0000] [1] [INFO] [_client] HTTP Request: POST http://scheduler:8000/queues/boefje-aa/pop "HTTP/1.1 200 OK" boefje-1 | [2024-05-16 07:49:21 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[bbafa94b-7905-433b-b098-f7 b3c2d51a00] boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request boefje-1 | response = conn.getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse boefje-1 | httplib_response = super().getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 1395, in getresponse boefje-1 | response.begin() boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 325, in begin boefje-1 | version, status, reason = self._read_status() boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 286, in _read_status boefje-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/socket.py", line 706, in readinto boefje-1 | return self._sock.recv_into(b) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1314, in recv_into boefje-1 | return self.read(nbytes, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1166, in read boefje-1 | return self._sslobj.read(len, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | TimeoutError: The read operation timed out boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send boefje-1 | resp = conn.urlopen( boefje-1 | ^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen boefje-1 | retries = retries.increment( boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment boefje-1 | raise reraise(type(error), error, _stacktrace) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise boefje-1 | raise value boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen boefje-1 | response = self._make_request( boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request boefje-1 | self._raise_timeout(err=e, url=url, timeout_value=read_timeout) boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout boefje-1 | raise ReadTimeoutError( boefje-1 | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30)

boefje-1 | boefje-1 | During handling of the above exception, another exception occurred: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 50, in request_certs boefje-1 | response = requests.get(CRT_SH_API, params=query, timeout=30) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get boefje-1 | return request("get", url, params=params, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request boefje-1 | return session.request(method=method, url=url, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request boefje-1 | resp = self.send(prep, send_kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send boefje-1 | r = adapter.send(request, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send boefje-1 | raise ReadTimeout(e, request=request) boefje-1 | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30) boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[bbafa94b-7905-433b-b098-f7b3c2d51a00] boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"


- Certificate search error 3:

boefje-1 | [2024-05-16 07:50:28 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[ccd9f68f-1c61-4fe3-99a6-b54fb4b2ddc4] boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs boefje-1 | response.raise_for_status() boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status boefje-1 | raise HTTPError(http_error_msg, response=self) boefje-1 | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://crt.sh/?Identity=.net&match=%3D&output=json&deduplicate=Y boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed


- Normalizer input warnings:

normalizer-1 | [2024-05-15 14:22:22 +0000] [15] [WARNING] [job_handler] Normalizer "kat_dns_normalize" returned input [Hostname|internet|mispo.es]


- Postgres duplicate keys

postgres-1 | 2024-05-15 14:18:53.497 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres-1 | 2024-05-15 14:18:53.503 UTC [71] LOG: database system was shut down at 2024-05-15 14:18:53 UTC postgres-1 | 2024-05-15 14:18:53.508 UTC [1] LOG: database system is ready to accept connections postgres-1 | 2024-05-15 14:19:42.397 UTC [127] ERROR: duplicate key value violates unique constraint "repository_id" postgres-1 | 2024-05-15 14:19:42.397 UTC [127] DETAIL: Key (id)=(LOCAL) already exists. postgres-1 | 2024-05-15 14:19:42.397 UTC [127] STATEMENT: INSERT INTO repository (id, name, base_url) VALUES ('LOCAL', 'Local Plugin Repository', 'http://dev/null') RETURNING repository.pk postgres-1 | 2024-05-15 14:19:47.727 UTC [142] ERROR: duplicate key value violates unique constraint "repository_id" postgres-1 | 2024-05-15 14:19:47.727 UTC [142] DETAIL: Key (id)=(LOCAL) already exists. postgres-1 | 2024-05-15 14:19:47.727 UTC [142] STATEMENT: INSERT INTO repository (id, name, base_url) VALUES ('LOCAL', 'Local Plugin Repository', 'http://dev/null') RETURNING repository.pk postgres-1 | 2024-05-15 14:23:53.587 UTC [69] LOG: checkpoint starting: time postgres-1 | 2024-05-15 14:25:00.669 UTC [69] LOG: checkpoint complete: wrote 672 buffers (4.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=67.036 s, sync=0.028 s, total=67.083 s; sync files=469, longest=0.011 s, average=0.001 s; distance=3032 kB, estimate=3032 kB


- Scheduler serialisation error

scheduler-1 | 2024-05-15 14:22:03 [info ] Created boefje task: d1b1025d-2a47-4663-b716-336b20398bda for ooi: KATFindingType|KAT-NO-SPF boefje_id=kat-finding-types caller=push_tasks_for_new_boefjes ooi_primary_key=KATFindingType|KAT-NO-SPF organisation_id=aa scheduler_id=boefje-aa task_id=UUID('d1b1025d-2a47-4663-b716-336b20398bda') scheduler-1 | /usr/local/lib/python3.11/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: scheduler-1 | Expected enum but got str - serialized value may not be as expected scheduler-1 | return self.__pydantic_serializer__.to_python( scheduler-1 | /usr/local/lib/python3.11/site-packages/pydantic/type_adapter.py:339: UserWarning: Pydantic serializer warnings: scheduler-1 | Expected enum but got str - serialized value may not be as expected scheduler-1 | return self.serializer.to_python( scheduler-1 | 2024-05-15 14:22:14 [info ] Created normalizer task: 5259fd52-acec-4e8c-863a-a8f5cc435a6c for raw data: da4b8194-444c-4dad-9f8d-d64ad0ab172f caller=push_tasks_for_received_raw_data normalizer_id=kat_dnssec_normalize organisation_id=aa raw_data_id=UUID('da4b8194-444c-4dad-9f8d-d64ad0ab172f') scheduler_id=normalizer-aa task_id=UUID('5259fd52-acec-4e8c-863a-a8f5cc435a6c')

underdarknl commented 3 months ago

* Some boefjes in the Katalogus do not have a clearance level (CWE Finding Types, CVE Finding Types, External Database, KAT finding types, RetireJS Finding Types, Snyk finding types)

Correct, these are 'hydrating' boefjes that need level 0, and are always allowed to run. This could be more visible / explained though.

underdarknl commented 3 months ago

* Certificate search error 1:

boefje-1  | [2024-05-15 14:29:15 +0000] [17] [INFO] [_client] HTTP Request: GET http://octopoes_api/aa/object?reference=Hostname%7Cin
ternet%7C<REDACTED>.com&valid_time=2024-05-15%2014%3A29%3A15.152641%2B00%3A00 "HTTP/1.1 200 OK"
boefje-1  | [2024-05-15 14:29:15 +0000] [17] [INFO] [job_handler] Starting boefje pdio-subfinder[4cc8252e-90bc-43ba-8e47-11ff253d8de3
]
boefje-1  | [2024-05-15 14:29:15 +0000] [18] [ERROR] [job_handler] Error running boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 59, in run
boefje-1  |     return boefje_resource.module.run(boefje_meta)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run
boefje-1  |     results = request_certs(fqdn)
boefje-1  |               ^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs
boefje-1  |     response.raise_for_status()
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
boefje-1  |     raise HTTPError(http_error_msg, response=self)
boefje-1  | requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://crt.sh/?Identity=<redacted>.com&match=%3D&output=json&deduplicate=Y
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 145, in handle
boefje-1  |     boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 61, in run
boefje-1  |     raise JobRuntimeError("Boefje failed") from e
boefje-1  | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
boefje-1  | [2024-05-15 14:29:15 +0000] [18] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9]
boefje-1  | [2024-05-15 14:29:15 +0000] [18] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"

* Certificate search error 2:

boefje-1  | [2024-05-16 07:49:21 +0000] [1] [INFO] [_client] HTTP Request: POST http://scheduler:8000/queues/boefje-aa/pop "HTTP/1.1 
200 OK"
boefje-1  | [2024-05-16 07:49:21 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[bbafa94b-7905-433b-b098-f7
b3c2d51a00]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
boefje-1  |     response = conn.getresponse()
boefje-1  |                ^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
boefje-1  |     httplib_response = super().getresponse()
boefje-1  |                        ^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/http/client.py", line 1395, in getresponse
boefje-1  |     response.begin()
boefje-1  |   File "/usr/local/lib/python3.11/http/client.py", line 325, in begin
boefje-1  |     version, status, reason = self._read_status()
boefje-1  |                               ^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/http/client.py", line 286, in _read_status
boefje-1  |     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
boefje-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
boefje-1  |     return self._sock.recv_into(b)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/ssl.py", line 1314, in recv_into
boefje-1  |     return self.read(nbytes, buffer)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/ssl.py", line 1166, in read
boefje-1  |     return self._sslobj.read(len, buffer)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  | TimeoutError: The read operation timed out
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
boefje-1  |     resp = conn.urlopen(
boefje-1  |            ^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
boefje-1  |     retries = retries.increment(
boefje-1  |               ^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
boefje-1  |     raise reraise(type(error), error, _stacktrace)
boefje-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
boefje-1  |     raise value
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
boefje-1  |     response = self._make_request(
boefje-1  |                ^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request
boefje-1  |     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
boefje-1  |     raise ReadTimeoutError(
boefje-1  | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30)

boefje-1  | 
boefje-1  | During handling of the above exception, another exception occurred:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 59, in run
boefje-1  |     return boefje_resource.module.run(boefje_meta)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run
boefje-1  |     results = request_certs(fqdn)
boefje-1  |               ^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 50, in request_certs
boefje-1  |     response = requests.get(CRT_SH_API, params=query, timeout=30)
boefje-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get
boefje-1  |     return request("get", url, params=params, **kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request
boefje-1  |     return session.request(method=method, url=url, **kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
boefje-1  |     resp = self.send(prep, **send_kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
boefje-1  |     r = adapter.send(request, **kwargs)
boefje-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
boefje-1  |     raise ReadTimeout(e, request=request)
boefje-1  | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30)
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 145, in handle
boefje-1  |     boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 61, in run
boefje-1  |     raise JobRuntimeError("Boefje failed") from e
boefje-1  | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
boefje-1  | [2024-05-16 07:49:21 +0000] [16] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[bbafa94b-7905-433b-b098-f7b3c2d51a00]
boefje-1  | [2024-05-16 07:49:21 +0000] [16] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"

* Certificate search error 3:

boefje-1  | [2024-05-16 07:50:28 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[ccd9f68f-1c61-4fe3-99a6-b54fb4b2ddc4]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 59, in run
boefje-1  |     return boefje_resource.module.run(boefje_meta)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run
boefje-1  |     results = request_certs(fqdn)
boefje-1  |               ^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs
boefje-1  |     response.raise_for_status()
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
boefje-1  |     raise HTTPError(http_error_msg, response=self)
boefje-1  | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://crt.sh/?Identity=<REDACTED>.net&match=%3D&output=json&deduplicate=Y
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 145, in handle
boefje-1  |     boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 61, in run
boefje-1  |     raise JobRuntimeError("Boefje failed") from e
boefje-1  | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed

Looks like a temporary problem on the side of crt.sh, these's url's seem to work fine for my setup atm.

minvws / nl-kat-coordination

Overview ticket - current bugs/warnings/errors on main #2965