Open stephanie0x00 opened 3 months ago
* Some boefjes in the Katalogus do not have a clearance level (CWE Finding Types, CVE Finding Types, External Database, KAT finding types, RetireJS Finding Types, Snyk finding types)
Correct, these are 'hydrating' boefjes that need level 0, and are always allowed to run. This could be more visible / explained though.
* Certificate search error 1:
boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [_client] HTTP Request: GET http://octopoes_api/aa/object?reference=Hostname%7Cin ternet%7C<REDACTED>.com&valid_time=2024-05-15%2014%3A29%3A15.152641%2B00%3A00 "HTTP/1.1 200 OK" boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [job_handler] Starting boefje pdio-subfinder[4cc8252e-90bc-43ba-8e47-11ff253d8de3 ] boefje-1 | [2024-05-15 14:29:15 +0000] [18] [ERROR] [job_handler] Error running boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9] boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs boefje-1 | response.raise_for_status() boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status boefje-1 | raise HTTPError(http_error_msg, response=self) boefje-1 | requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://crt.sh/?Identity=<redacted>.com&match=%3D&output=json&deduplicate=Y boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9] boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
* Certificate search error 2:
boefje-1 | [2024-05-16 07:49:21 +0000] [1] [INFO] [_client] HTTP Request: POST http://scheduler:8000/queues/boefje-aa/pop "HTTP/1.1 200 OK" boefje-1 | [2024-05-16 07:49:21 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[bbafa94b-7905-433b-b098-f7 b3c2d51a00] boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request boefje-1 | response = conn.getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse boefje-1 | httplib_response = super().getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 1395, in getresponse boefje-1 | response.begin() boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 325, in begin boefje-1 | version, status, reason = self._read_status() boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 286, in _read_status boefje-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/socket.py", line 706, in readinto boefje-1 | return self._sock.recv_into(b) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1314, in recv_into boefje-1 | return self.read(nbytes, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1166, in read boefje-1 | return self._sslobj.read(len, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | TimeoutError: The read operation timed out boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send boefje-1 | resp = conn.urlopen( boefje-1 | ^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen boefje-1 | retries = retries.increment( boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment boefje-1 | raise reraise(type(error), error, _stacktrace) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise boefje-1 | raise value boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen boefje-1 | response = self._make_request( boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request boefje-1 | self._raise_timeout(err=e, url=url, timeout_value=read_timeout) boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout boefje-1 | raise ReadTimeoutError( boefje-1 | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30) boefje-1 | boefje-1 | During handling of the above exception, another exception occurred: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 50, in request_certs boefje-1 | response = requests.get(CRT_SH_API, params=query, timeout=30) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get boefje-1 | return request("get", url, params=params, **kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request boefje-1 | return session.request(method=method, url=url, **kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request boefje-1 | resp = self.send(prep, **send_kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send boefje-1 | r = adapter.send(request, **kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send boefje-1 | raise ReadTimeout(e, request=request) boefje-1 | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30) boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[bbafa94b-7905-433b-b098-f7b3c2d51a00] boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
* Certificate search error 3:
boefje-1 | [2024-05-16 07:50:28 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[ccd9f68f-1c61-4fe3-99a6-b54fb4b2ddc4] boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs boefje-1 | response.raise_for_status() boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status boefje-1 | raise HTTPError(http_error_msg, response=self) boefje-1 | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://crt.sh/?Identity=<REDACTED>.net&match=%3D&output=json&deduplicate=Y boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
Looks like a temporary problem on the side of crt.sh, these's url's seem to work fine for my setup atm.
The purpose of this ticket is to give an overview on the current bugs/errors/warning that are found in KAT on the current main. Based on this ticket we can determine if/which items need to be picked up (this sprint).
Current bugs in main (commit a613e82d966b83a13c7669124e2440c164f9851e (HEAD -> main, origin/main, origin/HEAD)):
The raw files for KAT FindingTypes seem to (always) contain the output of the KAT Findings database. Unsure if this is a bug or intented behaviour.
Sometimes RAW files for failed tasks cannot be downloaded. #2864
Setting clearance level on object is not shown in the UI. As a user you have to press the button twice before the chosen clearance level is shown back correctly to the user. #2703
ValueError for staticfiles with DEBUG=False prevents UI from loading: #2973
Some boefjes in the Katalogus do not have a clearance level (CWE Finding Types, CVE Finding Types, External Database, KAT finding types, RetireJS Finding Types, Snyk finding types)
Error reading bcrypt version: #2675
Connection error (these connection errors also show up on the octopoes_api_worker, the logs for the octopoes_api_worker are in the second half of this snippet) (#2213)
octopoes_api_worker-1 | [2024-05-16 07:55:24,237: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.recalculate_scan_profiles[3079f3a9-9834-4f54-ab47-99897b9ceee8]: Finished scan profile recalculation [org=aa] [dur=0.37s] octopoes_api_worker-1 | [2024-05-16 07:56:23,859: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.schedule_scan_profile_recalculations[c1dc822c-58c1-4faf-ab68-89f56061a10c]: Scheduled scan profile recalculation [org=aa] octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [io_services_utils] _AsyncBaseTransport._produce() failed, aborting connection: error=ConnectionResetError(104, 'Connection reset by peer'); sock=<socket.socket fd=18, family=10, type=1, proto=6, laddr=('fc42:ca7::b', 50874, 0, 0)>; Caller's stack: octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable octopoes_api_worker-1 | self._produce() octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce octopoes_api_worker-1 | num_bytes_sent = self._sigint_safe_send(self._sock, octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap octopoes_api_worker-1 | return func(*args, *kwargs) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send octopoes_api_worker-1 | return sock.send(data) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^ octopoes_api_worker-1 | ConnectionResetError: [Errno 104] Connection reset by peer octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 1103, in _on_socket_writable octopoes_api_worker-1 | self._produce() octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 819, in _produce octopoes_api_worker-1 | num_bytes_sent = self._sigint_safe_send(self._sock, octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 79, in retry_sigint_wrap octopoes_api_worker-1 | return func(args, *kwargs) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^^^^^^^ octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/utils/io_services_utils.py", line 861, in _sigint_safe_send octopoes_api_worker-1 | return sock.send(data) octopoes_api_worker-1 | ^^^^^^^^^^^^^^^ octopoes_api_worker-1 | ConnectionResetError: [Errno 104] Connection reset by peer octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [base_connection] connection_lost: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",) octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [blocking_connection] Unexpected connection close detected: StreamLostError: ("Stream connection lost: ConnectionResetError(104, 'Connection reset by peer')",) octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [ERROR] [manager] Failed connecting to rabbitmq, retrying... octopoes_api_worker-1 | Traceback (most recent call last): octopoes_api_worker-1 | File "/app/octopoes/octopoes/events/manager.py", line 155, in _try_connect octopoes_api_worker-1 | self._connect() octopoes_api_worker-1 | File "/app/octopoes/octopoes/events/manager.py", line 167, in _connect octopoes_api_worker-1 | self.channel.queue_declare(queue=f"{self.client}__scan_profile_increments", durable=True) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 2524, in queue_declare octopoes_api_worker-1 | self._flush_output(declare_ok_result.is_ready) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 1353, in _flush_output octopoes_api_worker-1 | self._connection._flush_output(lambda: self.is_closed, waiters) octopoes_api_worker-1 | File "/usr/local/lib/python3.11/site-packages/pika/adapters/blocking_connection.py", line 523, in _flush_output octopoes_api_worker-1 | raise self._closed_result.value.error octopoes_api_worker-1 | pika.exceptions.StreamLostError: Stream connection lost: ConnectionResetError(104, 'Connection reset by peer') octopoes_api_worker-1 | [2024-05-16 07:56:23 +0000] [10] [INFO] [manager] Connected to RabbitMQ octopoes_api_worker-1 | [2024-05-16 07:56:24 +0000] [10] [INFO] [service] Recalculated scan profiles octopoes_api_worker-1 | [2024-05-16 07:56:24,253: INFO/ForkPoolWorker-2] octopoes.tasks.tasks.recalculate_scan_profiles[f09331e1-590a-495a-989b-acd6aed82395]: Finished scan profile recalculation [org=aa] [dur=0.39s] octopoes_api_worker-1 | [2024-05-16 07:57:23,857: INFO/ForkPoolWorker-3] octopoes.tasks.tasks.schedule_scan_profile_recalculations[64e0f628-6764-4595-8bc2-6c8fa4d929c9]: Scheduled scan profile recalculation [org=aa] octopoes_api_worker-1 | [2024-05-16 07:57:24 +0000] [12] [INFO] [service] Recalculated scan profiles
boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [_client] HTTP Request: GET http://octopoes_api/aa/object?reference=Hostname%7Cin ternet%7C.com&valid_time=2024-05-15%2014%3A29%3A15.152641%2B00%3A00 "HTTP/1.1 200 OK"
boefje-1 | [2024-05-15 14:29:15 +0000] [17] [INFO] [job_handler] Starting boefje pdio-subfinder[4cc8252e-90bc-43ba-8e47-11ff253d8de3
]
boefje-1 | [2024-05-15 14:29:15 +0000] [18] [ERROR] [job_handler] Error running boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9]
boefje-1 | Traceback (most recent call last):
boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run
boefje-1 | return boefje_resource.module.run(boefje_meta)
boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run
boefje-1 | results = request_certs(fqdn)
boefje-1 | ^^^^^^^^^^^^^^^^^^^
boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs
boefje-1 | response.raise_for_status()
boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
boefje-1 | raise HTTPError(http_error_msg, response=self)
boefje-1 | requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://crt.sh/?Identity=.com&match=%3D&output=json&deduplicate=Y
boefje-1 |
boefje-1 | The above exception was the direct cause of the following exception:
boefje-1 |
boefje-1 | Traceback (most recent call last):
boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle
boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run
boefje-1 | raise JobRuntimeError("Boefje failed") from e
boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[a020b0eb-8b20-4006-87d3-24d9f7c17de9]
boefje-1 | [2024-05-15 14:29:15 +0000] [18] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
boefje-1 | [2024-05-16 07:49:21 +0000] [1] [INFO] [_client] HTTP Request: POST http://scheduler:8000/queues/boefje-aa/pop "HTTP/1.1 200 OK" boefje-1 | [2024-05-16 07:49:21 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[bbafa94b-7905-433b-b098-f7 b3c2d51a00] boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request boefje-1 | response = conn.getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse boefje-1 | httplib_response = super().getresponse() boefje-1 | ^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 1395, in getresponse boefje-1 | response.begin() boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 325, in begin boefje-1 | version, status, reason = self._read_status() boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/http/client.py", line 286, in _read_status boefje-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/socket.py", line 706, in readinto boefje-1 | return self._sock.recv_into(b) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1314, in recv_into boefje-1 | return self.read(nbytes, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/ssl.py", line 1166, in read boefje-1 | return self._sslobj.read(len, buffer) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | TimeoutError: The read operation timed out boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send boefje-1 | resp = conn.urlopen( boefje-1 | ^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen boefje-1 | retries = retries.increment( boefje-1 | ^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment boefje-1 | raise reraise(type(error), error, _stacktrace) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise boefje-1 | raise value boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen boefje-1 | response = self._make_request( boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request boefje-1 | self._raise_timeout(err=e, url=url, timeout_value=read_timeout) boefje-1 | File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout boefje-1 | raise ReadTimeoutError( boefje-1 | urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30)
boefje-1 | boefje-1 | During handling of the above exception, another exception occurred: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 50, in request_certs boefje-1 | response = requests.get(CRT_SH_API, params=query, timeout=30) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 73, in get boefje-1 | return request("get", url, params=params, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/api.py", line 59, in request boefje-1 | return session.request(method=method, url=url, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request boefje-1 | resp = self.send(prep, send_kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send boefje-1 | r = adapter.send(request, kwargs) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send boefje-1 | raise ReadTimeout(e, request=request) boefje-1 | requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='crt.sh', port=443): Read timed out. (read timeout=30) boefje-1 | boefje-1 | The above exception was the direct cause of the following exception: boefje-1 | boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run boefje-1 | raise JobRuntimeError("Boefje failed") from e boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [job_handler] Saving to Bytes for boefje certificate-search[bbafa94b-7905-433b-b098-f7b3c2d51a00] boefje-1 | [2024-05-16 07:49:21 +0000] [16] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"
boefje-1 | [2024-05-16 07:50:28 +0000] [16] [ERROR] [job_handler] Error running boefje certificate-search[ccd9f68f-1c61-4fe3-99a6-b54fb4b2ddc4] boefje-1 | Traceback (most recent call last): boefje-1 | File "/app/boefjes/boefjes/local.py", line 59, in run boefje-1 | return boefje_resource.module.run(boefje_meta) boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 61, in run boefje-1 | results = request_certs(fqdn) boefje-1 | ^^^^^^^^^^^^^^^^^^^ boefje-1 | File "/app/boefjes/boefjes/plugins/kat_crt_sh/main.py", line 52, in request_certs boefje-1 | response.raise_for_status() boefje-1 | File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status boefje-1 | raise HTTPError(http_error_msg, response=self) boefje-1 | requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://crt.sh/?Identity=.net&match=%3D&output=json&deduplicate=Y
boefje-1 |
boefje-1 | The above exception was the direct cause of the following exception:
boefje-1 |
boefje-1 | Traceback (most recent call last):
boefje-1 | File "/app/boefjes/boefjes/job_handler.py", line 145, in handle
boefje-1 | boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1 | File "/app/boefjes/boefjes/local.py", line 61, in run
boefje-1 | raise JobRuntimeError("Boefje failed") from e
boefje-1 | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
normalizer-1 | [2024-05-15 14:22:22 +0000] [15] [WARNING] [job_handler] Normalizer "kat_dns_normalize" returned input [Hostname|internet|mispo.es]
postgres-1 | 2024-05-15 14:18:53.497 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" postgres-1 | 2024-05-15 14:18:53.503 UTC [71] LOG: database system was shut down at 2024-05-15 14:18:53 UTC postgres-1 | 2024-05-15 14:18:53.508 UTC [1] LOG: database system is ready to accept connections postgres-1 | 2024-05-15 14:19:42.397 UTC [127] ERROR: duplicate key value violates unique constraint "repository_id" postgres-1 | 2024-05-15 14:19:42.397 UTC [127] DETAIL: Key (id)=(LOCAL) already exists. postgres-1 | 2024-05-15 14:19:42.397 UTC [127] STATEMENT: INSERT INTO repository (id, name, base_url) VALUES ('LOCAL', 'Local Plugin Repository', 'http://dev/null') RETURNING repository.pk postgres-1 | 2024-05-15 14:19:47.727 UTC [142] ERROR: duplicate key value violates unique constraint "repository_id" postgres-1 | 2024-05-15 14:19:47.727 UTC [142] DETAIL: Key (id)=(LOCAL) already exists. postgres-1 | 2024-05-15 14:19:47.727 UTC [142] STATEMENT: INSERT INTO repository (id, name, base_url) VALUES ('LOCAL', 'Local Plugin Repository', 'http://dev/null') RETURNING repository.pk postgres-1 | 2024-05-15 14:23:53.587 UTC [69] LOG: checkpoint starting: time postgres-1 | 2024-05-15 14:25:00.669 UTC [69] LOG: checkpoint complete: wrote 672 buffers (4.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=67.036 s, sync=0.028 s, total=67.083 s; sync files=469, longest=0.011 s, average=0.001 s; distance=3032 kB, estimate=3032 kB
scheduler-1 | 2024-05-15 14:22:03 [info ] Created boefje task: d1b1025d-2a47-4663-b716-336b20398bda for ooi: KATFindingType|KAT-NO-SPF boefje_id=kat-finding-types caller=push_tasks_for_new_boefjes ooi_primary_key=KATFindingType|KAT-NO-SPF organisation_id=aa scheduler_id=boefje-aa task_id=UUID('d1b1025d-2a47-4663-b716-336b20398bda') scheduler-1 | /usr/local/lib/python3.11/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings: scheduler-1 | Expected
enum
but gotstr
- serialized value may not be as expected scheduler-1 | return self.__pydantic_serializer__.to_python( scheduler-1 | /usr/local/lib/python3.11/site-packages/pydantic/type_adapter.py:339: UserWarning: Pydantic serializer warnings: scheduler-1 | Expectedenum
but gotstr
- serialized value may not be as expected scheduler-1 | return self.serializer.to_python( scheduler-1 | 2024-05-15 14:22:14 [info ] Created normalizer task: 5259fd52-acec-4e8c-863a-a8f5cc435a6c for raw data: da4b8194-444c-4dad-9f8d-d64ad0ab172f caller=push_tasks_for_received_raw_data normalizer_id=kat_dnssec_normalize organisation_id=aa raw_data_id=UUID('da4b8194-444c-4dad-9f8d-d64ad0ab172f') scheduler_id=normalizer-aa task_id=UUID('5259fd52-acec-4e8c-863a-a8f5cc435a6c')