search

minvws / nl-kat-coordination

Repo nl-kat-coordination for minvws
European Union Public License 1.2
123 stars 55 forks source link

Boefjes stuck as dispatched, nl-kat-coordination_bytes error value too long for type character varying #121

Closed tobiasBDO closed 1 year ago

tobiasBDO commented 1 year ago

Please add bug, the name of any relevant modules (e.g. rocky), and any other relevant labels to your issue.

Describe the bug When trying to do an onboarding and adding the URL, the status of the Boefjes is stuck as dispatched.

To Reproduce Steps to reproduce the behavior:

  1. clone the nl-kat-coordination repo
  2. run the make command and create your admin user
  3. run the onboarding sequence and add a URL
  4. run the DNS scan

docker-compose via ghcr.io Steps to reproduce the behavior:

  1. clone the nl-kat-coordination repo
  2. run the docker-compose.release-example.yml
  3. run the seed, ooi database seed, superuser and setup_dev acount commands
  4. run the onboarding sequence and add a URL
  5. run the DNS scan

Expected behavior I expect the boefjes to run the DNS scans and for the status to be Completed after a short while, as it worked via GitPod for me.

Screenshots afbeelding

OpenKAT version OpenKAT built via the MakeFile, container version tag latest (1.5.0rc3) and ran on Docker on Ubuntu 20.04.5 LTS.

Desktop (please complete the following information if relevant):

Additional context


[2023-01-31 10:33:57 +0000] [10] [INFO] [sql_meta_repository] Closing session
[2023-01-31 10:33:59 +0000] [10] [INFO] [h11_impl] 172.18.0.12:42186 - "POST /token HTTP/1.1" 200
[2023-01-31 10:33:59 +0000] [10] [INFO] [sql_meta_repository] Inserting meta: {"id": "e733ff73f8b743d2a692567df9e75649", "started_at": "2023-01-31T10:33:58.673492+00:00", "ended_at": "2023-01-31T10:33:58.721416+00:00", "boefje": {"id": "dns-records", "version": null}, "input_ooi": "Hostname|internet|bdo.nl", "arguments": {"input": {"object_type": "Hostname", "scan_profile": "scan_profile_type='inherited' reference=Reference('Hostname|internet|bdo.nl') level=<ScanLevel.L2: 2>", "primary_key": "Hostname|internet|bdo.nl", "network": {"name": "internet"}, "name": "bdo.nl", "fqdn": {"network": {"name": "internet"}, "name": "bdo.nl."}, "dns_zone": "None"}}, "organization": "bdo-advisory-bv"}
[2023-01-31 10:33:59 +0000] [10] [INFO] [sql_meta_repository] Committing session
[2023-01-31 10:33:59 +0000] [10] [ERROR] [sql_meta_repository] An error occured: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(8)

[SQL: INSERT INTO boefje_meta (id, boefje_id, boefje_version, organization, input_ooi, arguments, started_at, ended_at) VALUES (%(id)s, %(boefje_id)s, %(boefje_version)s, %(organization)s, %(input_ooi)s, %(arguments)s, %(started_at)s, %(ended_at)s)]
[parameters: {'id': UUID('e733ff73-f8b7-43d2-a692-567df9e75649'), 'boefje_id': 'dns-records', 'boefje_version': None, 'organization': 'bdo-advisory-bv', 'input_ooi': 'Hostname|internet|bdo.nl', 'arguments': '{"input": {"object_type": "Hostname", "scan_profile": "scan_profile_type=\'inherited\' reference=Reference(\'Hostname|internet|bdo.nl\') level=<ScanL ... (41 characters truncated) ... ternet|bdo.nl", "network": {"name": "internet"}, "name": "bdo.nl", "fqdn": {"network": {"name": "internet"}, "name": "bdo.nl."}, "dns_zone": "None"}}', 'started_at': datetime.datetime(2023, 1, 31, 10, 33, 58, 673492, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2023, 1, 31, 10, 33, 58, 721416, tzinfo=datetime.timezone.utc)}]
(Background on this error at: https://sqlalche.me/e/14/9h9h). Rolling back session
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(8)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/bytes/./bytes/database/sql_meta_repository.py", line 183, in create_meta_data_repository
    yield repository
sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(8)

[SQL: INSERT INTO boefje_meta (id, boefje_id, boefje_version, organization, input_ooi, arguments, started_at, ended_at) VALUES (%(id)s, %(boefje_id)s, %(boefje_version)s, %(organization)s, %(input_ooi)s, %(arguments)s, %(started_at)s, %(ended_at)s)]
[parameters: {'id': UUID('e733ff73-f8b7-43d2-a692-567df9e75649'), 'boefje_id': 'dns-records', 'boefje_version': None, 'organization': 'bdo-advisory-bv', 'input_ooi': 'Hostname|internet|bdo.nl', 'arguments': '{"input": {"object_type": "Hostname", "scan_profile": "scan_profile_type=\'inherited\' reference=Reference(\'Hostname|internet|bdo.nl\') level=<ScanL ... (41 characters truncated) ... ternet|bdo.nl", "network": {"name": "internet"}, "name": "bdo.nl", "fqdn": {"network": {"name": "internet"}, "name": "bdo.nl."}, "dns_zone": "None"}}', 'started_at': datetime.datetime(2023, 1, 31, 10, 33, 58, 673492, tzinfo=datetime.timezone.utc), 'ended_at': datetime.datetime(2023, 1, 31, 10, 33, 58, 721416, tzinfo=datetime.timezone.utc)}]
(Background on this error at: https://sqlalche.me/e/14/9h9h)
[2023-01-31 10:33:59 +0000] [10] [INFO] [sql_meta_repository] Closing session
[2023-01-31 10:33:59 +0000] [10] [INFO] [h11_impl] 172.18.0.12:42186 - "POST /bytes/boefje_meta HTTP/1.1" 500
[2023-01-31 10:33:59 +0000] [10] [ERROR] [h11_impl] Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 366, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    raise exc from None
  File "/usr/local/lib/python3.8/site-packages/uvicorn/middleware/message_logger.py", line 78, in __call__
    await self.app(scope, inner_receive, inner_send)
  File "/usr/local/lib/python3.8/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 235, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.8/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/usr/local/lib/python3.8/site-packages/starlette/concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "/usr/local/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/app/bytes/./bytes/api/v2/router.py", line 29, in create_boefje_meta
    meta_repository.save_boefje_meta(boefje_meta)
  File "/app/bytes/./bytes/database/sql_meta_repository.py", line 43, in __exit__
    self.session.commit()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1451, in commit
    self._transaction.commit(_to_root=self.future)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 829, in commit
    self._prepare_impl()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl
    self.session.flush()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3444, in flush
    self._flush(objects)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3584, in _flush
    transaction.rollback(_capture_exception=True)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__
    compat.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 3544, in _flush
    flush_context.execute()
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute
    rec.execute(self)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute
    util.preloaded.orm_persistence.save_obj(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
    _emit_insert_statements(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements
    c = connection._execute_20(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
    return meth(self, args_10style, kwargs_10style, execution_options)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
    return connection._execute_clauseelement(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 210, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.DataError: (psycopg2.errors.StringDataRightTruncation) value too long for type character varying(8)```
underdarknl commented 1 year ago

Dear @tobiasBDO interesting. we'll dive right into this.

underdarknl commented 1 year ago

It seems we have a mismatch between what the onboarding form tells you is valid length for your organization code (32 chars) and what bytes accepts (8). Good catch, we'll get this fixed asap.

Donnype commented 1 year ago

Assuming you run make kat so you are pulling the main branches, this should be fixed.

tobiasBDO commented 1 year ago

Thank you very much, it is working! @underdarknl @Donnype