Closed ccben87 closed 2 weeks ago
taylorwalton
commented
2 weeks ago @ccben87 can you share the .env file you are using? it looks like you dont have the HAProxy Provisioning variable set. Here is an example: https://github.com/socfortress/CoPilot/blob/main/.env.example
ccben87
commented
2 weeks ago Hi @taylorwalton,
I thought that was the issue too at first so I ended up defining it but that didn't help:
2024-06-14 11:53:38.294 | INFO | app.db.db_populate:load_connector_data:47 - Loading connector data for Wazuh Worker Provisioning from environment variables with URL: http://1.1.1.1
2024-06-14 11:53:38.294 | INFO | app.db.db_populate:load_connector_data:47 - Loading connector data for Event Shipper from environment variables with URL: graylog_host
2024-06-14 11:53:38.294 | INFO | app.db.db_populate:load_connector_data:47 - Loading connector data for HAProxy Provisioning from environment variables with URL: http://1.1.1.1
2024-06-14 11:53:38.339 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh-Indexer
2024-06-14 11:53:38.343 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh-Manager
2024-06-14 11:53:38.345 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Graylog
2024-06-14 11:53:38.347 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Shuffle
2024-06-14 11:53:38.348 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: DFIR-IRIS
2024-06-14 11:53:38.350 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Velociraptor
2024-06-14 11:53:38.351 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Sublime
2024-06-14 11:53:38.353 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: InfluxDB
2024-06-14 11:53:38.355 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: AskSocfortress
2024-06-14 11:53:38.357 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: SocfortressThreatIntel
2024-06-14 11:53:38.359 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Cortex
2024-06-14 11:53:38.360 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Grafana
2024-06-14 11:53:38.361 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh Worker Provisioning
2024-06-14 11:53:38.363 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Event Shipper
2024-06-14 11:53:38.365 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: HAProxy Provisioning
2024-06-14 11:53:38.365 | INFO | app.db.db_populate:add_connectors_if_not_exist:181 - before commit
Exception ignored in: <function Connection.__del__ at 0x7f7ca85f5bc0>
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/aiomysql/connection.py", line 1131, in __del__
File "/opt/venv/lib/python3.11/site-packages/aiomysql/connection.py", line 339, in close
File "/usr/lib/python3.11/asyncio/selector_events.py", line 864, in close
File "/usr/lib/python3.11/asyncio/base_events.py", line 762, in call_soon
File "/usr/lib/python3.11/asyncio/base_events.py", line 520, in _check_closed
RuntimeError: Event loop is closedYou can see above that it now has a URL rather than None yet the error still happens. Excuse the debug prints I've added to try work out what was wrong.
Here are the envs:
ENV WAZUH_INDEXER_URL=https://1.1.1.1:9200
ENV WAZUH_INDEXER_USERNAME=admin
ENV WAZUH_INDEXER_PASSWORD=admin
ENV WAZUH_MANAGER_URL=https://1.1.1.1
ENV WAZUH_MANAGER_USERNAME=dummy
ENV WAZUH_MANAGER_PASSWORD=dummy
ENV GRAYLOG_URL=http://1.1.1.1
ENV GRAYLOG_USERNAME=dummy
ENV GRAYLOG_PASSWORD=dummy
ENV SHUFFLE_URL=https://1.1.1.1
ENV SHUFFLER_API_KEY=dummy
ENV DFIR_IRIS_URL=https://1.1.1.1
ENV DFIR_IRIS_API_KEY=dummy
ENV VELOCIRAPTOR_URL=https://1.1.1.1
ENV VELOCIRAPTOR_API_KEY_PATH=dummy
ENV SUBLIME_URL=http://1.1.1.1
ENV SUBLIME_API_KEY=dummy
ENV INFLUXDB_URL=http://1.1.1.1
ENV INFLUXDB_API_KEY=dummy
ENV INFLUXDB_ORG_AND_BUCKET=dummy,dummy
ENV ASKSOCFORTRESS_URL=https://knowledge.socfortress.co
ENV ASKSOCFORTRESS_API_KEY=dummy
ENV SOCFORTRESSTHREATINTEL_URL=https://intel.socfortress.co/search
ENV SOCFORTRESSTHREATINTEL_API_KEY=dummy
ENV CORTEX_URL=http://1.1.1.1
ENV CORTEX_API_KEY=dummy
ENV GRAFANA_URL=http://1.1.1.1
ENV GRAFANA_USERNAME=dummy
ENV GRAFANA_PASSWORD=dummy
ENV WAZUH_WORKER_PROVISIONING_URL=http://1.1.1.1
ENV EVENT_SHIPPER_URL=graylog_host
ENV GELF_INPUT_PORT=gelf_port
ENV ALERT_CREATION_PROVISIONING_URL=http://1.1.1.1
ARG COPILOT_API_KEY
ENV COPILOT_API_KEY=$COPILOT_API_KEY
ENV HAPROXY_PROVISIONING_URL=http://1.1.1.1@ccben87 hmm that is odd indeed...do you see any startup errors in the mysql container? What OS are you copilot on?
ccben87
commented
2 weeks ago I'm running on Kubernetes (Talos Linux underneath) actually but that shouldn't be the issue here. I am using MariaDB instead of Mysql which could be the issue but MariaDB is meant to be compatible with Mysql. I can switch over to Mysql to eliminate that as a cause.
I do get these logs from my MariaDB container:
2024-06-14 14:32:20 3982 [Warning] Aborted connection 3982 to db: 'unconnected' user: 'unauthenticated' host: '
' (This connection closed normally without authentication) 2024-06-14 14:32:44 3994 [Warning] Aborted connection 3994 to db: 'copilot' user: 'copilot' host: ' ' (Got an error reading communication packets)
I do wonder if it could be MTU related.. I do specifically apply an MTU on my interface in Talos Linux (because it is required) and have never had to apply it in my K8s containers before though.
taylorwalton
commented
2 weeks ago Sounds like you have a unique setup. I personally avoid kubernetes and do not have much experience with it but I do see where it can be beneficial. Yes, I would recommend to run the app as I recommend and have tested with. I've had trouble with MTU issues in the past but when it comes to containers usually MTU will only give issues if the container is attempting to access something outside of the service, for example the internet.
ccben87
commented
2 weeks ago Just tried Mysql 5.7, same issue.
taylorwalton
commented
2 weeks ago Maybe something unique to your k8 environment?
ccben87
commented
2 weeks ago I've just double checked and it won't be MTU related as my K8s CNI (Cillium) matches the MTU of the host interface - which is why I've never had to specifically define it for containers - so that should be fine. I did just do some Googling in relation to that odd error in the logs for both Mysql and MariaDB and came across this: https://github.com/sqlalchemy/sqlalchemy/issues/10893
I do note that you're using a fairly old version of SQLAlchemy. Maybe upgrading might work. Do you know of any incompatibilites in later versions?
ccben87
commented
2 weeks ago No luck on attempting to change SQL Alchemy versions. I'm going to try to debug the SQL queries but otherwise I might need to rewrite/rework this code to make it work :(
I note it fails on await session.commit() for some reason, I wonder if it's because of the session.adds that aren't await'd.
taylorwalton
commented
2 weeks ago If you try outside of K8s environment do you run into same issue?
ccben87
commented
2 weeks ago I'll try in Docker on my desktop but I really do need to run it in K8s though.
Does this maybe help? I note that I get a rollback which is odd and seems related to HAProxy Provisioning. I could try removing that connector.
INFO [sqlalchemy.engine.Engine] SELECT DATABASE()
INFO [sqlalchemy.engine.Engine] [raw sql] ()
INFO [sqlalchemy.engine.Engine] SELECT @@sql_mode
INFO [sqlalchemy.engine.Engine] [raw sql] ()
INFO [sqlalchemy.engine.Engine] SELECT @@lower_case_table_names
INFO [sqlalchemy.engine.Engine] [raw sql] ()
INFO [sqlalchemy.engine.Engine] BEGIN (implicit)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [generated in 0.00034s] ('Wazuh-Indexer',)
2024-06-14 15:59:48.416 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh-Indexer
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [generated in 0.00029s] ('Wazuh-Indexer', '4.4.1', 'https://127.1.1.1:9200', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), 'admin', 'admin', None, 'Connection to Wazuh-Indexer.', 'Not specified.', 1, 0, 0, 0, 1, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.005759s ago] ('Wazuh-Manager',)
2024-06-14 15:59:48.421 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh-Manager
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.004033s ago] ('Wazuh-Manager', '4.4.1', 'https://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), 'dummy', 'dummy', None, 'Connection to Wazuh-Manager. Default is wazuh-wui:wazuh-wui', 'Not specified.', 1, 0, 0, 0, 1, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.009166s ago] ('Graylog',)
2024-06-14 15:59:48.424 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Graylog
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.007035s ago] ('Graylog', '5.0.7', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), 'dummy', 'dummy', None, 'Connection to Graylog.', 'Not specified.', 1, 0, 0, 0, 1, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.01186s ago] ('Shuffle',)
2024-06-14 15:59:48.426 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Shuffle
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.009231s ago] ('Shuffle', '1.1.0', 'https://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, None, 'Connection to Shuffle.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.01397s ago] ('DFIR-IRIS',)
2024-06-14 15:59:48.428 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: DFIR-IRIS
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.01165s ago] ('DFIR-IRIS', '2.0', 'https://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to DFIR-IRIS.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.01639s ago] ('Velociraptor',)
2024-06-14 15:59:48.431 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Velociraptor
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.01372s ago] ('Velociraptor', '0.6.8', 'https://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, None, 'Connection to Velociraptor. Make sure you have generated the api file first.', 'Not specified.', 1, 0, 0, 0, 0, 1, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.01846s ago] ('Sublime',)
2024-06-14 15:59:48.433 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Sublime
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.01577s ago] ('Sublime', '3', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to Sublime.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.02056s ago] ('InfluxDB',)
2024-06-14 15:59:48.435 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: InfluxDB
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.01798s ago] ('InfluxDB', '3', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to InfluxDB.', 'Not specified.', 1, 0, 0, 1, 0, 0, 1, 'dummy,dummy', 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.02284s ago] ('AskSocfortress',)
2024-06-14 15:59:48.437 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: AskSocfortress
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.02022s ago] ('AskSocfortress', '3', 'https://knowledge.socfortress.co', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to AskSocfortress. Make sure you have requested an API key.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.02496s ago] ('SocfortressThreatIntel',)
2024-06-14 15:59:48.439 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: SocfortressThreatIntel
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.02229s ago] ('SocfortressThreatIntel', '3', 'https://intel.socfortress.co/search', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to Socfortress Threat Intel. Make sure you have requested an API key.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.02706s ago] ('Cortex',)
2024-06-14 15:59:48.441 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Cortex
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.02434s ago] ('Cortex', '3', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, 'dummy', 'Connection to Cortex. Make sure you have created an API key.', 'Not specified.', 1, 0, 0, 1, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.02904s ago] ('Grafana',)
2024-06-14 15:59:48.443 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Grafana
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.0263s ago] ('Grafana', '3', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), 'dummy', 'dummy', None, 'Connection to Grafana.', 'Not specified.', 1, 0, 0, 0, 1, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.03112s ago] ('Wazuh Worker Provisioning',)
2024-06-14 15:59:48.445 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Wazuh Worker Provisioning
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.02838s ago] ('Wazuh Worker Provisioning', '3', 'http://127.1.1.1', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, None, 'Connection to Wazuh Worker Provisioning. Make sure you have deployed the Wazuh Worker Provisioning Application provided by SOCFortress: https://github.com/socfortress/Customer-Provisioning-Worker', 'Not specified.', 1, 0, 1, 0, 0, 0, 0, None, 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.03319s ago] ('Event Shipper',)
2024-06-14 15:59:48.447 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: Event Shipper
INFO [sqlalchemy.engine.Engine] INSERT INTO connectors (connector_name, connector_type, connector_url, connector_last_updated, connector_username, connector_password, connector_api_key, connector_description, connector_supports, connector_configured, connector_verified, connector_accepts_host_only, connector_accepts_api_key, connector_accepts_username_password, connector_accepts_file, connector_accepts_extra_data, connector_extra_data, connector_enabled) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
INFO [sqlalchemy.engine.Engine] [cached since 0.03046s ago] ('Event Shipper', '3', 'graylog_host', datetime.datetime(2024, 6, 14, 15, 59, 44, 530728), None, None, None, 'Connection to Graylog GELF Input to receive events from integrations. Make sure you have created a GELF Input in Graylog.', 'Not specified.', 1, 0, 1, 0, 0, 0, 1, 'gelf_port', 0)
INFO [sqlalchemy.engine.Engine] SELECT connectors.id, connectors.connector_name, connectors.connector_type, connectors.connector_url, connectors.connector_last_updated, connectors.connector_username, connectors.connector_password, connectors.connector_api_key, connectors.connector_description, connectors.connector_supports, connectors.connector_configured, connectors.connector_verified, connectors.connector_accepts_host_only, connectors.connector_accepts_api_key, connectors.connector_accepts_username_password, connectors.connector_accepts_file, connectors.connector_accepts_extra_data, connectors.connector_extra_data, connectors.connector_enabled
FROM connectors
WHERE connectors.connector_name = %s
INFO [sqlalchemy.engine.Engine] [cached since 0.0352s ago] ('HAProxy Provisioning',)
2024-06-14 15:59:48.449 | INFO | app.db.db_populate:add_connectors_if_not_exist:177 - Added new connector: HAProxy Provisioning
2024-06-14 15:59:48.450 | INFO | app.db.db_populate:add_connectors_if_not_exist:181 - before commit
INFO [sqlalchemy.engine.Engine] ROLLBACK
Exception ignored in: <function Connection.__del__ at 0x7f243b165bc0>
Traceback (most recent call last):
File "/opt/venv/lib/python3.11/site-packages/aiomysql/connection.py", line 1131, in __del__
File "/opt/venv/lib/python3.11/site-packages/aiomysql/connection.py", line 339, in close
File "/usr/lib/python3.11/asyncio/selector_events.py", line 864, in close
File "/usr/lib/python3.11/asyncio/base_events.py", line 762, in call_soon
File "/usr/lib/python3.11/asyncio/base_events.py", line 520, in _check_closed
RuntimeError: Event loop is closedOkay, I couldn't get docker to work but I believe that's due to another reason.
@taylorwalton However, good news, I believe I've found a fix. await session.commit() is redundant when using session.begin() which as it turns out handles rollback and commit. I am tired and will double check tomorrow but looks to be working if the line for await session.commit() is removed.
taylorwalton
commented
2 weeks ago sounds good, goodluck!
ccben87
commented
2 weeks ago So this turns out to be an issue with using HA MariaDB cluster. Not sure why it was also failing when I tried a single Mysql container but I can't replicate the failure. No issues with the code to be fixed. I'll close this off. Thanks for your time!
Describe the bug
When first starting the container, during adding of connectors, the below log and error is produced. Error suggests that there's an issue with the async operations not being handled properly. I will try to resolve myself.
To Reproduce Steps to reproduce the behavior:
Run backend container with v0.0.9
Expected behavior Connectors should be loaded without issue.
Screenshots If applicable, add screenshots to help explain your problem.
**Container Logs