Open ebelious opened 15 hours ago
Describe the bug Not all of the agents are showing connected to CoPilot. When attempting to sync, there is an error (I removed the DNS name)
500 POST http://COPILOT_IP/api/agents/sync (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (copilot.agents, CONSTRAINT agents_ibfk_1 FOREIGN KEY (customer_code) REFERENCES customers (customer_code))') [SQL: INSERT INTO agents (agent_id, ip_address, os, hostname, label, critical_asset, wazuh_last_seen, velociraptor_id, velociraptor_last_seen, wazuh_agent_version, wazuh_agent_status, velociraptor_agent_version, customer_code, quarantined, velociraptor_org) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)] [parameters: ('001', '10.1.1.10', 'Debian GNU/Linux', 'proxmox-backup', 'Linux_Server', 0, datetime.datetime(2024, 10, 2, 19, 50, 54), None, None, 'Wazuh v4.7.4', 'active', None, 'Server', 0, None)] (Background on this error at: https://sqlalche.me/e/14/gkpj)
copilot
agents
agents_ibfk_1
customer_code
customers
To Reproduce Steps to reproduce the behavior:
Expected behavior The agents should be populating, the only one that populates is the wazuh agent
/opt/CoPilot# docker-compose logs copilot-mysql Attaching to copilot_copilot-mysql_1 copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1debian12 started. copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1debian12 started. copilot-mysql_1 | 2024-10-02 19:17:47+00:00 [Note] [Entrypoint]: Initializing database files copilot-mysql_1 | 2024-10-02T19:17:47.177921Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.38) initializing of server in progress as process 81 copilot-mysql_1 | 2024-10-02T19:17:47.213268Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:17:55.544742Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:18:08.817854Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option. copilot-mysql_1 | 2024-10-02 19:18:39+00:00 [Note] [Entrypoint]: Database files initialized copilot-mysql_1 | 2024-10-02 19:18:39+00:00 [Note] [Entrypoint]: Starting temporary server copilot-mysql_1 | 2024-10-02T19:18:40.031692Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.38) starting as process 125 copilot-mysql_1 | 2024-10-02T19:18:40.073764Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:18:42.402461Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:18:44.357818Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. copilot-mysql_1 | 2024-10-02T19:18:44.357884Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. copilot-mysql_1 | 2024-10-02T19:18:44.382281Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory. copilot-mysql_1 | 2024-10-02T19:18:44.416362Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock copilot-mysql_1 | 2024-10-02T19:18:44.416532Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.38' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server - GPL. copilot-mysql_1 | 2024-10-02 19:18:44+00:00 [Note] [Entrypoint]: Temporary server started. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Creating database copilot copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Creating user copilot copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Giving user copilot access to schema copilot copilot-mysql_1 | copilot-mysql_1 | 2024-10-02 19:18:51+00:00 [Note] [Entrypoint]: Stopping temporary server copilot-mysql_1 | 2024-10-02T19:18:51.292639Z 13 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.38). copilot-mysql_1 | 2024-10-02T19:18:59.440279Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.38) MySQL Community Server - GPL. copilot-mysql_1 | 2024-10-02 19:19:00+00:00 [Note] [Entrypoint]: Temporary server stopped copilot-mysql_1 | copilot-mysql_1 | 2024-10-02 19:19:00+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up. copilot-mysql_1 | copilot-mysql_1 | 2024-10-02T19:19:00.623400Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.38) starting as process 1 copilot-mysql_1 | 2024-10-02T19:19:00.634922Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:19:03.072370Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:19:04.656077Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. copilot-mysql_1 | 2024-10-02T19:19:04.656145Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. copilot-mysql_1 | 2024-10-02T19:19:04.668455Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory. copilot-mysql_1 | 2024-10-02T19:19:04.703001Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock copilot-mysql_1 | 2024-10-02T19:19:04.703191Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.38' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
copilot-backend_1 | 2024-10-02 19:50:06.110 | INFO | app.agents.routes.agents:get_agents:164 - Fetching all agents copilot-backend_1 | 2024-10-02 19:50:06.114 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.agents.routes.agents:sync_all_agents:293 - Syncing agents as part of scheduled job copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.agents.wazuh.services.agents:collect_wazuh_agents:22 - Collecting all agents from Wazuh Manager copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.connectors.wazuh_manager.utils.universal:send_get_request:141 - Sending GET request to /agents copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:85 - Getting Wazuh Manager authentication token copilot-backend_1 | 2024-10-02 19:50:07.362 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | 2024-10-02 19:50:07.368 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:92 - Verifying the wazuh-manager connection to https://10.1.1.9:55000 copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:50:07.889 | DEBUG | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:106 - Wazuh Authentication Token successful copilot-backend_1 | 2024-10-02 19:50:07.889 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:50:07.981 | INFO | app.agents.services.sync:sync_agents_wazuh:249 - Collected Wazuh Agents: agents=[WazuhAgent(agent_id='000', agent_name='wazuh', agent_ip='127.0.0.1', agent_os='Ubuntu', agent_label='Unknown', agent_last_seen='9999-12-31T23:59:59+00:00', wazuh_agent_version='Wazuh v4.8.2', wazuh_agent_status='active'), WazuhAgent(agent_id='001', agent_name='proxmox-backup', agent_ip='10.1.1.10', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='002', agent_name='term-server', agent_ip='10.1.1.3', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='003', agent_name='twingate', agent_ip='10.1.1.2', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='004', agent_name='proxy-manager', agent_ip='10.1.1.14', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='005', agent_name='shuffle', agent_ip='any', agent_os='Unknown', agent_label='Linux_Server', agent_last_seen='Unknown', wazuh_agent_version='n/a', wazuh_agent_status='never_connected'), WazuhAgent(agent_id='006', agent_name='adguard', agent_ip='10.1.1.13', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='007', agent_name='AD-WIN-2022', agent_ip='10.1.1.18', agent_os='Microsoft Windows Server 2022 Standard', agent_label='Windows_Server', agent_last_seen='2024-10-02T19:49:58+00:00', wazuh_agent_version='Wazuh v4.8.0', wazuh_agent_status='active'), WazuhAgent(agent_id='008', agent_name='nessus', agent_ip='10.1.1.11', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='009', agent_name='proxmox', agent_ip='10.1.1.7', agent_os='Debian GNU/Linux', agent_label='Linux_Endpoint', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='010', agent_name='twingate2', agent_ip='10.1.1.12', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active')] success=True message='Agents collected successfully' copilot-backend_1 | 2024-10-02 19:50:07.981 | INFO
copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:58:41.049 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 19:58:41.098 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 19:58:41.098 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:10.028 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:10.029 | INFO | app.agents.routes.agents:get_agents:164 - Fetching all agents copilot-backend_1 | 2024-10-02 19:59:10.033 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.agents.routes.agents:sync_all_agents:293 - Syncing agents as part of scheduled job copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.agents.wazuh.services.agents:collect_wazuh_agents:22 - Collecting all agents from Wazuh Manager copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.connectors.wazuh_manager.utils.universal:send_get_request:141 - Sending GET request to /agents copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:85 - Getting Wazuh Manager authentication token copilot-backend_1 | 2024-10-02 19:59:11.329 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | 2024-10-02 19:59:11.335 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:92 - Verifying the wazuh-manager connection to https://10.1.1.9:55000 copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:12.021 | DEBUG | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:106 - Wazuh Authentication Token successful copilot-backend_1 | 2024-10-02 19:59:12.021 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:12.121 | INFO | app.agents.services.sync:sync_agents_wazuh:249 - Collected Wazuh Agents: agents=[WazuhAgent(agent_id='000', agent_name='wazuh', agent_ip='127.0.0.1', agent_os='Ubuntu', agent_label='Unknown', agent_last_seen='9999-12-31T23:59:59+00:00', wazuh_agent_version='Wazuh v4.8.2', wazuh_agent_status='active'), WazuhAgent(agent_id='001', agent_name='proxmox-backup', agent_ip='10.1.1.10', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='002', agent_name='term-server', agent_ip='10.1.1.3', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='003', agent_name='twingate', agent_ip='10.1.1.2', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='004', agent_name='proxy-manager', agent_ip='10.1.1.14', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='005', agent_name='shuffle', agent_ip='any', agent_os='Unknown', agent_label='Linux_Server', agent_last_seen='Unknown', wazuh_agent_version='n/a', wazuh_agent_status='never_connected'), WazuhAgent(agent_id='006', agent_name='adguard', agent_ip='10.1.1.13', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='007', agent_name='AD-WIN-2022', agent_ip='10.1.1.18', agent_os='Microsoft Windows Server 2022 Standard', agent_label='Windows_Server', agent_last_seen='2024-10-02T19:59:09+00:00', wazuh_agent_version='Wazuh v4.8.0', wazuh_agent_status='active'), WazuhAgent(agent_id='008', agent_name='nessus', agent_ip='10.1.1.11', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='009', agent_name='proxmox', agent_ip='10.1.1.7', agent_os='Debian GNU/Linux', agent_label='Linux_Endpoint', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='010', agent_name='twingate2', agent_ip='10.1.1.12', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active')] success=True message='Agents collected successfully' copilot-backend_1 | 2024-10-02 19:59:12.121 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:12.125 | INFO | app.agents.services.sync:update_wazuh_agent_in_db:151 - Agent wazuh updated in the database copilot-backend_1 | 2024-10-02 19:59:12.128 | INFO | app.agents.services.sync:add_wazuh_agent_in_db:121 - Adding agent proxmox-backup to the database copilot-backend_1 | 2024-10-02 19:59:12.156 | ERROR | app.agents.services.sync:add_wazuh_agent_in_db:125 - Failed to add agent proxmox-backup to the database: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (copilot.agents, CONSTRAINT agents_ibfk_1 FOREIGN KEY (customer_code) REFERENCES customers (customer_code))') copilot-backend_1 | [SQL: INSERT INTO agents (agent_id, ip_address, os, hostname, label, critical_asset, wazuh_last_seen, velociraptor_id, velociraptor_last_seen, wazuh_agent_version, wazuh_agent_status, velociraptor_agent_version, customer_code, quarantined, velociraptor_org) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)] copilot-backend_1 | [parameters: ('001', '10.1.1.10', 'Debian GNU/Linux', 'proxmox-backup', 'Linux_Server', 0, datetime.datetime(2024, 10, 2, 19, 59, 4), None, None, 'Wazuh v4.7.4', 'active', None, 'Server', 0, None)] copilot-backend_1 | (Background on this error at: https://sqlalche.me/e/14/gkpj) copilot-backend_1 | 2024-10-02 19:59:12.156 | ERROR | app.db.db_session:get_db_session:64 - Error during DB session: copilot-backend_1 | 2024-10-02 19:59:12.157 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:40.959 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 19:59:40.960 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:40.960 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 19:59:40.970 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:40.986 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 19:59:41.022 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 19:59:41.023 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.schedulers.services.invoke_alert_creation:invoke_alert_creation_collect:21 - Invoking alert creation collection via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.146 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:00:41.147 | INFO | app.schedulers.services.invoke_sigma_queries:invoke_sigma_queries_collect:18 - Invoking sigma queries collection via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.147 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.153 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:00:41.153 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.154 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:00:41.156 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.160 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.incidents.services.alert_collection:get_alerts_not_created_in_copilot:63 - Indices: [] copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:00:41.188 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:00:41.190 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.193 | INFO | app.incidents.services.alert_collection:get_alerts_not_created_in_copilot:72 - Alerts not created: 0 alerts found copilot-backend_1 | 2024-10-02 20:00:41.193 | INFO | app.incidents.routes.incident_alert:create_alert_auto_route:157 - Alerts to create in CoPilot: alerts=[] success=True message='Success' copilot-backend_1 | 2024-10-02 20:00:41.227 | INFO | app.schedulers.services.invoke_sigma_queries:invoke_sigma_queries_collect:30 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.227 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.259 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.260 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.282 | INFO | app.schedulers.services.invoke_alert_creation:invoke_alert_creation_collect:33 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.282 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:01:41.101 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:01:41.101 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:01:41.102 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:01:41.107 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:01:41.108 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:01:41.108 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:01:41.113 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:01:41.130 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:01:41.192 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:01:41.193 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:02:40.998 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:02:40.998 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:02:40.999 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:02:41.007 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:02:41.022 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:02:41.046 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:02:41.046 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:03:41.034 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:03:41.035 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:03:41.035 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:03:41.043 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:03:41.059 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:03:41.166 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:03:41.166 | INFO | app.db.db_session:get_db_session:68 - Closing DB session
Describe the bug Not all of the agents are showing connected to CoPilot. When attempting to sync, there is an error (I removed the DNS name)
500 POST http://COPILOT_IP/api/agents/sync (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (
copilot.agents, CONSTRAINTagents_ibfk_1FOREIGN KEY (customer_code) REFERENCEScustomers(customer_code))') [SQL: INSERT INTO agents (agent_id, ip_address, os, hostname, label, critical_asset, wazuh_last_seen, velociraptor_id, velociraptor_last_seen, wazuh_agent_version, wazuh_agent_status, velociraptor_agent_version, customer_code, quarantined, velociraptor_org) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)] [parameters: ('001', '10.1.1.10', 'Debian GNU/Linux', 'proxmox-backup', 'Linux_Server', 0, datetime.datetime(2024, 10, 2, 19, 50, 54), None, None, 'Wazuh v4.7.4', 'active', None, 'Server', 0, None)] (Background on this error at: https://sqlalche.me/e/14/gkpj)To Reproduce Steps to reproduce the behavior:
copilot.agents, CONSTRAINTagents_ibfk_1FOREIGN KEY (customer_code) REFERENCEScustomers(customer_code))') [SQL: INSERT INTO agents (agent_id, ip_address, os, hostname, label, critical_asset, wazuh_last_seen, velociraptor_id, velociraptor_last_seen, wazuh_agent_version, wazuh_agent_status, velociraptor_agent_version, customer_code, quarantined, velociraptor_org) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)] [parameters: ('001', '10.1.1.10', 'Debian GNU/Linux', 'proxmox-backup', 'Linux_Server', 0, datetime.datetime(2024, 10, 2, 19, 50, 54), None, None, 'Wazuh v4.7.4', 'active', None, 'Server', 0, None)] (Background on this error at: https://sqlalche.me/e/14/gkpj)Expected behavior The agents should be populating, the only one that populates is the wazuh agent
MYSQL Logs
/opt/CoPilot# docker-compose logs copilot-mysql Attaching to copilot_copilot-mysql_1 copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1debian12 started. copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' copilot-mysql_1 | 2024-10-02 19:17:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1debian12 started. copilot-mysql_1 | 2024-10-02 19:17:47+00:00 [Note] [Entrypoint]: Initializing database files copilot-mysql_1 | 2024-10-02T19:17:47.177921Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.38) initializing of server in progress as process 81 copilot-mysql_1 | 2024-10-02T19:17:47.213268Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:17:55.544742Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:18:08.817854Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option. copilot-mysql_1 | 2024-10-02 19:18:39+00:00 [Note] [Entrypoint]: Database files initialized copilot-mysql_1 | 2024-10-02 19:18:39+00:00 [Note] [Entrypoint]: Starting temporary server copilot-mysql_1 | 2024-10-02T19:18:40.031692Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.38) starting as process 125 copilot-mysql_1 | 2024-10-02T19:18:40.073764Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:18:42.402461Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:18:44.357818Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. copilot-mysql_1 | 2024-10-02T19:18:44.357884Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. copilot-mysql_1 | 2024-10-02T19:18:44.382281Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory. copilot-mysql_1 | 2024-10-02T19:18:44.416362Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock copilot-mysql_1 | 2024-10-02T19:18:44.416532Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.38' socket: '/var/run/mysqld/mysqld.sock' port: 0 MySQL Community Server - GPL. copilot-mysql_1 | 2024-10-02 19:18:44+00:00 [Note] [Entrypoint]: Temporary server started. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it. copilot-mysql_1 | Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it. copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Creating database copilot copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Creating user copilot copilot-mysql_1 | 2024-10-02 19:18:50+00:00 [Note] [Entrypoint]: Giving user copilot access to schema copilot copilot-mysql_1 | copilot-mysql_1 | 2024-10-02 19:18:51+00:00 [Note] [Entrypoint]: Stopping temporary server copilot-mysql_1 | 2024-10-02T19:18:51.292639Z 13 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.38). copilot-mysql_1 | 2024-10-02T19:18:59.440279Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.38) MySQL Community Server - GPL. copilot-mysql_1 | 2024-10-02 19:19:00+00:00 [Note] [Entrypoint]: Temporary server stopped copilot-mysql_1 | copilot-mysql_1 | 2024-10-02 19:19:00+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up. copilot-mysql_1 | copilot-mysql_1 | 2024-10-02T19:19:00.623400Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.38) starting as process 1 copilot-mysql_1 | 2024-10-02T19:19:00.634922Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. copilot-mysql_1 | 2024-10-02T19:19:03.072370Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. copilot-mysql_1 | 2024-10-02T19:19:04.656077Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. copilot-mysql_1 | 2024-10-02T19:19:04.656145Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. copilot-mysql_1 | 2024-10-02T19:19:04.668455Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory. copilot-mysql_1 | 2024-10-02T19:19:04.703001Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock copilot-mysql_1 | 2024-10-02T19:19:04.703191Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.38' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server - GPL.
FRONTEND Logs
copilot-backend_1 | 2024-10-02 19:50:06.110 | INFO | app.agents.routes.agents:get_agents:164 - Fetching all agents copilot-backend_1 | 2024-10-02 19:50:06.114 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.agents.routes.agents:sync_all_agents:293 - Syncing agents as part of scheduled job copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.agents.wazuh.services.agents:collect_wazuh_agents:22 - Collecting all agents from Wazuh Manager copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.connectors.wazuh_manager.utils.universal:send_get_request:141 - Sending GET request to /agents copilot-backend_1 | 2024-10-02 19:50:07.361 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:85 - Getting Wazuh Manager authentication token copilot-backend_1 | 2024-10-02 19:50:07.362 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | 2024-10-02 19:50:07.368 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:92 - Verifying the wazuh-manager connection to https://10.1.1.9:55000 copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:50:07.889 | DEBUG | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:106 - Wazuh Authentication Token successful copilot-backend_1 | 2024-10-02 19:50:07.889 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:50:07.981 | INFO | app.agents.services.sync:sync_agents_wazuh:249 - Collected Wazuh Agents: agents=[WazuhAgent(agent_id='000', agent_name='wazuh', agent_ip='127.0.0.1', agent_os='Ubuntu', agent_label='Unknown', agent_last_seen='9999-12-31T23:59:59+00:00', wazuh_agent_version='Wazuh v4.8.2', wazuh_agent_status='active'), WazuhAgent(agent_id='001', agent_name='proxmox-backup', agent_ip='10.1.1.10', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='002', agent_name='term-server', agent_ip='10.1.1.3', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='003', agent_name='twingate', agent_ip='10.1.1.2', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='004', agent_name='proxy-manager', agent_ip='10.1.1.14', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='005', agent_name='shuffle', agent_ip='any', agent_os='Unknown', agent_label='Linux_Server', agent_last_seen='Unknown', wazuh_agent_version='n/a', wazuh_agent_status='never_connected'), WazuhAgent(agent_id='006', agent_name='adguard', agent_ip='10.1.1.13', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='007', agent_name='AD-WIN-2022', agent_ip='10.1.1.18', agent_os='Microsoft Windows Server 2022 Standard', agent_label='Windows_Server', agent_last_seen='2024-10-02T19:49:58+00:00', wazuh_agent_version='Wazuh v4.8.0', wazuh_agent_status='active'), WazuhAgent(agent_id='008', agent_name='nessus', agent_ip='10.1.1.11', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='009', agent_name='proxmox', agent_ip='10.1.1.7', agent_os='Debian GNU/Linux', agent_label='Linux_Endpoint', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='010', agent_name='twingate2', agent_ip='10.1.1.12', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:50:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active')] success=True message='Agents collected successfully' copilot-backend_1 | 2024-10-02 19:50:07.981 | INFO
BACKEND Logs
copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:58:41.049 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 19:58:41.098 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 19:58:41.098 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:10.028 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:10.029 | INFO | app.agents.routes.agents:get_agents:164 - Fetching all agents copilot-backend_1 | 2024-10-02 19:59:10.033 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.agents.routes.agents:sync_all_agents:293 - Syncing agents as part of scheduled job copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.agents.wazuh.services.agents:collect_wazuh_agents:22 - Collecting all agents from Wazuh Manager copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.connectors.wazuh_manager.utils.universal:send_get_request:141 - Sending GET request to /agents copilot-backend_1 | 2024-10-02 19:59:11.328 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:85 - Getting Wazuh Manager authentication token copilot-backend_1 | 2024-10-02 19:59:11.329 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | 2024-10-02 19:59:11.335 | INFO | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:92 - Verifying the wazuh-manager connection to https://10.1.1.9:55000 copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:12.021 | DEBUG | app.connectors.wazuh_manager.utils.universal:create_wazuh_manager_client:106 - Wazuh Authentication Token successful copilot-backend_1 | 2024-10-02 19:59:12.021 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Manager from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:12.121 | INFO | app.agents.services.sync:sync_agents_wazuh:249 - Collected Wazuh Agents: agents=[WazuhAgent(agent_id='000', agent_name='wazuh', agent_ip='127.0.0.1', agent_os='Ubuntu', agent_label='Unknown', agent_last_seen='9999-12-31T23:59:59+00:00', wazuh_agent_version='Wazuh v4.8.2', wazuh_agent_status='active'), WazuhAgent(agent_id='001', agent_name='proxmox-backup', agent_ip='10.1.1.10', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='002', agent_name='term-server', agent_ip='10.1.1.3', agent_os='Debian GNU/Linux', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='003', agent_name='twingate', agent_ip='10.1.1.2', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='004', agent_name='proxy-manager', agent_ip='10.1.1.14', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:04+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='005', agent_name='shuffle', agent_ip='any', agent_os='Unknown', agent_label='Linux_Server', agent_last_seen='Unknown', wazuh_agent_version='n/a', wazuh_agent_status='never_connected'), WazuhAgent(agent_id='006', agent_name='adguard', agent_ip='10.1.1.13', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='007', agent_name='AD-WIN-2022', agent_ip='10.1.1.18', agent_os='Microsoft Windows Server 2022 Standard', agent_label='Windows_Server', agent_last_seen='2024-10-02T19:59:09+00:00', wazuh_agent_version='Wazuh v4.8.0', wazuh_agent_status='active'), WazuhAgent(agent_id='008', agent_name='nessus', agent_ip='10.1.1.11', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='009', agent_name='proxmox', agent_ip='10.1.1.7', agent_os='Debian GNU/Linux', agent_label='Linux_Endpoint', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active'), WazuhAgent(agent_id='010', agent_name='twingate2', agent_ip='10.1.1.12', agent_os='Ubuntu', agent_label='Linux_Server', agent_last_seen='2024-10-02T19:59:03+00:00', wazuh_agent_version='Wazuh v4.7.4', wazuh_agent_status='active')] success=True message='Agents collected successfully' copilot-backend_1 | 2024-10-02 19:59:12.121 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:12.125 | INFO | app.agents.services.sync:update_wazuh_agent_in_db:151 - Agent wazuh updated in the database copilot-backend_1 | 2024-10-02 19:59:12.128 | INFO | app.agents.services.sync:add_wazuh_agent_in_db:121 - Adding agent proxmox-backup to the database copilot-backend_1 | 2024-10-02 19:59:12.156 | ERROR | app.agents.services.sync:add_wazuh_agent_in_db:125 - Failed to add agent proxmox-backup to the database: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (
copilot.agents, CONSTRAINTagents_ibfk_1FOREIGN KEY (customer_code) REFERENCEScustomers(customer_code))') copilot-backend_1 | [SQL: INSERT INTO agents (agent_id, ip_address, os, hostname, label, critical_asset, wazuh_last_seen, velociraptor_id, velociraptor_last_seen, wazuh_agent_version, wazuh_agent_status, velociraptor_agent_version, customer_code, quarantined, velociraptor_org) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)] copilot-backend_1 | [parameters: ('001', '10.1.1.10', 'Debian GNU/Linux', 'proxmox-backup', 'Linux_Server', 0, datetime.datetime(2024, 10, 2, 19, 59, 4), None, None, 'Wazuh v4.7.4', 'active', None, 'Server', 0, None)] copilot-backend_1 | (Background on this error at: https://sqlalche.me/e/14/gkpj) copilot-backend_1 | 2024-10-02 19:59:12.156 | ERROR | app.db.db_session:get_db_session:64 - Error during DB session: copilot-backend_1 | 2024-10-02 19:59:12.157 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 19:59:40.959 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 19:59:40.960 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:40.960 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 19:59:40.966 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 19:59:40.970 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 19:59:40.986 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 19:59:41.022 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 19:59:41.023 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.143 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.schedulers.services.invoke_alert_creation:invoke_alert_creation_collect:21 - Invoking alert creation collection via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.145 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.146 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:00:41.147 | INFO | app.schedulers.services.invoke_sigma_queries:invoke_sigma_queries_collect:18 - Invoking sigma queries collection via scheduler... copilot-backend_1 | 2024-10-02 20:00:41.147 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.153 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:00:41.153 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.154 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:00:41.156 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.160 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.incidents.services.alert_collection:get_alerts_not_created_in_copilot:63 - Indices: [] copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:00:41.172 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:00:41.188 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:00:41.190 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.193 | INFO | app.incidents.services.alert_collection:get_alerts_not_created_in_copilot:72 - Alerts not created: 0 alerts found copilot-backend_1 | 2024-10-02 20:00:41.193 | INFO | app.incidents.routes.incident_alert:create_alert_auto_route:157 - Alerts to create in CoPilot: alerts=[] success=True message='Success' copilot-backend_1 | 2024-10-02 20:00:41.227 | INFO | app.schedulers.services.invoke_sigma_queries:invoke_sigma_queries_collect:30 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.227 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.259 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.260 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:00:41.282 | INFO | app.schedulers.services.invoke_alert_creation:invoke_alert_creation_collect:33 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:00:41.282 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:01:41.101 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:01:41.101 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:01:41.102 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:01:41.107 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:01:41.108 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:01:41.108 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:01:41.113 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:01:41.130 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:01:41.192 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:01:41.193 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:02:40.998 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:02:40.998 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:02:40.999 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:02:41.003 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:02:41.007 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:02:41.022 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:02:41.046 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:02:41.046 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | 2024-10-02 20:03:41.034 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:24 - Resizing Wazuh index fields via scheduler... copilot-backend_1 | 2024-10-02 20:03:41.035 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:03:41.035 | INFO | app.connectors.utils:is_connector_verified:52 - Checking if connector Wazuh-Indexer is verified copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:29 - Wazuh Indexer connector is verified. copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.db.db_session:get_db_session:60 - DB session created copilot-backend_1 | 2024-10-02 20:03:41.039 | INFO | app.connectors.utils:get_connector_info_from_db:29 - Fetching connector Wazuh-Indexer from database copilot-backend_1 | 2024-10-02 20:03:41.043 | INFO | app.db.db_session:get_db_session:68 - Closing DB session copilot-backend_1 | /opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py:1056: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.1.9'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings copilot-backend_1 | warnings.warn( copilot-backend_1 | 2024-10-02 20:03:41.059 | INFO | app.connectors.wazuh_indexer.utils.universal:resize_wazuh_index_fields:588 - Successfully resized the Wazuh index fields copilot-backend_1 | 2024-10-02 20:03:41.166 | INFO | app.schedulers.services.wazuh_index_resize:resize_wazuh_index_fields:40 - Updated job metadata with the last success timestamp. copilot-backend_1 | 2024-10-02 20:03:41.166 | INFO | app.db.db_session:get_db_session:68 - Closing DB session