socfortress / CoPilot

SOCFortress CoPilot
https://www.socfortress.co
GNU Affero General Public License v3.0
185 stars 39 forks source link

error on startup - log file #123

Closed thisledome5 closed 7 months ago

thisledome5 commented 7 months ago

Describe the bug exit NPM on startup

After modifying the .env and saving, then running docker-compose up -d, I can not get to the Ip address:5173

after checking logs, the following errors are appearing

INFO: Application startup complete. [0] INFO: Uvicorn running on http://208.87.135.193:5000 (Press CTRL+C to quit) [1] node:internal/errors:496 [1] ErrorCaptureStackTrace(err);  ^ [1] [1] Error: ENOSPC: System limit for number of file watchers reached, watch '/opt/copilot/backend/.venv/Lib/site-packages/influxdb_client/domain/template_export_by_id_resource_filters.py' [1] at FSWatcher. (node:internal/fs/watchers:247:19) [1] at Object.watch (node:fs:2418:34) [1] at createFsWatchInstance (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46210:17) [1] at setFsWatchListener (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46257:15) [1] at NodeFsHandler._watchWithNodeFs (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46412:14) [1] at NodeFsHandler._handleFile (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46476:23) [1] at NodeFsHandler._addToNodeFs (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46718:21) [1] Emitted 'error' event on FSWatcher instance at: [1] at FSWatcher._handleError (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:47909:10) [1] at NodeFsHandler._addToNodeFs (file:///opt/copilot/node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:46726:18) { [1] errno: -28, [1] syscall: 'watch', [1] code: 'ENOSPC', [1] path: '/opt/copilot/backend/.venv/Lib/site-packages/influxdb_client/domain/template_export_by_id_resource_filters.py', [1] filename: '/opt/copilot/backend/.venv/Lib/site-packages/influxdb_client/domain/template_export_by_id_resource_filters.py' [1] } [1] [1] Node.js v18.19.0 [1] npm run start-vue exited with code 1l


[_copilot_app_1_logs.txt](https://github.com/socfortress/CoPilot/files/14119870/_copilot_app_1_logs.txt)
taylorwalton commented 7 months ago

Hey there,

Try increasing your number of file watchers on the system.

nano /etc/sysctl.conf

add the following at the end of the file

fs.inotify.max_user_watches=524288

reload system settings

sysctl -p
thisledome5 commented 7 months ago

Thanks Taylor

I got around it by switching to another Docker server on another machine. The system loads up fine now. but when I go to login, it gives me an error at the top in red An error occured. please try again later. see image

The docker package loads on the web browser. there are a few errors in the docker container log but i can't make sense of them attached as well There were no errors when the docker compose file ran after I removed the ports from docker-compose It was conflicting with an error that said either remove the environment "hosts" network or remove ports.

I can not get it to run up as per your docker compose file if I leave it alone


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:05 AM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Hey there,

Try increasing your number of file watchers on the system.

nano /etc/sysctl.conf

add the following at the end of the file

fs.inotify.max_user_watches=524288

reload system settings

sysctl -p

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1921405095, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLCT43F36DYIVIGHZODYROOKTAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDKMBZGU. You are receiving this because you authored the thread.Message ID: @.***>

[0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn [0] conn = connection.create_connection( [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection [0] raise err [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection [0] sock.connect(sa) [0] ConnectionRefusedError: [Errno 111] Connection refused [0] [0] During handling of the above exception, another exception occurred: [0] [0] Traceback (most recent call last): [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 714, in urlopen [0] httplib_response = self._make_request( [0] ^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 415, in _make_request [0] conn.request(method, url, *httplib_request_kw) [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 244, in request [0] super(HTTPConnection, self).request(method, url, body=body, headers=headers) [0] File "/usr/lib/python3.11/http/client.py", line 1294, in request [0] self._send_request(method, url, body, headers, encode_chunked) [0] File "/usr/lib/python3.11/http/client.py", line 1340, in _send_request [0] self.endheaders(body, encode_chunked=encode_chunked) [0] File "/usr/lib/python3.11/http/client.py", line 1289, in endheaders [0] self._send_output(message_body, encode_chunked=encode_chunked) [0] File "/usr/lib/python3.11/http/client.py", line 1048, in _send_output [0] self.send(msg) [0] File "/usr/lib/python3.11/http/client.py", line 986, in send [0] self.connect() [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect [0] conn = self._new_conn() [0] ^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn [0] raise NewConnectionError( [0] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fd4942cf750>: Failed to establish a new connection: [Errno 111] Connection refused [0] [0] During handling of the above exception, another exception occurred: [0] [0] Traceback (most recent call last): [0] File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 486, in send [0] resp = conn.urlopen( [0] ^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 798, in urlopen [0] retries = retries.increment( [0] ^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment [0] raise MaxRetryError(_pool, url, error or ResponseError(cause)) [0] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd4942cf750>: Failed to establish a new connection: [Errno 111] Connection refused')) [0] [0] During handling of the above exception, another exception occurred: [0] [0] Traceback (most recent call last): [0] File "/opt/venv/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job [0] retval = job.func(job.args, job.kwargs) [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/copilot/backend/app/schedulers/services/agent_sync.py", line 27, in agent_sync [0] headers = scheduler_login() [0] ^^^^^^^^^^^^^^^^^ [0] File "/opt/copilot/backend/app/schedulers/utils/universal.py", line 18, in scheduler_login [0] token_response = requests.post( [0] ^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/requests/api.py", line 115, in post [0] return request("post", url, data=data, json=json, kwargs) [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request [0] return session.request(method=method, url=url, kwargs) [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request [0] resp = self.send(prep, send_kwargs) [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send [0] r = adapter.send(request, **kwargs) [0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [0] File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 519, in send [0] raise ConnectionError(e, request=request) [0] requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded with url: /auth/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd4942cf750>: Failed to establish a new connection: [Errno 111] Connection refused')) [0] WARNING: Invalid HTTP request received. [0] WARNING: Invalid HTTP request received. [0] INFO: 143.198.2.178:54478 - "GET / HTTP/1.1" 200 OK [1] (node:51) Warning: An error event has already been emitted on the socket. Please use the destroy method on the socket while handling a 'clientError' event. [1] (Use node --trace-warnings ... to show where the warning was created) [0] INFO: 1.146.101.197:2357 - "OPTIONS /wazuh_indexer/health HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2356 - "OPTIONS /graylog/metrics HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2358 - "OPTIONS /influxdb/alerts HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2309 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2311 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2310 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2352 - "OPTIONS /auth/token HTTP/1.1" 200 OK [0] 2024-02-02 00:42:05.188 | INFO | app.auth.utils:authenticate_user:102 - Password is not verified [0] INFO: 1.146.101.197:2352 - "POST /auth/token HTTP/1.1" 401 Unauthorized [0] 2024-02-02 00:42:23.765 | INFO | app.auth.utils:authenticate_user:102 - Password is not verified [0] INFO: 1.146.101.197:2305 - "POST /auth/token HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2306 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2307 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2306 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized [0] 2024-02-02 00:43:40.631 | INFO | app.auth.utils:authenticate_user:102 - Password is not verified [0] INFO: 1.146.101.197:2309 - "POST /auth/token HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2316 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2318 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2317 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized

thisledome5 commented 7 months ago

sorry i didnt realise the full log wasn't downloaded here it is


From: David Cartledge @.> Sent: Friday, February 2, 2024 10:48 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Thanks Taylor

I got around it by switching to another Docker server on another machine. The system loads up fine now. but when I go to login, it gives me an error at the top in red An error occured. please try again later. see image

The docker package loads on the web browser. there are a few errors in the docker container log but i can't make sense of them attached as well There were no errors when the docker compose file ran after I removed the ports from docker-compose It was conflicting with an error that said either remove the environment "hosts" network or remove ports.

I can not get it to run up as per your docker compose file if I leave it alone


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:05 AM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Hey there,

Try increasing your number of file watchers on the system.

nano /etc/sysctl.conf

add the following at the end of the file

fs.inotify.max_user_watches=524288

reload system settings

sysctl -p

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1921405095, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLCT43F36DYIVIGHZODYROOKTAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDKMBZGU. You are receiving this because you authored the thread.Message ID: @.***>

thisledome5 commented 7 months ago

fresh rebuild of CoPilot without ports on compose file


From: David Cartledge @.> Sent: Friday, February 2, 2024 11:23 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

sorry i didnt realise the full log wasn't downloaded here it is


From: David Cartledge @.> Sent: Friday, February 2, 2024 10:48 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Thanks Taylor

I got around it by switching to another Docker server on another machine. The system loads up fine now. but when I go to login, it gives me an error at the top in red An error occured. please try again later. see image

The docker package loads on the web browser. there are a few errors in the docker container log but i can't make sense of them attached as well There were no errors when the docker compose file ran after I removed the ports from docker-compose It was conflicting with an error that said either remove the environment "hosts" network or remove ports.

I can not get it to run up as per your docker compose file if I leave it alone


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:05 AM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Hey there,

Try increasing your number of file watchers on the system.

nano /etc/sysctl.conf

add the following at the end of the file

fs.inotify.max_user_watches=524288

reload system settings

sysctl -p

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1921405095, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLCT43F36DYIVIGHZODYROOKTAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDKMBZGU. You are receiving this because you authored the thread.Message ID: @.***>

thisledome5 commented 7 months ago

and this is the docker log build file ➜ CoPilot git:(main) ✗ docker logs cce3941af291 --follow

@.*** start concurrently "npm run start-server" "npm run start-vue"

[1] [1] > @. start-vue [1] > vite --host 0.0.0.0 [1] [0] [0] > @. start-server [0] > cd backend && /opt/venv/bin/python copilot.py [0] [0] 2024-02-02 01:27:26.384 | INFO | settings::15 - Loading environment from /opt/copilot/.env [1] Re-optimizing dependencies because lockfile has changed [1] [1] VITE v5.0.12 ready in 1413 ms [1] [1] ➜ Local: http://localhost:5173/ [1] ➜ Network: http://208.87.135.101:5173/ [1] ➜ Network: http://172.29.0.122:5173/ [1] ➜ Network: http://10.120.193.5:5173/ [1] ➜ Network: http://172.18.0.1:5173/ [1] ➜ Network: http://172.17.0.1:5173/ [0] INFO: Started server process [58] [0] INFO: Waiting for application startup. [0] 2024-02-02 01:27:29.658 | INFO | app.db.db_setup:create_tables:26 - Creating tables [0] 2024-02-02 01:27:29.699 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Wazuh-Indexer from environment variables with URL: https://1.1.1.1:9200 [0] 2024-02-02 01:27:29.700 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Wazuh-Manager from environment variables with URL: https://1.1.1.1 [0] 2024-02-02 01:27:29.700 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Graylog from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.701 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Shuffle from environment variables with URL: https://1.1.1.1 [0] 2024-02-02 01:27:29.701 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for DFIR-IRIS from environment variables with URL: https://1.1.1.1 [0] 2024-02-02 01:27:29.701 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Velociraptor from environment variables with URL: https://1.1.1.1 [0] 2024-02-02 01:27:29.701 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Sublime from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.702 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for InfluxDB from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.702 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for AskSocfortress from environment variables with URL: https://knowledge.socfortress.co [0] 2024-02-02 01:27:29.702 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for SocfortressThreatIntel from environment variables with URL: https://intel.socfortress.co/search [0] 2024-02-02 01:27:29.702 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Cortex from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.702 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Grafana from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.703 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Wazuh Worker Provisioning from environment variables with URL: http://1.1.1.1 [0] 2024-02-02 01:27:29.703 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Event Shipper from environment variables with URL: graylog_host [0] 2024-02-02 01:27:29.703 | INFO | app.db.db_populate:load_connector_data:35 - Loading connector data for Alert Creation Provisioning from environment variables with URL: http://example.com [0] 2024-02-02 01:27:29.876 | INFO | app.db.db_setup:create_roles:46 - Creating roles [0] 2024-02-02 01:27:29.876 | INFO | app.db.db_populate:add_roles_if_not_exist:152 - Checking for existence of role admin [0] 2024-02-02 01:27:29.883 | INFO | app.db.db_populate:add_roles_if_not_exist:152 - Checking for existence of role analyst [0] 2024-02-02 01:27:29.886 | INFO | app.db.db_populate:add_roles_if_not_exist:152 - Checking for existence of role scheduler [0] 2024-02-02 01:27:29.889 | INFO | app.db.db_populate:add_roles_if_not_exist:163 - Role check and addition completed. [0] 2024-02-02 01:27:29.890 | INFO | app.db.db_setup:create_available_integrations:62 - Creating available integrations [0] 2024-02-02 01:27:29.891 | INFO | app.db.db_populate:load_available_integrations_data:177 - Loading available integrations data for Office365. [0] 2024-02-02 01:27:29.891 | INFO | app.db.db_populate:load_available_integrations_data:177 - Loading available integrations data for Mimecast. [0] 2024-02-02 01:27:29.905 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Office365. [0] 2024-02-02 01:27:29.908 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Office365. [0] 2024-02-02 01:27:29.911 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Office365. [0] 2024-02-02 01:27:29.913 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Office365. [0] 2024-02-02 01:27:29.916 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Mimecast. [0] 2024-02-02 01:27:29.919 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Mimecast. [0] 2024-02-02 01:27:29.921 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Mimecast. [0] 2024-02-02 01:27:29.923 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Mimecast. [0] 2024-02-02 01:27:29.926 | INFO | app.db.db_populate:load_available_integrations_auth_keys:260 - Loading available integrations auth keys data for Mimecast. [0] 2024-02-02 01:27:29.962 | INFO | app.db.db_setup:ensure_admin_user:80 - Ensuring admin user exists [0] 2024-02-02 01:27:29.971 | INFO | app.auth.services.universal:create_admin_user:121 - Admin user already exists. [0] 2024-02-02 01:27:29.974 | INFO | app.db.db_setup:ensure_scheduler_user:97 - Ensuring scheduler user exists [0] 2024-02-02 01:27:29.979 | INFO | app.auth.services.universal:create_scheduler_user:153 - Scheduler user already exists. [0] 2024-02-02 01:27:30.014 | INFO | main:init_db:122 - Starting scheduler [0] INFO: Application startup complete. [0] INFO: Uvicorn running on http://208.87.135.101:5000 (Press CTRL+C to quit) [0] INFO: 1.146.101.197:2309 - "OPTIONS /graylog/metrics HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2311 - "OPTIONS /wazuh_indexer/health HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2310 - "OPTIONS /influxdb/alerts HTTP/1.1" 200 OK [0] INFO: 1.146.101.197:2309 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2311 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2310 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized [1] @.***/compiler-sfc] This project is using reactive props destructure, which is an experimental feature. It may receive breaking changes or be removed in the future, so use at your own risk. [1] To stay updated, follow the RFC at https://github.com/vuejs/rfcs/discussions/502. [1] [0] INFO: 1.146.101.197:2310 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2312 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2312 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2315 - "GET /influxdb/alerts HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2314 - "GET /wazuh_indexer/health HTTP/1.1" 401 Unauthorized [0] INFO: 1.146.101.197:2316 - "GET /graylog/metrics HTTP/1.1" 401 Unauthorized ^C ➜ CoPilot git:(main) ✗


From: David Cartledge @.> Sent: Friday, February 2, 2024 11:30 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

fresh rebuild of CoPilot without ports on compose file


From: David Cartledge @.> Sent: Friday, February 2, 2024 11:23 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

sorry i didnt realise the full log wasn't downloaded here it is


From: David Cartledge @.> Sent: Friday, February 2, 2024 10:48 AM To: socfortress/CoPilot @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Thanks Taylor

I got around it by switching to another Docker server on another machine. The system loads up fine now. but when I go to login, it gives me an error at the top in red An error occured. please try again later. see image

The docker package loads on the web browser. there are a few errors in the docker container log but i can't make sense of them attached as well There were no errors when the docker compose file ran after I removed the ports from docker-compose It was conflicting with an error that said either remove the environment "hosts" network or remove ports.

I can not get it to run up as per your docker compose file if I leave it alone


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:05 AM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Hey there,

Try increasing your number of file watchers on the system.

nano /etc/sysctl.conf

add the following at the end of the file

fs.inotify.max_user_watches=524288

reload system settings

sysctl -p

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1921405095, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLCT43F36DYIVIGHZODYROOKTAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGQYDKMBZGU. You are receiving this because you authored the thread.Message ID: @.***>

taylorwalton commented 7 months ago

Is your VITE_API_URL set to http://208.87.135.101:5000 as well?

thisledome5 commented 7 months ago

Yes both set to same IP

Sent from Outlook for iOShttps://aka.ms/o0ukef


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:02:29 PM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Is your VITE_API_URL set to http://208.87.135.101:5000 as well?

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1922653274, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLDSK5FOVKS2H2O4NTDYRRCLLAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRSGY2TGMRXGQ. You are receiving this because you authored the thread.Message ID: @.***>

thisledome5 commented 7 months ago

here is the env file


From: David Cartledge @.> Sent: Friday, February 2, 2024 2:04 PM To: socfortress/CoPilot @.>; socfortress/CoPilot @.> Cc: Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Yes both set to same IP

Sent from Outlook for iOShttps://aka.ms/o0ukef


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:02:29 PM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Is your VITE_API_URL set to http://208.87.135.101:5000 as well?

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1922653274, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLDSK5FOVKS2H2O4NTDYRRCLLAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRSGY2TGMRXGQ. You are receiving this because you authored the thread.Message ID: @.***>

thisledome5 commented 7 months ago

this is what happens on your unmodified compose file. network and ports ERROR: for copilot_app_1 "host" network_mode is incompatible with port_bindings

ERROR: for app "host" network_mode is incompatible with port_bindings Traceback (most recent call last): File "/usr/bin/docker-compose", line 33, in sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')()) File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 81, in main command_func() File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 203, in perform_command handler(command, command_options) File "/usr/lib/python3/dist-packages/compose/metrics/decorator.py", line 18, in wrapper result = fn(*args, *kwargs) File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1186, in up to_attach = up(False) File "/usr/lib/python3/dist-packages/compose/cli/main.py", line 1166, in up return self.project.up( File "/usr/lib/python3/dist-packages/compose/project.py", line 697, in up results, errors = parallel.parallel_execute( File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute raise error_to_reraise File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer result = func(obj) File "/usr/lib/python3/dist-packages/compose/project.py", line 679, in do return service.execute_convergence_plan( File "/usr/lib/python3/dist-packages/compose/service.py", line 559, in execute_convergence_plan return self._execute_convergence_create( File "/usr/lib/python3/dist-packages/compose/service.py", line 473, in _execute_convergence_create containers, errors = parallel_execute( File "/usr/lib/python3/dist-packages/compose/parallel.py", line 108, in parallel_execute raise error_to_reraise File "/usr/lib/python3/dist-packages/compose/parallel.py", line 206, in producer result = func(obj) File "/usr/lib/python3/dist-packages/compose/service.py", line 478, in lambda service_name: create_and_start(self, service_name.number), File "/usr/lib/python3/dist-packages/compose/service.py", line 457, in create_and_start container = service.create_container(number=n, quiet=True) File "/usr/lib/python3/dist-packages/compose/service.py", line 330, in create_container container_options = self._get_container_create_options( File "/usr/lib/python3/dist-packages/compose/service.py", line 939, in _get_container_create_options container_options['host_config'] = self._get_container_host_config( File "/usr/lib/python3/dist-packages/compose/service.py", line 1014, in _get_container_host_config return self.client.create_host_config( File "/usr/lib/python3/dist-packages/docker/api/container.py", line 596, in create_host_config return HostConfig(args, **kwargs) File "/usr/lib/python3/dist-packages/docker/types/containers.py", line 336, in init raise host_config_incompatible_error( docker.errors.InvalidArgument: "host" network_mode is incompatible with port_bindings


From: taylor_socfortress @.> Sent: Friday, February 2, 2024 12:02 PM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Author @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

Is your VITE_API_URL set to http://208.87.135.101:5000 as well?

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1922653274, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLDSK5FOVKS2H2O4NTDYRRCLLAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRSGY2TGMRXGQ. You are receiving this because you authored the thread.Message ID: @.***>

thisledome5 commented 7 months ago

ok Taylor . after much browsing, I have now come to the conclusion that it is working fine. I have the same issue as z00mz00m95 . seems the traffic is not getting through on 5000 but I have no firewall setup on this IP. I get the same messages in developer tools as he does.

POST http://10.120.193.5:5000/auth/token net::ERR_CONNECTION_REFUSED

I set it back up on an internal server. But login obviously fails. if no firewall,where is 5000 being stopped???

Also If I remove the network hosts line, The Docker container installs correctly

thisledome5 commented 7 months ago

prob not a firewall rule as i jjust found this in the log on startup

[Errno 99] error while attempting to bind on address ('10.120.193.5', 5000): cannot assign requested address

z00mz00m95 commented 7 months ago

ok Taylor . after much browsing, I have now come to the conclusion that it is working fine. I have the same issue as z00mz00m95 . seems the traffic is not getting through on 5000 but I have no firewall setup on this IP. I get the same messages in developer tools as he does.

POST http://10.120.193.5:5000/auth/token net::ERR_CONNECTION_REFUSED

Hey thisledome5, in my case it was the network firewall of my hoster, who i have setup the machine to quickly test the project. Maybe you have to set some custom ports for this? Greetings!

thisledome5 commented 7 months ago

Hi z00mz00m Ok I have already set custom ports. I get exactly the same error Connection_refused. I have other ports in the range I am now using working on that server so I don't understand what the issue could be

thisledome5 commented 7 months ago

It does the same thing if I move the ports down to 81 an 82 nothing seems to work.

taylorwalton commented 7 months ago

What version of docker are you running? does running the netstat -ltpnd command show the host listening on the correct ports?

thisledome5 commented 7 months ago

here is the netstat result Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:8080 0.0.0.0: LISTEN 1924/crowdsec tcp 0 0 0.0.0.0:9993 0.0.0.0: LISTEN 804/zerotier-one tcp 0 0 127.0.0.1:6060 0.0.0.0: LISTEN 1924/crowdsec tcp 0 0 127.0.0.1:6010 0.0.0.0: LISTEN 282771/sshd: root@p tcp 0 0 0.0.0.0:9001 0.0.0.0: LISTEN 1200/docker-proxy tcp 0 0 0.0.0.0:80 0.0.0.0: LISTEN 732/nginx: master p tcp 0 0 127.0.0.1:27017 0.0.0.0: LISTEN 783/mongod tcp 0 0 127.0.0.1:35623 0.0.0.0: LISTEN 699/urbackupclientb tcp 0 0 10.120.193.5:22 0.0.0.0: LISTEN 4109748/sshd: /usr/ tcp 0 0 127.0.0.1:42077 0.0.0.0: LISTEN 722/containerd tcp 0 0 0.0.0.0:5173 0.0.0.0: LISTEN 286849/docker-proxy tcp 0 0 127.0.0.53:53 0.0.0.0: LISTEN 513/systemd-resolve tcp 0 0 0.0.0.0:5000 0.0.0.0: LISTEN 286872/docker-proxy tcp6 0 0 ::1:35623 ::: LISTEN 699/urbackupclientb tcp6 0 0 :::9993 ::: LISTEN 804/zerotier-one tcp6 0 0 :::9001 ::: LISTEN 1206/docker-proxy tcp6 0 0 :::9090 ::: LISTEN 1/init tcp6 0 0 :::80 ::: LISTEN 732/nginx: master p tcp6 0 0 ::1:6010 ::: LISTEN 282771/sshd: root@p tcp6 0 0 :::5173 ::: LISTEN 286856/docker-proxy tcp6 0 0 :::5000 :::* LISTEN 286879/docker-proxy

and docker version Client: Version: 24.0.5 API version: 1.43 Go version: go1.20.3 Git commit: 24.0.5-0ubuntu1~22.04.1 Built: Mon Aug 21 19:50:14 2023 OS/Arch: linux/amd64 Context: default

Server: Engine: Version: 24.0.5 API version: 1.43 (minimum version 1.12) Go version: go1.20.3 Git commit: 24.0.5-0ubuntu1~22.04.1 Built: Mon Aug 21 19:50:14 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.7.2 GitCommit: runc: Version: 1.1.7-0ubuntu1~22.04.2 GitCommit: docker-init: Version: 0.19.0 GitCommit:

thisledome5 commented 7 months ago

and this is the log from a fresh install

[0] 2024-02-02 23:29:15.189 | INFO | main:init_db:122 - Starting scheduler [0] INFO: Application startup complete. [0] ERROR: [Errno 99] error while attempting to bind on address ('10.120.193.5', 5000): cannot assign requested address [0] INFO: Waiting for application shutdown. [0] 2024-02-02 23:29:15.196 | INFO | main:shutdown_scheduler:134 - Shutting down scheduler [0] 2024-02-02 23:29:15.209 | INFO | app.db.db_setup:ensure_scheduler_user_removed:114 - Ensuring scheduler user exists [0] 2024-02-02 23:29:15.235 | INFO | app.auth.services.universal:remove_scheduler_user:177 - Scheduler user removed. [0] INFO: Application shutdown complete. [0] npm run start-server exited with code 1 [1] [@vue/compiler-sfc] This project is using reactive props destructure, which is an experimental feature. It may receive breaking changes or be removed in the future, so use at your own risk. [1] To stay updated, follow the RFC at https://github.com/vuejs/rfcs/discussions/502.

thisledome5 commented 7 months ago

k so after much brain strain and many ups and then deletes, I found that in the compose file, the network_mode: "host" does not work and has been the issue with all these errors. I found that correcting it to Network: "hosts" allowed the docker-compose file to run without errors. yay fixed it myself.

however I am now presented with a different error when logging in. Well the same message at top of screen Please try again later etc.

The development console screen now shows a different message POST http://10.120.193.5:5173/auth/register 404 (Not Found)

whether I login with my new password or try and create a new user. So 1 step forward and 1 step back lol

Any idea on this new one. Cant seem to find something 404

I really want this to work. I love the approach with opensource

thisledome5 commented 7 months ago

and this on login attempt with new passsword

auth.ts:9 POST http://10.120.193.5:5173/auth/token 404 (Not Found)

taylorwalton commented 7 months ago

tbh im having a hard time duplicating your issue and if i modify from host to hosts, the container fails to start...do you have any selinux or localfirewall running on the VM your testing on?

geantrevisan commented 7 months ago

I fix this change on backend /copilot.py line 56 server_ip = os.getenv("SERVER_IP", "localhost") for server_ip = os.getenv("0.0.0.0")

thisledome5 commented 7 months ago

Hi @taylorwalton .I am running no firewall or selinux on my ubuntu 22:04 Jammy version. It runs on a VM in SSDNodes systems. They block no ports at all There are other docker containers running on that samemachine, all working perfectly. I guess this is part of testing

@geantrevisan I did your suggested edit of the .py file. and now the container starts cleaning. Uvicorn is running on 5000 which I have not yet ever had run.

But the same error message when attempting to login. I restarted the container to be sure it loaded correctly 🚀 ~ ENV: prod auth.ts:9 POST http://10.120.193.5:5173/auth/token 404 (Not Found)

thisledome5 commented 7 months ago

There are no errorsin the log file. apart from all the connections to each external source like wazuh etc, to be expected as I have not been able to login and set those parameters yet. but the error at top of page happens immediately when you press login, the developer console tab spits out 🚀 ~ ENV: prod auth.ts:9
POST http://10.120.193.5:5173/auth/token 404 (Not Found)

But nothing errors in the log file at all not even the failed attempt at login

thisledome5 commented 7 months ago

@taylorwalton sorry for confusion Taylor. My compose file is now set back to the original settings for network_mode: "host"

Now that The .py file has changed to 0.0.0.0 the container starts correctly. but still same error at top of page

geantrevisan commented 7 months ago

@thisledome5 edit file docker-compose.yml line 7 comment this network_mode: "host" image and build you image Exemple: docker build -f Dockerfile.deb -t soc . change line 4 -> image: ghcr.io/socfortress/copilot:latest for image: soc

and work :D

thisledome5 commented 7 months ago

Hi @geantrevisan I am a bit lost on the build image part. I have commented the line in docker-comose.yml.

but the steps after that don't make a lot od sense for me. This would be the first time doing anything like that

geantrevisan commented 7 months ago

@thisledome5 why when we modify Arequipa copilot.py you need to generate a new build, because you are using the unmodified build that would be Taylor's.

thisledome5 commented 7 months ago

Hi @geantrevisan

I have to admit . I have never done a build or how to do it. I understand the need to redo the build. but am a noob to doing it. Could you share with me exactly what I need to do in steps please. sorry for being a pain.

thisledome5 commented 7 months ago

ok so here is the absolutely weird thing. I once again removed the container and image commented hosts line in the compose file and ran up the container. had a failure in the log. Wouldnt attach the port 5000 to the ip So went back in and put the host line back in. removed the container and image. and ranup the container again and presto it works. I can log in. now I did nothing different from the previous 30-40 times I did this.

Very odd. It now works exactly from the Taylor build. All I changed was the IP in the .env and . well who knows. Mystery to me. now I have an issue trying to add Graylog. keeps failing

taylorwalton commented 7 months ago

closing this issue out since working now, please open a new issue if Graylog is not working for you... make sure that copilot is in the same network as graylog and there are no firewalls blocking copilot from reaching graylog api

thisledome5 commented 7 months ago

many thanks. And yes for no obvious reason it works. Hopeful it will stay that way.

Keep up the good work


From: taylor_socfortress @.> Sent: Wednesday, February 7, 2024 11:35 PM To: socfortress/CoPilot @.> Cc: thisledome5 @.>; Mention @.> Subject: Re: [socfortress/CoPilot] error on startup - log file (Issue #123)

closing this issue out since working now, please open a new issue if Graylog is not working for you... make sure that copilot is in the same network as graylog and there are no firewalls blocking copilot from reaching graylog api

— Reply to this email directly, view it on GitHubhttps://github.com/socfortress/CoPilot/issues/123#issuecomment-1932065500, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AK5LYLF2LXPWEPPMYS6MAB3YSN7KTAVCNFSM6AAAAABCUEUHCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZSGA3DKNJQGA. You are receiving this because you were mentioned.Message ID: @.***>