search

azukaar / Cosmos-Server

☁️ The Most Secure and Easy Selfhosted Home Server. Take control of your data and privacy without sacrificing security and stability (Authentication, anti-DDOS, anti-bot)
https://cosmos-cloud.io
Other
2.99k stars 108 forks source link

[BUG]: Cosmos-cloud inaccessible after codeeserver installation #139

Closed dalso0418 closed 8 months ago

dalso0418 commented 8 months ago

What happened?

Hello Cosmos-cloud is inaccessible after codeserver installation.

What should have happened?

If it is installed normally, the code server should also be accessible The codeserver and cosmos cloud are also inaccessible. (Even if the code server installation fails, I think the cosmos cloud should be accessed.)

How to reproduce the bug?

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

The codeserver was being installed with the following configuration and was not accessible after the installation was completed.

image

Relevant log output

cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Using config file: /config/cosmos.config.json
cosmos-server    | 2023/11/20 02:14:41 [INFO] Validating config file...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Docker Connected
cosmos-server    | 2023/11/20 02:14:41 [INFO] Bootstrap Container From Tags: 8ae27742adc2d41a587592179b361d8a83104a99bda726ed1a67f61c4ac0022a
cosmos-server    | 2023/11/20 02:14:41 [INFO] Done bootstrapping Container From Tags: /cosmos-server
cosmos-server    | 2023/11/20 02:14:41 [INFO] Bootstrap Container From Tags: c23f0f2e294e0ada5c30257e3b543c2b9bd9f6fccaaf2fb095e6894d4e3f4ad9
cosmos-server    | 2023/11/20 02:14:41 [INFO] /Code-Server: Checking Force network secured
cosmos-server    | 2023/11/20 02:14:41 [INFO] Done bootstrapping Container From Tags: /Code-Server
cosmos-server    | 2023/11/20 02:14:41 [INFO] Bootstrap Container From Tags: f2a9ebece18803d6a8241445da14e8843a5278f80127b49200e7dbc2f405bc84
cosmos-server    | 2023/11/20 02:14:41 [INFO] /cosmos-mongo-LFn: Checking Force network secured
cosmos-server    | 2023/11/20 02:14:41 [INFO] Done bootstrapping Container From Tags: /cosmos-mongo-LFn
cosmos-server    | 2023/11/20 02:14:41 [INFO] Checking for self updater agent
cosmos-server    | 2023/11/20 02:14:41 [INFO] Found. Removing self updater agent
cosmos-server    | 2023/11/20 02:14:41 [ERROR] RemoveSelfUpdater : Error response from daemon: Cannot kill container: 1fab7ef732a19ff21d02db69860f189533d983bb5a23991a0404d4eca778b932: Container 1fab7ef732a19ff21d02db69860f189533d983bb5a23991a0404d4eca778b932 is not running
cosmos-server    | 2023/11/20 02:14:41 [INFO] Docker API version: 1.43
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting monitoring services...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting market services...
cosmos-server    | 2023/11/20 02:14:41 [INFO] MarketInit: Added market cosmos-cloud
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting OpenID services...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Using config file: /config/cosmos.config.json
cosmos-server    | 2023/11/20 02:14:41 [INFO] Metrics - Monitoring the server at /mnt/host
cosmos-server    | 2023/11/20 02:14:41 [INFO] Metrics: Agglomeration Initialized
cosmos-server    | 2023/11/20 02:14:41 [INFO] OpenID server initialized
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting constellation services...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting server...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Initialising HTTP(S) Router and all routes
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting DNS server on :53
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting in /app
cosmos-server    | 2023/11/20 02:14:41 [INFO] Added route: [SERVAPP] 192.168.1.9:7200 to http://Code-Server:8443
cosmos-server    | 2023/11/20 02:14:41 [INFO] TLS certificate exist, starting HTTPS servers and redirecting HTTP to HTTPS
cosmos-server    | 2023/11/20 02:14:41 [INFO] Listening to HTTP on :80
cosmos-server    | 2023/11/20 02:14:41 [INFO] Listening to HTTPS on :443
cosmos-server    | 2023/11/20 02:14:41 [INFO] Setup: Checking Docker port mapping
cosmos-server    | 2023/11/20 02:14:41 [INFO] Port mapping changed. Needs update.
cosmos-server    | 2023/11/20 02:14:41 [INFO] New ports: 7200:443
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Starting...
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Container name: 8ae27742adc2
cosmos-server    | 2023/11/20 02:14:41 [INFO] Checking for self updater agent
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Creating updater service
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting creation of new service...
cosmos-server    | 2023/11/20 02:14:41 [INFO] Using config file: /config/cosmos.config.json
cosmos-server    | 2023/11/20 02:14:41 [INFO] Pulling image azukaar/docker-self-updater:latest
cosmos-server    | 2023/11/20 02:14:42 [INFO] Metrics: Agglomeration of metrics
cosmos-server    | 2023/11/20 02:14:42 [INFO] (Re) Connecting to the database...
cosmos-server    | 2023/11/20 02:14:42 [INFO] Successfully connected to the database.
cosmos-server    | 2023/11/20 02:14:42 [INFO] Metrics: Agglomeration done. Saving to DB
cosmos-server    | 2023/11/20 02:14:42 [INFO] Metrics: Agglomeration saved to DB
cosmos-server    | 2023/11/20 02:14:43 [INFO] Image azukaar/docker-self-updater:latest pulled
cosmos-server    | 2023/11/20 02:14:43 [INFO] Checking service cosmos-self-updater-agent...
cosmos-server    | 2023/11/20 02:14:43 [INFO] Creating container cosmos-self-updater-agent...
cosmos-server    | 2023/11/20 02:14:43 [INFO] Checking directory /mnt/host/var/run/docker.sock for bind mount
cosmos-server    | 2023/11/20 02:14:43 [INFO] Container cosmos-self-updater-agent created
cosmos-server    | 2023/11/20 02:14:43 [INFO] Container cosmos-self-updater-agent initiated
cosmos-server    | 2023/11/20 02:14:43 [INFO] Config file saved.
cosmos-server    | 2023/11/20 02:14:43 [INFO] CreateService: Operation succeeded. SERVICE STARTED
cosmos-server    | 2023/11/20 02:14:43 [INFO] Now listening to HTTPS on :443
cosmos-server    | 2023/11/20 02:14:43 [INFO] Bootstrap Container From Tags: 989f540a66582b96092775619da235c7cdef0076377b3117d8e94b7602232ff6
cosmos-server    | 2023/11/20 02:14:43 [INFO] Done bootstrapping Container From Tags: /cosmos-self-updater-agent
cosmos-server exited with code 137

Other details

No response

System details

azukaar commented 8 months ago
dalso0418 commented 8 months ago

Neither the portainer nor the unraid stack is used.

Cosmos executed as docker-compose.

It is currently unavailable at docker-compose restart with the default cosmos-cloud not running normally.

azukaar commented 8 months ago
dalso0418 commented 8 months ago

It is not accessed through a VPN. The internal network was accessing via https://ip and will not connect when the container is restarted.

image

image

azukaar commented 8 months ago

did you change anything in the code-server installation? I see it exposes the port 8443 but if installed from the market it shouldn't, on the other hand Cosmos exposes 80 / 443 and is missing the port 7200 that it exposes, see from the logs:

cosmos-server    | 2023/11/20 02:14:41 [INFO] Setup: Checking Docker port mapping
cosmos-server    | 2023/11/20 02:14:41 [INFO] Port mapping changed. Needs update.
cosmos-server    | 2023/11/20 02:14:41 [INFO] New ports: 7200:443
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Starting...
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Container name: 8ae27742adc2
cosmos-server    | 2023/11/20 02:14:41 [INFO] Checking for self updater agent
cosmos-server    | 2023/11/20 02:14:41 [INFO] SelUpdatePorts - Creating updater service
cosmos-server    | 2023/11/20 02:14:41 [INFO] Starting creation of new service...
dalso0418 commented 8 months ago

You changed the volume mapping location and timezone without changing the port. image

Is there any way to restore it?

*Please understand if there is a mistranslation due to the use of a translator.

azukaar commented 8 months ago

걱정 마 ;D so the env variable PROXY_DOMAIN tells VSCode that it is reachable under 192.168.1.9:7200 That does not mean the container listens on 7200. Basically what happens is this:

  1. You visit 192.168.1.9:7200 on your browser
  2. This reaches Cosmos, the reverse proxy, which sees that port 7200 should be VS Code
  3. Cosmos will transmit the request to VSCode on the port VSCode actually listens to
  4. When receiving the request, VSCode still need to know that the original request was to 192.168.1.9:7200 in order to accept it

So by default when you create an app when using IP address, Cosmos will automatically bind 7200 to itself so it can proxy request to the container. The container itself does not need to export ports at all

dalso0418 commented 8 months ago

The important thing is that cosmos management web is also not accessible.

I stopped codeserver through docker stop, but I can't access the admin page

azukaar commented 8 months ago

The important thing is that cosmos management web is also not accessible.

Can you check in incognito mode? might be your cached certificate that cause this, I dont seem to see an error in your logs that would justify this, and if the container is started and the port is opened there's no reason you owuldnt be able toaccess the ui

dalso0418 commented 8 months ago

It does not connect to the web even in secret mode.

image

Logs after re-running docker-compose restart.

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 141f0171bac8 azukaar/docker-self-updater:latest "./docker-self-updat…" 5 seconds ago Up 4 seconds cosmos-self-updater-agent f41f7d2f747c azukaar/cosmos-server:latest "sh -c './$(cat /bin…" 7 seconds ago Up 7 seconds 80/tcp, 443/tcp cosmos-server f2a9ebece188 mongo:4.4 "docker-entrypoint.s…" 30 hours ago Up 30 hours 27017/tcp cosmos-mongo-LFn [dalso][01:15:09][root@ds-proxy-caddy /data/cosmos]

docker-compose logs -f

Attaching to cosmos-server cosmos-server | 2023/11/20 16:15:02 [INFO] Starting... cosmos-server | 2023/11/20 16:15:02 [INFO] Using config file: /config/cosmos.config.json cosmos-server | 2023/11/20 16:15:02 [INFO] Validating config file... cosmos-server | 2023/11/20 16:15:02 [INFO] Docker Connected cosmos-server | 2023/11/20 16:15:02 [INFO] Bootstrap Container From Tags: f41f7d2f747c17994718bf26a93219e2410597f4e1adf1b9ad164dda639885b0 cosmos-server | 2023/11/20 16:15:02 [INFO] Done bootstrapping Container From Tags: /cosmos-server cosmos-server | 2023/11/20 16:15:02 [INFO] Bootstrap Container From Tags: f2a9ebece18803d6a8241445da14e8843a5278f80127b49200e7dbc2f405bc84 cosmos-server | 2023/11/20 16:15:02 [INFO] /cosmos-mongo-LFn: Checking Force network secured cosmos-server | 2023/11/20 16:15:02 [INFO] Done bootstrapping Container From Tags: /cosmos-mongo-LFn cosmos-server | 2023/11/20 16:15:02 [INFO] Checking for self updater agent cosmos-server | 2023/11/20 16:15:02 [INFO] Found. Removing self updater agent cosmos-server | 2023/11/20 16:15:02 [ERROR] RemoveSelfUpdater : Error response from daemon: Cannot kill container: e749a6a133ee49566ab3dd587f49154c98d8be3cb9f39aaa1575a21538f1958d: Container e749a6a133ee49566ab3dd587f49154c98d8be3cb9f39aaa1575a21538f1958d is not running cosmos-server | 2023/11/20 16:15:02 [INFO] Docker API version: 1.43 cosmos-server | 2023/11/20 16:15:02 [INFO] Starting monitoring services... cosmos-server | 2023/11/20 16:15:02 [INFO] Starting market services... cosmos-server | 2023/11/20 16:15:02 [INFO] MarketInit: Added market cosmos-cloud cosmos-server | 2023/11/20 16:15:02 [INFO] Starting OpenID services... cosmos-server | 2023/11/20 16:15:02 [INFO] Using config file: /config/cosmos.config.json cosmos-server | 2023/11/20 16:15:02 [INFO] Metrics: Agglomeration Initialized cosmos-server | 2023/11/20 16:15:02 [INFO] Metrics - Monitoring the server at /mnt/host cosmos-server | 2023/11/20 16:15:02 [INFO] OpenID server initialized cosmos-server | 2023/11/20 16:15:02 [INFO] Starting constellation services... cosmos-server | 2023/11/20 16:15:02 [INFO] Starting server... cosmos-server | 2023/11/20 16:15:02 [INFO] Initialising HTTP(S) Router and all routes cosmos-server | 2023/11/20 16:15:02 [INFO] Starting DNS server on :53 cosmos-server | 2023/11/20 16:15:02 [INFO] Starting in /app cosmos-server | 2023/11/20 16:15:02 [INFO] Added route: [SERVAPP] 192.168.1.9:7200 to http://Code-Server:8443 cosmos-server | 2023/11/20 16:15:02 [INFO] TLS certificate exist, starting HTTPS servers and redirecting HTTP to HTTPS cosmos-server | 2023/11/20 16:15:02 [INFO] Listening to HTTP on :80 cosmos-server | 2023/11/20 16:15:02 [INFO] Listening to HTTPS on :443 cosmos-server | 2023/11/20 16:15:02 [INFO] Setup: Checking Docker port mapping cosmos-server | 2023/11/20 16:15:02 [INFO] Port mapping changed. Needs update. cosmos-server | 2023/11/20 16:15:02 [INFO] New ports: 7200:443 cosmos-server | 2023/11/20 16:15:02 [INFO] SelUpdatePorts - Starting... cosmos-server | 2023/11/20 16:15:02 [INFO] SelUpdatePorts - Container name: f41f7d2f747c cosmos-server | 2023/11/20 16:15:02 [INFO] Checking for self updater agent cosmos-server | 2023/11/20 16:15:02 [INFO] SelUpdatePorts - Creating updater service cosmos-server | 2023/11/20 16:15:02 [INFO] Starting creation of new service... cosmos-server | 2023/11/20 16:15:02 [INFO] Using config file: /config/cosmos.config.json cosmos-server | 2023/11/20 16:15:02 [INFO] Pulling image azukaar/docker-self-updater:latest cosmos-server | 2023/11/20 16:15:03 [INFO] Metrics: Agglomeration of metrics cosmos-server | 2023/11/20 16:15:03 [INFO] (Re) Connecting to the database... cosmos-server | 2023/11/20 16:15:03 [INFO] Successfully connected to the database. cosmos-server | 2023/11/20 16:15:03 [INFO] Metrics: Agglomeration done. Saving to DB cosmos-server | 2023/11/20 16:15:03 [INFO] Metrics: Agglomeration saved to DB cosmos-server | 2023/11/20 16:15:04 [INFO] Image azukaar/docker-self-updater:latest pulled cosmos-server | 2023/11/20 16:15:04 [INFO] Checking service cosmos-self-updater-agent... cosmos-server | 2023/11/20 16:15:04 [INFO] Creating container cosmos-self-updater-agent... cosmos-server | 2023/11/20 16:15:04 [INFO] Checking directory /mnt/host/var/run/docker.sock for bind mount cosmos-server | 2023/11/20 16:15:04 [INFO] Container cosmos-self-updater-agent created cosmos-server | 2023/11/20 16:15:05 [INFO] Container cosmos-self-updater-agent initiated cosmos-server | 2023/11/20 16:15:05 [INFO] Config file saved. cosmos-server | 2023/11/20 16:15:05 [INFO] CreateService: Operation succeeded. SERVICE STARTED cosmos-server | 2023/11/20 16:15:05 [INFO] Now listening to HTTPS on :443 cosmos-server | 2023/11/20 16:15:05 [INFO] Bootstrap Container From Tags: 141f0171bac878c5b8460f9124154b7625ce3cf09e150f334cc086bdbcef340c cosmos-server | 2023/11/20 16:15:05 [INFO] Done bootstrapping Container From Tags: /cosmos-self-updater-agent cosmos-server exited with code 137

azukaar commented 8 months ago

AHHH ok that is interesting!

When a container exits with code 137, it typically means that the container was forcibly killed due to exceeding its allocated memory limits. In the context of Docker, this often happens when a process within the container uses more memory than is allocated to it, leading the container management system to terminate it.

The number 137 is significant because it's usually interpreted as follows:

So, an exit code of 137 typically indicates that the container received a SIGKILL signal because it ran out of memory. To resolve this, you might need to allocate more memory to the container, or check your system available RAM

dalso0418 commented 8 months ago

The container is on and the memory is relaxed...ㅠㅠㅠ

[dalso][22:15:17][root@ds-proxy-caddy /home/admin-sv]

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3b5711427fe3 azukaar/docker-self-updater:latest "./docker-self-updat…" 1 second ago Up Less than a second cosmos-self-updater-agent 3fce29586a7c azukaar/cosmos-server:latest "sh -c './$(cat /bin…" 4 seconds ago Up 3 seconds 80/tcp, 443/tcp cosmos-server f2a9ebece188 mongo:4.4 "docker-entrypoint.s…" 2 days ago Up 2 days 27017/tcp cosmos-mongo-LFn [dalso][22:15:19][root@ds-proxy-caddy /home/admin-sv]

free -h

           total        used        free      shared  buff/cache   available

Mem: 3.8Gi 414Mi 110Mi 0.0Ki 3.3Gi 3.1Gi Swap: 8.0Gi 61Mi 7.9Gi

azukaar commented 8 months ago

can you try to use docker stats?