detecting state - Githubissues

konne commented 2 years ago

Bug description

like you see in this picture this service runs perfectly and can be accessed with the browser, but it is marked with a detecting state.

Steps to reproduce

I don't know if you can reproduce it.

We have this situation that the port detection works perfect for node services that are running in docker containers, but not for the pgadmin that is also running inside a docker container.

Workspace affected

workspace: apricot-scorpion-yeo25j03

Expected behavior

No response

Example repository

No response

Anything else?

No response

mahmoudajawad commented 2 years ago

This issue does exactly occur to me now randomly. I'm working around this by stopping the workspace and starting it again.

akosyakov commented 2 years ago

Just to clarify: it happens only when you run docker in Gitpod?

konne commented 2 years ago

@akosyakov for me, we run everything inside docker, so I can confirm that it happened with something that runs inside docker, but no other assumptions around that

akosyakov commented 2 years ago

@csweichel I looked at supervisor logs when it happens and https://github.com/gitpod-io/gitpod/blob/bb1349fd9489661007bcd92cbff91438212e1e86/components/supervisor/pkg/ports/ports.go#L486-L495 is never executed for such port. The last log is auto-exposing port. There was not changes in exposed ports service besides removing global port, so it seems it blocks on control port request to server. I am not sure. Can we see traces?

csweichel commented 2 years ago

I am not sure. Can we see traces?

Unfortunately we don't trace this interface 🤦

That said, auto-exposure is only neccesary in the "public exposure" case since we've moved away from Kubernetes services. Hence we could make this part be a bit more "tolerant" towards auto-exposure failure.

akosyakov commented 2 years ago

It actually happened to me very often today and it seems first hanging port blocks all consequent:

akosyakov commented 2 years ago

Hence we could make this part be a bit more "tolerant" towards auto-exposure failure.

I am not sure what do you mean? It does not fail, it hangs forever.

corneliusludmann commented 2 years ago

/assign

corneliusludmann commented 2 years ago

@akosyakov Do you have a workspace ID or a link to the logs for your case?

I had a look in the logs of the workspace apricot-scorpion-yeo25j03 of the issue opener and I cannot find the issue that you observed there (missing auto-exposed port log). For the port that is marked as detecting in the screenshot (3980) there is a auto-exposed port message (log entry).

akosyakov commented 2 years ago

@akosyakov Do you have a workspace ID or a link to the logs for your case?

I actually was able to reproduce it with new workspaces today. I think the workspace id was ivory-swift-u1uxar3a and port 24678. It happened like 4-6 hours ago.

It was opened for this context: https://github.com/trusktr/solid-lume-starter/commit/35c342675ca82244027be2f309edea615ba4c051 and start a server with npm i && npm run dev command.

corneliusludmann commented 2 years ago

Here is an update of my findings so far:

It's pretty hard to reproduce. I get this in production from time to time but never in a preview env on core-dev. That makes it hard to debug.
I don't see the behavior @akosyakov described above. I always see both logs auto-exposing port and auto-exposed port, even in the workspaces of Anton. Either I missed something or Anton skipped the entry by mistake.
My current hypothesis is that it is related to the server memory leak. Which makes the server not reliably accessible in prod from time to time. However, I don't fully understand the role of the server in this process (yet).
If we want to improve the situation from our side I think we should address @csweichel's comment to be a bit more “tolerant” towards auto-exposure failure. However, haven't found an actual error in the logs yet and I currently don't see the proper leverage right now to address this. Could someone give me some pointers to that?

akosyakov commented 2 years ago

I don't see the behavior @akosyakov described above. I always see both logs auto-exposing port and auto-exposed port, even in the workspaces of Anton. Either I missed something or Anton skipped the entry by mistake.

I think you are right, I overlooked something by mistake. I could not see them hanging right now. Actually all returns successfully.

akosyakov commented 2 years ago

@iQQBot you can follow internal investigation here: https://gitpod.slack.com/archives/C01KGM9BH54/p1638873457412200

geropl commented 2 years ago

Relevant update: https://github.com/gitpod-io/gitpod/issues/7054#issuecomment-988777393

akosyakov commented 2 years ago

@iQQBot supervisor reconnects to the server each 10 mins, during such reconnect we can miss some events, so let's add a mitigation which sync with server on reconnect.

Do you need some code pointers?

iQQBot commented 2 years ago

Why reconnects to server each 10min?

marcoaleixo commented 2 years ago

I'm having this "detecting..." problem. By accident I discovered that if we run "gp forward-port any_port" Gitpod automatically is able to detect the state of the ports. I'm posting this if anyone in future have the same bug until the team doesn't fix it.

akosyakov commented 2 years ago

By accident I discovered that if we run "gp forward-port any_port" Gitpod automatically is able to detect the state of the ports.

Yeah, it sounds exactly like a problem of missing events. If you forward another port a new event is issued which delivers proper state for first port too.

akosyakov commented 2 years ago

should be fixed by https://github.com/gitpod-io/gitpod/pull/7621

olleyt commented 1 year ago

@akosyakov the issue is reproducing again after my account was migrated to the new Gitpod version. Port 300 is not opening when I run a Docker container on it.

Could this issue be reproduced in the new release?