Lando port scanning hangs when port is in use but is not accepting connections

[dgervalle]

My issue looks very similar to what has been previously discussed in lando/lando#979. It happens on very different projects, and therefore config and plugin. I am running lando v3.21.2 with all packages updated.

Without any verbosity, the issue looks like this:

$ lando start
Let's get this party started! Starting app myapp...

This stay hanging there (almost) forever, without giving any clue about what is the issue. With full verbosity, it's not much better, because it ends up hanging with:

...
lando https://127.0.0.1:4433 not currently accessible  +0ms
lando https://127.0.0.1:4444 not currently accessible  +1ms
lando https://127.0.0.1:4443 not currently accessible  +0ms
lando scan completed.  +0ms
lando scan results. url=https://127.0.0.1:443, status=false, color=red, url=https://127.0.0.1:444, status=false, color=red, url=https://127.0.0.1:4433, status=false, color=red, url=https://127.0.0.1:4444, status=false, color=red, url=https://127.0.0.1:4443, status=false, color=red +0ms

Still no clue, but thanks to lando/lando#979, I tried closing some running programs... nothing happened until I close VSCode, which releases the start with:

lando scan failed for http://127.0.0.1:8888 code=ECONNRESET, message=socket hang up +639696ms
lando No response for http://127.0.0.1:8888. Setting to bad  +2ms
lando http://127.0.0.1:8888 not currently accessible  +0ms
lando checking to see if http://127.0.0.1:8888 is ready.  +504ms
lando scan failed for http://127.0.0.1:8888 code=ECONNREFUSED, message=connect ECONNREFUSED 127.0.0.1:8888 +5ms
lando No response for http://127.0.0.1:8888. Setting to bad  +0ms
lando http://127.0.0.1:8888 not currently accessible  +0ms
lando http://127.0.0.1:8888 not currently accessible  +3ms
lando scan completed.  +0ms
lando scan results. url=http://127.0.0.1:80, status=true, color=green, url=http://127.0.0.1:8000, status=false, color=red, url=http://127.0.0.1:8080, status=false, color=red, url=http://127.0.0.1:8888, status=false, color=red, url=http://127.0.0.1:8008, status=false, color=red +0ms
lando emitting event engine-autostart  +12ms
lando event engine-autostart has 0 listeners  +1ms
lando engine is up.  +578ms
...

So, I conclude that the issue was lando waiting on a reply from port 8888, which it never received. I can confirm that VSCode is listening on that port and I can imagine that it did not reply to the connection request from lando. What is puzzling is that lando is not timeout quickly on those checks, waiting forever...

Hope you can find a way to fix this, because for now, if any program is listening in the same way on any of the following ports, lando could hang: 80, 443, 444, 4433, 4443, 4444, 8000, 8008, 8080, 8888

[AaronFeledy]

If closing VSCode fixes the issue, it's usually because of a debugger like xdebug freezing the process so that you can step through it. Lando is waiting for your app to respond, but it won't because the debugger is waiting for you to interact with it.

[dgervalle]

@AaronFeledy I am sorry, but this is not related, I don't see how my app could be concerned by the initialization of Lando. From the log above, you can see that it hangs during the port scanning sequence, which has nothing to do with my app at all. The fact that it is VSCode that is listening to 8888 (and it happened to me previously on 8080) is just the cause of the connection failure, but nothing else. Moreover, I could not unconsciously start a debugging session in VSCode. I am still investigating why VSCode is listening on 8888 (and previously 8080), but anyway, I don't see a reason for Lando port scanning to be hung by anything, since, if I get it properly, this scanning is used to decide if a port is available, and if for any reason, you stay stuck on a port, it means it's not, and you should just cancel and pursue.

[dgervalle]

@AaronFeledy FYI, I found out what is behind 8888 on my local... It's an auto-forwarded port made by VSCode from a remote on which I have a unrelated project running, and behind that forward, it's an instance of haproxy... So, as you can see, no debugger or anything like that.

[AaronFeledy]

Thank you for the details. I think there is a bug here but it's probably an edge case.

For some background, the purpose of Lando's port scanner is to work around conflicting web servers running on your machine (WAMP, MAMP, DDEV, etc). So, it's sending web requests to localhost on 80, 8000, 8080, 8888, 8008 to find suitable ports that are not in use. The individual requests will immediately fail when nothing is listening on a port, so Lando knows it's available. If a web server accepts the connection, which usually happens quickly, Lando knows the port is in use. If all the ports are free, this process takes milliseconds, if some are in use, it usually takes only moments longer for the round trip on the http requests.

The problem here might be that VSCode/HAProxy are holding the port open and relaying the connection but ultimately there is nothing on the other end to receive the request. So, Lando gets neither the immediate failure from a non-existent socket nor does it get a response from a conflicting server. Lando is using a web request library for this process which can handle timeouts on response times, but the problem here is that this is happening at the network layer. The request isn't getting to a destination, so the timeout won't start counting against the response time of the server. Eventually, the proxy cancels its attempt to forward the connection(code=ECONNRESET, message=socket hang up), which finally gives Lando a failure.

So yes, there is an issue here. Lando hangs when a port is held open by a proxy but there is nothing on the other end accepting connections. The fix on the Lando side is likely to implement something that can abort the connection attempt if it takes too long, or to refactor the port scanning to use a more low-level method that doesn't rely on establishing a TCP connection.

For your case, you can likely work around the problem by configuring Lando to use ports that you know your proxy won't be using. See the proxy docs for details: https://docs.lando.dev/core/v3/proxy.html#configuration

[AaronFeledy]

Some technical notes:

scanPorts() calls getFirstOpenPort() which calls scanUrls().

[dgervalle]

Thanks @AaronFeledy for this thorough and very accurate follow up now. I fully agree and understand your analysis. And the ECONNRESET happened when I quit VSCode, which is equivalent to the proxy hang up since the port forwarding is dropped.

Just so you know, I was experiencing this issue from time to time for weeks, and it was not easy to tackle down, mainly because the verbosity of Lando on this port scanning is low, and even with -vvvv you do not know the port currently being scanned until you find the way to cancel the connection. My first experience was in a meeting where I wanted to show up something and my Lando never starts, and I was confused and very annoyed.

So, until you find a way to cancel the attempt, a great improvement could be to improve the verbosity of what is happening. That might help others understand which port is the issue, and like I have done, prevent this port to be listening or prevent Lando to scan it. I would even show up the scanning during Lando start without any verbosity options, maybe on single line, says: Scanning port XXXX, replacing XXXX as it progress, and replacing the line once done. Most of the time, this will be so quick, that it will be invisible, but this way, when it hangs, the user will see the operation in progress, and the port concerned, which would give a great clue to the actual problem.

And I agree, this is an edge case, but when you fall into it, and I am sure seeing the issue #979 that I am not the only one, it really hurts !