error "Underflow Somehow"

[quodlibetor]

We're currently getting an Underflow somehow panic. (From here.) This is a server crash, so the client just exits without doing anything.

I know you're averse to larger dependencies, but how would you feel about ripping out the JSON processing code and depending on serde_json?

I'm happy to do the work, but I'd rather know that it's inline with your preferences before I start.

[DeCarabas]

Can you send the JSON response that’s causing it? Like, hack it to print out the JSON that's triggering the failure.

I'll look at taking serde_json....

[DeCarabas]

I don't actually think it's going to be fixed by serde_json, actually, my guess is that what's causing it is an empty input. I don't know why docker would be sending us empty input though....

[DeCarabas]

I think I've fixed that problem but there's a deeper problem with the docker integration. It would be helpful to know what's going on there, I'll see if I can get some more diagnostics in place for docker.

[quodlibetor]

Yeah the client is starting correctly in e27b788 but it's not correctly finding the docker containers for me:

$ psgrep docker-proxy | rg 7474
  17675    1669  0.0  0.0    02:29:07 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7474 -container-ip 172.19.0.2 -container-port 7474
  17689    1669  0.0  0.0    02:29:07 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 7474 -container-ip 172.19.0.2 -container-port 7474

but

[quodlibetor]

and, yes, fwiw docker is just returning nothing but empty responses for our queries, but doing it directly works fine:

$ ncat -vlk 1080 -c 'ncat -U /var/run/docker.sock'
Ncat: Version 7.80 ( https://nmap.org/ncat )
Ncat: Listening on :::1080
Ncat: Listening on 0.0.0.0:1080
Ncat: Connection from 127.0.0.1.
Ncat: Connection from 127.0.0.1:47178.

$ curl -v http://localhost:1080/containers/json
*   Trying 127.0.0.1:1080...
* Connected to localhost (127.0.0.1) port 1080 (#0)
> GET /containers/json HTTP/1.1
> Host: localhost:1080
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.46
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/27.1.1 (linux)
< Date: Mon, 12 Aug 2024 17:11:41 GMT
< Transfer-Encoding: chunked
<
[{"Id":"...

[quodlibetor]

it does not seem like it's returning a content-size, which might be the problem.

[DeCarabas]

it does not seem like it's returning a content-size, which might be the problem.

WILD. My docker is for sure returning content-length.

What does curl -v -v --unix-socket /var/run/docker.sock http://localhost/containers/json give you? Just to take the netcat out of the equation...

I just pushed a change that logs the docker interactions at the trace level, which might be helpful? (I'm also working on the ability to write logs to a file to make it easier to actually read the dang things.)

[quodlibetor]

yeah same:

$ curl  -v -v --unix-socket /var/run/docker.sock http://localhost/containers/json
*   Trying /var/run/docker.sock:0...
* Connected to localhost (/run/docker.sock) port 80 (#0)
> GET /containers/json HTTP/1.1
> Host: localhost
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Api-Version: 1.46
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/27.1.1 (linux)
< Date: Mon, 12 Aug 2024 17:31:20 GMT
< Transfer-Encoding: chunked
<
[{"Id":"...

[DeCarabas]

OK, I've almost got this fixed, give me a minute. I don't know why docker is not sending a content-length but I can deal with it.

[quodlibetor]

no worries, thanks for debugging with me.

[DeCarabas]

Can you try from 542127f and see if that fixes it for you?

[quodlibetor]

nice that works

[DeCarabas]

FYI this is fixed in v0.9.1.

[quodlibetor]

Thank you!

[declanwilson-lw]

many thanks from all us quiet-lurkers ... top work y'all ... hope you are all doing ok.

[diggyk]