Open almusil opened 2 months ago
Thanks for reporting this @almusil
Can you please provide the version of the service that crashed? Looks like it's the assisted-service, you can extract the version by podman inspect and paste here ImageDigest
field
How was the service deployed? Is this a podman deployment? Where is the output you have pasted above coming from? (what's the access log line, is it the UI component logs interleaved with assisted-service's logs?)
Thanks for reporting this @almusil
Can you please provide the version of the service that crashed? Looks like it's the assisted-service, you can extract the version by podman inspect and paste here
ImageDigest
field How was the service deployed?
ImageDigest for all the AI images, even though the log suggests that it is the assisted-service.
# podman inspect quay.io/edge-infrastructure/assisted-image-service -f "{{ .Digest }}"
sha256:3b6baaeef183b158872468e4027f7851acb8c6dd6f45d80479a86360ab8c2682
# podman inspect quay.io/edge-infrastructure/assisted-service -f "{{ .Digest }}"
sha256:b5e0616e829acef9592fd87936b2f1da709426b24b4273879baad13c1d894db1
# podman inspect quay.io/edge-infrastructure/assisted-installer-ui -f "{{ .Digest }}"
sha256:38f20bd8214ed7a414e813daf1531b5e98db55433fc423024ddea87adb6d4798
Is this a podman deployment?
Yes it is podman deployment.
Where is the output you have pasted above coming from?
This was observed in the journal. The AI is running as part of bigger project to automate openshift deployments.
(what's the access log line, is it the UI component logs interleaved with assisted-service's logs?)
Not sure if I understand this one, can you please elaborate?
Not sure if I understand this one, can you please elaborate?
I'm interested in those lines:
10.45.224.87 - - [09/Jul/2024:14:01:35 +0000] "GET /api/assisted-install/v2/clusters/20e31b11-3aba-4457-9e44-7c5f9efa3a69 HTTP/1.1" 200 43515 "http://wsfd-advnetlab157.anl.eng.rdu2.dc.redhat.com:8080/clusters/20e31b11-3aba-4457-9e44-7c5f9efa3a69" "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0" "-"
2024/07/09 14:01:35 http: panic serving 127.0.0.1:43844: write tcp 127.0.0.1:8090->127.0.0.1:43844: write: broken pipe
The first line seems to be an access log, which should not have been emitted by assisted-service.
The second line is a panic for unhandled broken connection, which might be assisted service
. Can you confirm it's the assisted service emitting the panic log, and not other applications?
If that's the case, could you describe what is "accessing the cluster page"? Is it accessing assisted-service-ui at the cluster page? If so, could you also try with curl
and the API if we get the same crash?
It seems that the service could be trying to write to a closed connection.
From the journal it is the assisted-installer-service:
assisted-installer-service[2211655]: 2024/07/10 13:09:40 http: panic serving 127.0.0.1:42856: write tcp 127.0.0.1:8090->127.0.0.1:42856: write: broken pipe
I cannot pinpoint exact URL that causes this with CURL.
Clicking the ovnscale
causes the failure, not sure how to better describe it.
the path is GET /api/assisted-install/v2/clusters/20e31b11-3aba-4457-9e44-7c5f9efa3a69
(as in the access logs), but you probably need authentication, would you be able to run that and see if it causes the same issue?
You can also inspect what the UI is doing with developer tools and replay the same request, but from curl instead
Tried both without any success unfortunately
WDYM "without any success"? Still crashing? Can you try to paste the curl's output?
The curl doesn't crash, I don't have the cluster anymore unfortunately
Do you have a minimal example/setup where we can reproduce this issue?
I've got the following crash in the logs when I try to access the cluster page during installation:
The container version of AI: