search

TritonDataCenter / sdc-imgapi

SDC internal API for managing OS images
Mozilla Public License 2.0
5 stars 18 forks source link

riton-docker run postgres:latest hangs (bug in imageapi) #54

Open Adel-Magebinary opened 1 month ago

Adel-Magebinary commented 1 month ago

Hey guys,

We are just experiencing an issue with imageapi (imgapi release-20240502-20240502T001455Z-g15467c4) when docker pull "RUN XXX" layer, the imageapi hangs forever because the image size is 0 bytes. And adminui throws this error:

error (DownloadError): image 3ee6e62a-5e91-aa2f-4741-07d783821de4 file checksum (sha1) error: expected "98d8fa60dc51138e1ab283a998419faad2fcbf16", downloaded checksum was "da39a3ee5e6b4b0d3255bfef95601890afd80709"; caused by DownloadError: image 3ee6e62a-5e91-aa2f-4741-07d783821de4 file size error: expected 116 bytes, downloaded 0 bytes

ImageAPI 0 byte file [root@91e23173-1ccb-4917-83df-5a9af4f9f8b0 (magiclab:imgapi0) /data/imgapi/images/3ee/3ee6e62a-5e91-aa2f-4741-07d783821de4]# ls -lah total 3.5K drwxr-xr-x 2 nobody nobody 3 May 21 05:14 . drwxr-xr-x 4 nobody nobody 4 May 21 02:22 .. -rwxr-xr-x 1 root root 0 May 21 04:35 file0

manually downloading as it hangs forever. root@fde8253b-3a08-69cb-a9e3-a53e8628fa10:[~]: wget http://192.168.1.9/images/3ee6e62a-5e91-aa2f-4741-07d783821de4/file -vvvk --2024-05-21 05:15:44-- http://192.168.1.9/images/3ee6e62a-5e91-aa2f-4741-07d783821de4/file Connecting to 192.168.1.9:80... connected. HTTP request sent, awaiting response... 200 OK Length: 116 [application/octet-stream] Saving to: 'file.18'

0% [ ] 0 --.-K/s

You can easily reproduce this issue by triton-docker pull postgres:latest triton-docker run postgres:latest

This issue has halted our deployments, and any suggestions are appreciated.

Adel-Magebinary commented 1 month ago

I have just made sure that this is not a network issue. I zlogin to the imgapi and try to download the empty docker layer file. The download stuck forever.

[root@91e23173-1ccb-4917-83df-5a9af4f9f8b0 (magiclab:imgapi0) ~]# wget http://localhost/images/c70fdbe7-08bf-113f-53af-cabbe26014ac/file --2024-05-21 08:13:19-- http://localhost/images/c70fdbe7-08bf-113f-53af-cabbe26014ac/file Resolving localhost... ::1, 127.0.0.1 Connecting to localhost|::1|:80... failed: Connection refused. Connecting to localhost|127.0.0.1|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1164 (1.1K) [application/octet-stream] Saving to: 'file'

file 0%[ ] 0 --.-KB/s

Adel-Magebinary commented 1 month ago

@bahamat, can you replicate this issue with your coal? Any help would be greatly appreciated.