TritonDataCenter / sdc-imgapi

SDC internal API for managing OS images
Mozilla Public License 2.0
5 stars 18 forks source link

Docker Image is not importing. #53

Closed Adel-Magebinary closed 3 months ago

Adel-Magebinary commented 3 months ago

TDC, with the latest release, is not importing the image; it feels like something has timed out. This is happening with

triton-docker --tls run nginx:latest

It feels like there is some kind of lock timeout. Then I try to replicate the issue by running imgadm manually on CN, and the same thing happens.

[root@xxx-cn-r420-01 (xxx) ~]# imgadm import 8916ec83-38fd-6deb-fa44-f94976fc9035
Importing 8916ec83-38fd-6deb-fa44-f94976fc9035 (docker-layer@5252b206aac2) from "http://imgapi.dc.zone"
Gather image 8916ec83-38fd-6deb-fa44-f94976fc9035 ancestry
Must download and install 3 images (67.6 MiB)
Download 3 images       [==========================>       ]  79%  53.97MB                  

Download 3 images       [================================> ]  98%  67.00MB 571.51KB/s     1s
Aborting (multiple (2) errors
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file size error: expected 629 bytes, downloaded 0 bytes
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file checksum (sha1) error: expected "abef6a607cbad1a9b1ebf12c2198c2b40d2e23bc", downloaded checksum was "da39a3ee5e6b4b0d3255bfef95601890afd80709")
imgadm import: error (MultiError): multiple (2) errors
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file size error: expected 629 bytes, downloaded 0 bytes
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file checksum (sha1) error: expected "abef6a607cbad1a9b1ebf12c2198c2b40d2e23bc", downloaded checksum was "da39a3ee5e6b4b0d3255bfef95601890afd80709"
Adel-Magebinary commented 3 months ago

Doing bit more debugging:

{"name":"imgadm","req_id":"f8ece068-a634-4aa5-beae-f85590a301ce","hostname":"magiclab-cn-r420-01","pid":24026,"level":20,"uuid":"8916ec83-38fd-6deb-fa44-f94976fc9035","msg":"creating sha1 checksum hash","time":"2024-05-06T05:46:01.517Z","v":0}

[root@xx-cn-r420-01 (magiclab) ~]# export IMGADM_LOG_LEVEL=debug
[root@xx-cn-r420-01 (magiclab) ~]# imgadm import 8916ec83-38fd-6deb-fa44-f94976fc9035

imgadm import: error (MultiError): MultiError: multiple (2) errors
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file size error: expected 629 bytes, downloaded 0 bytes
    error (DownloadError): image 8916ec83-38fd-6deb-fa44-f94976fc9035 file checksum (sha1) error: expected "abef6a607cbad1a9b1ebf12c2198c2b40d2e23bc", downloaded checksum was "da39a3ee5e6b4b0d3255bfef95601890afd80709"; caused by DownloadError: image 8916ec83-38fd-6deb-fa44-f94976fc9035 file size error: expected 629 bytes, downloaded 0 bytes
    at Object.checksum [as func] (/usr/img/lib/imgadm.js:1511:22)
    at Object._onImmediate (/usr/img/node_modules/vasync/lib/vasync.js:213:20)
    at processImmediate [as _immediateCallback] (timers.js:330:15)
Adel-Magebinary commented 3 months ago

There is definitely a checksum issue.

Adel-Magebinary commented 3 months ago

This can close. it's IO issue from zfs pool. I replaced the disk and it is fixed.

Adel-Magebinary commented 3 months ago

Here is how i debuged it. I tried to manually download the image from the imgapi

wget imgapi.zones/images/6ecb61c4-1345-8759-a544-a02e329f5950/file

for error image, if it stuck, means something is going on with the io.