NERSC / shifter

Shifter - Linux Containers for HPC
Other
352 stars 63 forks source link

Fails to pull image #229

Closed ynop closed 6 years ago

ynop commented 6 years ago

Hi

I have installed the most recent version (master) of the image manager. When I try to pull an image i get the following error in the error-log-file:

[2018-08-13 18:14:08 +0000] [13701] [INFO] about to exec: ls /cluster/operation/shifter/images/34c2bfcf230016b0fb8ccbef2897fad3828769416378a6deede464d92855c1f4.meta
[2018-08-13 18:14:08 +0000] [13701] [ERROR] ls stderr: ls: cannot access '/cluster/operation/shifter/images/34c2bfcf230016b0fb8ccbef2897fad3828769416378a6deede464d92855c1f4.meta': No such file or directory
[2018-08-13 18:14:08 +0000] [13701] [INFO] about to exec: ls /cluster/operation/shifter/images/34c2bfcf230016b0fb8ccbef2897fad3828769416378a6deede464d92855c1f4.squashfs
[2018-08-13 18:14:08 +0000] [13701] [ERROR] ls stderr: ls: cannot access '/cluster/operation/shifter/images/34c2bfcf230016b0fb8ccbef2897fad3828769416378a6deede464d92855c1f4.squashfs': No such file or directory

The directory exists and the user has the permissions.

Any ideas?

Best Regards Matthias

scanon commented 6 years ago

Those errors are normal. We should really surpress them. Do the images eventually show up in the listing? Are there any errors after this?

ynop commented 6 years ago

With shifterimg images they don't show up. But when i request the queue:

curl -H "authentication: ..." -X GET -v http://gpu10:5000/api/queue/jessy/
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 160.85.37.237...
* Connected to gpu10 (160.85.37.237) port 5000 (#0)
> GET /api/queue/jessy/ HTTP/1.1
> Host: gpu10:5000
> User-Agent: curl/7.47.0
> Accept: */*
> authentication: ...
>
< HTTP/1.1 200 OK
< Server: gunicorn/19.7.1
< Date: Mon, 13 Aug 2018 16:36:33 GMT
< Connection: close
< Content-Type: application/json
< Content-Length: 238
<
{
  "list": [
    {
      "image": "ubuntu:latest",
      "status": "PULLING"
    },
    {
      "image": "cirros:latest",
      "status": "PULLING"
    },
    {
      "image": "debian:latest",
      "status": "ENQUEUED"
    }
  ]
}
* Closing connection 0

But there are no other errors.

scanon commented 6 years ago

So it says they are pulling the images. Can you look in the cache area and see if any of the tar layers are showing up?

ynop commented 6 years ago

There are some .partial files:

-rw------- 1 shifter shifter 4194304 Aug 13 18:19 sha256:3d6427f49fe318fc8062066677acb50b2f755b716313f238de517f2b751f15b9_ENj8i.partial
-rw------- 1 shifter shifter 5917431 Aug 13 17:48 sha256:c64513b741452f95d8a147b69c30f403f6289542dd7b2b51dd8ba0cb35d0e08bNbu8eD.partial

But i just tried pull with docker and it gets stuck too. Maybe there is some other problem ...

scanon commented 6 years ago

Maybe a firewall issue.

ynop commented 6 years ago

This seems to be a non-shifter problem. Thanks for the help.