search

containerd / stargz-snapshotter

Fast container image distribution plugin with lazy pulling
https://github.com/containerd/containerd/issues/3731
Apache License 2.0
1.17k stars 114 forks source link

Support non-registry mount sources #668

Open ktock opened 2 years ago

ktock commented 2 years ago

Stargz snapshotter should support more mount sources other than registry. They include the following (discussed at https://github.com/moby/buildkit/pull/2685#issuecomment-1058888215).

qi0523 commented 2 years ago

I want to implement P2P lazy pulling with containerd and stargz-snapshotter. Each node deploys a agent listening to 5001, which implements p2p distribution.

When i use ctr-remote image rpull docker.io/qi0523/ubuntu:latest to pull image lazily. I find that index.json, manifest.json, config.json can be downloaded from docker hub. The layer.tar doesn't exist in /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

However, when i use ctr-remote image rpull 192.168.56.101:5001/qi0523/ubuntu:latest to pull image lazily. I find that layer.tar also needs to be downloaded from agent.

I find that it starts to pull layer.tar after this line is executed.

ktock commented 2 years ago

@qi0523 Could you provide a reproducible step and the log for that issue? There are several docs for IPFS + stargz-snapshotter. Have you followed some of these steps?

qi0523 commented 2 years ago

@qi0523 Could you provide a reproducible step and the log for that issue? There are several docs for IPFS + stargz-snapshotter. Have you followed some of these steps?

Thanks.

The agent is a simple component implemented by myself. It uses the image data under the directory(/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256) to serve HEAD and GET requests, such as http://xxx/v2/xxx/manifests/xxx and http://xxx/v2/xxx/blobs/xxx

ktock commented 2 years ago

@qi0523 So you're implementing your own registry API, right? Does that API fully implements OCI Distribution Spec? For successful lazy pulling, it needs to implement HTTP Range Request for blobs API at least.

Could you provide the log printed by stargz snapshotter (containerd-stargz-grpc) so that we can deeply investigate the issue?

You can also find the registry client logic at https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/remote/resolver.go

qi0523 commented 2 years ago

@qi0523 So you're implementing your own registry API, right? Does that API fully implements OCI Distribution Spec? For successful lazy pulling, it needs to implement HTTP Range Request for blobs API at least.

Could you provide the log printed by stargz snapshotter (containerd-stargz-grpc) so that we can deeply investigate the issue?

You can also find the registry client logic at https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/fs/remote/resolver.go

Yes, i read the code of distribution/distribution and use http.ServeContent(w, r, dgst.String(), time.Time{}, br) to implement HTTP Range Request for blobs API.

logs

The logs show that stargz snapshotter can not extract sha256:73d944bd82

ktock commented 2 years ago

@qi0523 The error logs look truncated. Could you copy them and post here as text?

qi0523 commented 2 years ago

Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"info","msg":"preparing filesy stem mount at mountpoint=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","parent":"","time":"2022-10-25T08:13:54.693558513Z"} Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib /containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"resolving","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6 ","time":"2022-10-25T08:13:54.703000767Z"} Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"digest":"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","error":null,"key":"default/6/extract-643806365-5X68 sha256:73d944bd 82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"using default handler","parent":"","ref":"192. 168.56.106:5001/qi0523/ubuntu:latest","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","time":"2022-10-25T08:13:54.706477638Z"} Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:54.716664705Z"} Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:54.778420021Z"} Oct 25 08:13:54 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:54.899760506Z"} Oct 25 08:13:55 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:55.037032914Z"} Oct 25 08:13:55 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:55.518904153Z"} Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: serv er gave HTTP response to HTTPS client","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug","mountpoint":"/var/lib/containerd -stargz-grpc/snapshotter/snapshots/4/fs","msg":"Retrying request","parent":"","src":"192.168.56.106:5001/qi0523/ubuntu:latest/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6","t ime":"2022-10-25T08:13:56.344004361Z"} Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"failed to resolve layer \"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\" from \"192.168.56.106:5001/qi0523/ubuntu:l atest\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"192.168.56.106:5001\", ref:\"192.168.56.106:5001/qi0523/ubuntu:latest\", digest:\"sha 256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\"): failed to request: GET https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88 c3a3afb27c60939c6 giving up after 6 attempt(s): Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\": http: server gave HTT P response to HTTPS client: failed to resolve: failed to resolve target","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c268150107fab2058","level":"debug" ,"mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/4/fs","msg":"failed to resolve layer","parent":"","time":"2022-10-25T08:13:56.347004192Z"} Oct 25 08:13:56 xv6 containerd-stargz-grpc[645]: {"error":"failed to resolve layer: failed to resolve layer \"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\" from \"192.168.56 .106:5001/qi0523/ubuntu:latest\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"192.168.56.106:5001\", ref:\"192.168.56.106:5001/qi0523/ubun tu:latest\", digest:\"sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6\"): failed to request: GET https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb 23d447c74ff234d01da7b6d88c3a3afb27c60939c6 giving up after 6 attempt(s): Get \"https://192.168.56.106:5001/v2/qi0523/ubuntu/blobs/sha256:407f4ecfeabd9e7fefbbeb23d447c74ff234d01da7b6d88c3a3afb27c60939c6 \": http: server gave HTTP response to HTTPS client: failed to resolve: failed to resolve target","key":"default/6/extract-643806365-5X68 sha256:73d944bd82a9c645ec64e7a4b41e2453945687efcfd83b8c26815010 7fab2058","level":"warning","msg":"failed to prepare remote snapshot","parent":"","remote-snapshot-prepared":"false","time":"2022-10-25T08:13:56.348244797Z"}

ktock commented 2 years ago

@qi0523

http: server gave HTTP response to HTTPS client

The error message says the server gave HTTP response to HTTPS client. So could you configure stargz snapshotter (/etc/containerd-stargz-grpc/config.toml) to use plain HTTP for connecting to your registry?

https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-mirrors-and-insecure-connection

qi0523 commented 2 years ago

@qi0523

http: server gave HTTP response to HTTPS client

The error message says the server gave HTTP response to HTTPS client. So could you configure stargz snapshotter (/etc/containerd-stargz-grpc/config.toml) to use plain HTTP for connecting to your registry?

https://github.com/containerd/stargz-snapshotter/blob/v0.12.1/docs/overview.md#registry-mirrors-and-insecure-connection

Thanks.

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

ktock commented 2 years ago

@qi0523

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

This is because of lazy pulling. containerd doesn't pull layer blobs but stargz-snapshotter lazily pulls chunks of layer blobs.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

Yes. Blobs are stored under that directory and they are chunked. Note that some chunks of blobs can be lack until stargz-snapshotter completes downloading the entire blob contents to the node.

The followings are some of packages managing them:

qi0523 commented 1 year ago

@qi0523

Another question is that when we use stargz-snaphotter, layer.tar doesn't exist in directory /var/lib/containerd/io.containerd.content.v1.content/blobs/sha256.

This is because of lazy pulling. containerd doesn't pull layer blobs but stargz-snapshotter lazily pulls chunks of layer blobs.

The data of /var/lib/containerd-stargz-grpc/ can be used to serve blob GET request?

Yes. Blobs are stored under that directory and they are chunked. Note that some chunks of blobs can be lack until stargz-snapshotter completes downloading the entire blob contents to the node.

The followings are some of packages managing them:

Hi, i try to use the data of /var/lib/containerd-stargz-grpc/ to serve blob GET request.
Generally, the URL of GET request is http://xxxxx/blobs/sha256:xxx. So, we can only know the digest of the blob. Although the above packages can manage the stargz layer, the mapping of blobs‘ digest and stargz layer is not maintained. How can i find out stargz layer by using the digest of the blob(sha256:xxx)?