search

containerd / overlaybd

Overlaybd: a block based remote image format. The storage backend of containerd/accelerated-container-image.
Apache License 2.0
259 stars 58 forks source link

RegistryFS V2 fails to connect to registry while RegistryFS V1 works fine #229

Closed gorbak25 closed 9 months ago

gorbak25 commented 1 year ago

What happened in your environment?

I have an overlaybd lazy pulling config which works on RegistryFS v1 but fails on Registry FS v2

What did you expect to happen?

No response

How can we reproduce it?

With the following overlaybd config

{
   "hocusBaseRemoteRef":"host.docker.internal:9999/hocus/hocus-block-registry-tests@sha256:cd03e262d2462f12649eb4ad7c8e037edccfd3fa1b49fa116635a5658687c514",
   "repoBlobUrl":"host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs",
   "download":{
      "enable":true,
      "delay":0
   },
   "lowers":[
      {
         "hocusLayerType":"lazy",
         "dir":"/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/layers/sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21",
         "digest":"sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21",
         "size":4739584
      },
      {
         "hocusLayerType":"lazy",
         "dir":"/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/layers/sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190",
         "digest":"sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190",
         "size":57344
      },
      {
         "hocusLayerType":"lazy",
         "dir":"/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/layers/sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059",
         "digest":"sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059",
         "size":57856
      }
   ],
   "resultFile":"/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/run/obd-result-im_im1",
   "hocusImageId":"im_im1"
}

Everything works fine on V1 but on V2 I get this in the log file:

2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/image_service.cpp:223|read_global_config_and_set:log config: [log_level=1][log_path=/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/logs/overlaybd.log][log_size=10485760][log_num=3]
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/image_service.cpp:325|init:cache config: [cache_type=download][cache_dir=/srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/obd_registry_cache][cache_size_GB=4][refill_size=262144]
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/image_service.cpp:340|init:create registryfs with cafile:/etc/ssl/certs/ca-certificates.crt, version:v2
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/build/_deps/photon-src/io/epoll.cpp:289|new_epoll_engine:Init event engine: epoll
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/build/_deps/photon-src/io/epoll.cpp:289|new_epoll_engine:Init event engine: epoll
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/overlaybd/cache/download_cache/download_cache.cpp:115|DownloadCacheFs:new DownloadCacheFs
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/image_service.cpp:416|init:use gzip file cache
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:417|main:blocking netlink
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:420|main:blocking netlink done
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:428|main:resetting netlink
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:430|main:reset netlink done
2023/06/16 00:22:01|INFO |th=00007F649EF5F740|/overlaybd/src/main.cpp:292|dev_open:dev open /srv/jailer/tests/4fdbf6e3-be13-411d-9957-7fdec4f26ec2/block-registry/block_config/im_im1
2023/06/16 00:22:01|INFO |th=00007F645DF55E80|/overlaybd/src/image_file.cpp:141|__open_ro_remote:open file from remotefs: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21, size: 4739584
2023/06/16 00:22:01|INFO |th=00007F645D752740|/overlaybd/src/image_file.cpp:141|__open_ro_remote:open file from remotefs: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190, size: 57344
2023/06/16 00:22:01|INFO |th=00007F645CF4AEC0|/overlaybd/src/image_file.cpp:141|__open_ro_remote:open file from remotefs: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059, size: 57856
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/kernel_socket.cpp:260|do_connect:Failed to connect socket errno=111(Connection refused)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:155|do_roundtrip:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:400|get_length:failed to get meta from server
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:426|open:failed to open and stat registry file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21, ret -1
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/overlaybd/cache/download_cache/download_cache.cpp:123|open:failed to open src: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21 errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/image_file.cpp:161|__open_ro_remote:failed to open remote file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:a8b5fca80efae55088290f3da8110d7742de55c2a378d5ab53226a483f390e21: No such file or directory errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645DF55E80|/overlaybd/src/image_file.cpp:246|do_parallel_open_files:failed to open files
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:400|get_length:failed to get meta from server
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:426|open:failed to open and stat registry file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190, ret -1
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/overlaybd/cache/download_cache/download_cache.cpp:123|open:failed to open src: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190 errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/image_file.cpp:161|__open_ro_remote:failed to open remote file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:0f4cc9cb34d36dbb2e59b3f466c200cf33a09e6064ee2bc0fc052969c3c2e190: No such file or directory errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645D752740|/overlaybd/src/image_file.cpp:246|do_parallel_open_files:failed to open files
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/build/_deps/photon-src/net/http/client.cpp:246|call:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:260|get_scope_auth:connection failed
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:400|get_length:failed to get meta from server
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/registryfs/registryfs_v2.cpp:426|open:failed to open and stat registry file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059, ret -1
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/overlaybd/cache/download_cache/download_cache.cpp:123|open:failed to open src: host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059 errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/image_file.cpp:161|__open_ro_remote:failed to open remote file host.docker.internal:9999/v2/hocus/hocus-block-registry-tests/blobs/sha256:aa2a28232710ebbcc24947127eac53e0a47db8b9ec84c2e83829469bf01da059: No such file or directory errno=2(No such file or directory)
2023/06/16 00:22:01|ERROR|th=00007F645CF4AEC0|/overlaybd/src/image_file.cpp:246|do_parallel_open_files:failed to open files
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/src/image_file.cpp:337|open_lowers:layer index 0 open failed, exit.
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/src/image_file.cpp:470|init_image_file:open lower layer failed.
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/src/main.cpp:302|dev_open:create image file failed
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/build/_deps/tcmu-src/libtcmu.cpp:605|device_add:handler open failed for uio0
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/build/_deps/tcmu-src/libtcmu.cpp:560|device_add:could not find handler for uio1
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/build/_deps/tcmu-src/libtcmu.cpp:560|device_add:could not find handler for uio2
2023/06/16 00:22:01|ERROR|th=00007F649EF5F740|/overlaybd/build/_deps/tcmu-src/libtcmu.cpp:640|device_remove:Could not remove device uio0: not found.
2023/06/16 00:22:01|INFO |th=00007F64A47C7740|/overlaybd/src/main.cpp:371|sigint_handler:sigint received
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:460|main:main loop exited
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/main.cpp:463|main:tcmulib closed
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/overlaybd/cache/download_cache/download_cache.cpp:118|~DownloadCacheFs:delete DownloadCacheFs
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/build/_deps/photon-src/io/epoll.cpp:75|~EventEngineEPoll:Finish event engine: epoll
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/build/_deps/photon-src/io/epoll.cpp:75|~EventEngineEPoll:Finish event engine: epoll
2023/06/16 00:22:01|INFO |th=00000000037458E0|/overlaybd/src/image_service.cpp:485|~ImageService:image service is fully stopped

What is the version of your Overlaybd?

https://github.com/hocus-dev/overlaybd/tree/bd560ed9066345fd71a1d44489d90e98662ba234 https://github.com/hocus-dev/hocus/blob/3e163c1792f12cacdc5054ed6faa3bc20162ac31/ops/docker/prebuilds.Dockerfile#L7

What is your OS environment?

Ubuntu 22.04.2 LTS running on Hocus

Are you willing to submit PRs to fix it?

liulanzheng commented 1 year ago

@gorbak25 Thank you for your feedback. Please use registryfs_v1 for now. We currently recommend users to use registryfs_v1 as possible. Regitryfs_v2 still has some known issues that have not been resolved, including the parsing problem when the URL contains a port, which may be the problem you encountered.

gorbak25 commented 1 year ago

@liulanzheng I've checked that if I use quay directly without a proxy in between(host.docker.internal:9999) the lazy pulling still fails on v2 but works on v1

liulanzheng commented 1 year ago

I will fix this within few days.

liulanzheng commented 1 year ago

@gorbak25 hi~ I updated the PhotonLibOS version in https://github.com/containerd/overlaybd/commit/7fd3f39255f5fae0b492cbdda9a1491673f6d49d. Registryfs v2 uses http module in PhotonLibOS and i fixed the problem about address with port in PhotonLibOS several days ago. Please have a try.