search

mc256 / starlight

Fast Container Provisioning on the Edge and over the WAN
Apache License 2.0
43 stars 6 forks source link

Cannot pull starlight images #63

Closed hangvane closed 7 months ago

hangvane commented 8 months ago

Describe your question here

Hi, I'm trying to run starlight images, but there's some issues that I can't make it work.

  1. The quickstart 1, 2 are hard to follow, because it is outdated. But finally I managed to configure proxy and snapshotter, at least I think it worked.
  2. I've converted a wordpress image into starlight format: 
    ctr-starlight convert --notify \
--platform=linux/amd64 --insecure-source --insecure-destination --profile myproxy \
mydomain/wordpress:latest mydomain:5000/wordpress:latest-sl

    The proxy said there's no error:

    root@ip-172-31-34-159:~/starlight/demo/compose# docker logs 5f16cb6c2847
time="Jan 24 08:07:45.199986106" level=info msg=starlight-proxy version=0.6.0-20231127
time="Jan 24 08:07:45.200056436" level=info msg="loaded configuration" log=debug new=true path=/etc/starlight/starlight-proxy.json
time="Jan 24 08:07:45.200075854" level=info msg="default backend registry: http://mydomain:5000"
time="Jan 24 08:07:45.201070585" level=error msg="failed to connect to database, retrying in 5 seconds (1/10)" error="dial tcp 172.22.0.3:5432: connect: connection refused"
time="Jan 24 08:07:50.231868242" level=info msg="database initialized"
time="Jan 24 08:07:50.231939720" level=info msg="listen on 0.0.0.0:8090"
time="Jan 24 08:11:27.187301146" level=info msg="request received" action=notify ip="172.31.37.107:55342"
time="Jan 24 08:11:28.632002187" level=debug msg="saved ToC" hash="sha256:2d9538d065959444bc6f78942878ac24ec1bd8d82e290dd41f75682878ee3781" image="mydomain:5000/wordpress" platform=uni-arch serial=1 tag=latest-sl
time="Jan 24 08:11:28.632029805" level=info msg="cached ToC" container="mydomain:5000/wordpress:latest-sl"

    Note that although I specified --platform=linux/amd64, the proxy still saved it as uni-arch.

  3. I try to pull starlight image by: 
    ctr-starlight pull --profile myproxy mydomain:5000/wordpress:latest-sl

    But it failed:

    ctr-starlight pull --profile myproxy mydomain:5000/wordpress:latest-sl
pull image failed: failed to pull image mydomain:5000/wordpress:latest-sl: {"status":"Bad Request","code":400,"error":"requested image mydomain:5000/wordpress:latest-sl not found"}

    I have to manuly modify the platform from uni-arch to linux/amd64 in tag database to continue.

But an error still occurred on the first try:

root@ip-172-31-37-107:~/starlight# ctr-starlight pull --profile myproxy mydomain:5000/wordpress:latest-sl
pull image failed: failed to pull image mydomain:5000/wordpress:latest-sl: {"status":"Bad Request","code":400,"error":"failed to load all compressed layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client"}
root@ip-172-31-37-107:~/starlight# ctr-starlight pull --profile myproxy mydomain:5000/wordpress:latest-sl
requested to pull image mydomain:5000/wordpress:latest-sl in 1499ms 
delta image 249.1MiB / original 255.1MiB (skipped: 6.1MiB = 2.37%)

The image isn't pulled correctly.

time="Jan 24 08:12:19.490889785" level=info msg="request received" action=delta ip="172.31.37.107:49294"
time="Jan 24 08:12:19.493557860" level=info msg="find existing file contents" _step=1 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" total=0 unique=0
time="Jan 24 08:12:19.497219737" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00018:02]sha256:592e0d17765a0d0ef67e4b8163af9745d34e4865ef5f0cbae1d793f6f33b7377-107379483"
time="Jan 24 08:12:19.497305283" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00001:03]sha256:57773387eb6bee5e1fb7f54187bc190fae16844b4efc303355af96a73eb6f6fe-656"
time="Jan 24 08:12:19.497368654" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00002:06]sha256:be50837fac8b94900458ac907ad8fc93a6489c533371658d5a4c8b0791a3afed-1201"
time="Jan 24 08:12:19.497424800" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00005:16]sha256:28904abea0a638a029ae1ada96adf7e378d352750f0845eb620b1dedf74b9333-848"
time="Jan 24 08:12:19.497479087" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00004:15]sha256:2f1ca1f66cc4626de5ab613511737159afcc91f276e032f4cd64dcb60abfd8c6-1039"
time="Jan 24 08:12:19.497531184" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00017:12]sha256:44688d4bc6c8fe47d7564c2417852a138ccfc0d2f193a735fd87075fe4500027-1335"
time="Jan 24 08:12:19.497580702" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00012:01]sha256:429a81eb9ef59357ab9dc626accb4ee17d6411382fab18d95174c34ec073335e-668"
time="Jan 24 08:12:19.497640068" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00020:10]sha256:19d82ac87a32ab4fd87c6fca68f16ebd95cd52364050cee12863514cce3ab3ff-4708"
time="Jan 24 08:12:19.497682422" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00021:08]sha256:f8207675971b9f0e4bbe665b6d4df7585c6b6a3fd0bcaf6047245d1ef9f99693-939"
time="Jan 24 08:12:19.497694816" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00003:05]sha256:c67ed66059b092e4e09579d6aed9a414f555a880a1bfc945310f51632e05c365-988"
time="Jan 24 08:12:19.497755408" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00007:19]sha256:7a532a96a66f44d9e9071c5991b4a259398a08c127f5ffba13b858260427118e-2814"
time="Jan 24 08:12:19.497756289" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00016:20]sha256:94b84a62a5e359745ad76c367bdca9bf2ce3bdc8bb116a2393259a9921908dc0-2165"
time="Jan 24 08:12:19.497818488" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00010:09]sha256:d131dca527404f803ca14286a282ff54ae7d8cb4b2425f39a694e1f8b8d5f9ca-11584821"
time="Jan 24 08:12:19.497823945" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00006:14]sha256:adb908868ec8b98b233dcf64d4630231da94d9304dff51b5c096a4f81e7a3a05-31137002"
time="Jan 24 08:12:19.497880953" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00009:04]sha256:cc291c0f04ce615e4c404a5917bdd2b31477c821ccf4a35dd3dfb70267fafcaf-20637845"
time="Jan 24 08:12:19.497886913" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00019:17]sha256:b52220c829a31f50a974358e881e6a48dd3ae0f58d86371cf3fc3c4b0b1a41f5-30327"
time="Jan 24 08:12:19.497934573" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00011:07]sha256:c44de33f5c800c4dc1fe0c961db0b672bbd850a7b12681baf41b45407189f0b6-12408281"
time="Jan 24 08:12:19.497947786" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00014:11]sha256:d023f13378905e4245835eb0c90151794cb7b0976b9197b57642d86d57c7e9a5-706"
time="Jan 24 08:12:19.497989034" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00015:13]sha256:58e1055b1087ab23611b14b7978f38cac12117f20e31af2c44a30e08574e7976-26905737"
time="Jan 24 08:12:19.498007648" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00008:18]sha256:136fc8617fa3e7c23cb11d31b4c285b765c4fa8d36dcd609293ab16abb06524e-25927104"
time="Jan 24 08:12:19.498041063" level=error msg="failed to load layer: Get \"https://mydomain:5000/v2/\": http: server gave HTTP response to HTTPS client" layer="[00013:00]sha256:bd83efce2a367393cfd51477c15b8111e333e6d2a2f0bf06ff8e70c8ee98acd4-31500556"
time="Jan 24 08:12:19.645177374" level=info msg="find requested file contents" _step=2 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" total=20685 unique=15511
time="Jan 24 08:12:19.649755778" level=info msg="find the best file content references" _step=3 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" compressedSize=258944523 content=15511
time="Jan 24 08:12:22.266492900" level=info msg="request received" action=delta ip="172.31.37.107:49294"
time="Jan 24 08:12:22.267498744" level=info msg="fetched layer" layer="[00002:06]sha256:be50837fac8b94900458ac907ad8fc93a6489c533371658d5a4c8b0791a3afed-1201" shared=true
time="Jan 24 08:12:22.267509832" level=info msg="fetched layer" layer="[00018:02]sha256:592e0d17765a0d0ef67e4b8163af9745d34e4865ef5f0cbae1d793f6f33b7377-107379483" shared=true
time="Jan 24 08:12:22.267517768" level=info msg="fetched layer" layer="[00009:04]sha256:cc291c0f04ce615e4c404a5917bdd2b31477c821ccf4a35dd3dfb70267fafcaf-20637845" shared=true
time="Jan 24 08:12:22.267532116" level=info msg="fetched layer" layer="[00001:03]sha256:57773387eb6bee5e1fb7f54187bc190fae16844b4efc303355af96a73eb6f6fe-656" shared=true
time="Jan 24 08:12:22.267553365" level=info msg="fetched layer" layer="[00003:05]sha256:c67ed66059b092e4e09579d6aed9a414f555a880a1bfc945310f51632e05c365-988" shared=true
time="Jan 24 08:12:22.267541640" level=info msg="fetched layer" layer="[00013:00]sha256:bd83efce2a367393cfd51477c15b8111e333e6d2a2f0bf06ff8e70c8ee98acd4-31500556" shared=true
time="Jan 24 08:12:22.267566894" level=info msg="fetched layer" layer="[00006:14]sha256:adb908868ec8b98b233dcf64d4630231da94d9304dff51b5c096a4f81e7a3a05-31137002" shared=true
time="Jan 24 08:12:22.267639796" level=info msg="fetched layer" layer="[00015:13]sha256:58e1055b1087ab23611b14b7978f38cac12117f20e31af2c44a30e08574e7976-26905737" shared=true
time="Jan 24 08:12:22.267660060" level=info msg="fetched layer" layer="[00012:01]sha256:429a81eb9ef59357ab9dc626accb4ee17d6411382fab18d95174c34ec073335e-668" shared=true
time="Jan 24 08:12:22.267575819" level=info msg="fetched layer" layer="[00008:18]sha256:136fc8617fa3e7c23cb11d31b4c285b765c4fa8d36dcd609293ab16abb06524e-25927104" shared=true
time="Jan 24 08:12:22.267653977" level=info msg="fetched layer" layer="[00014:11]sha256:d023f13378905e4245835eb0c90151794cb7b0976b9197b57642d86d57c7e9a5-706" shared=true
time="Jan 24 08:12:22.267587468" level=info msg="fetched layer" layer="[00004:15]sha256:2f1ca1f66cc4626de5ab613511737159afcc91f276e032f4cd64dcb60abfd8c6-1039" shared=true
time="Jan 24 08:12:22.267585728" level=info msg="fetched layer" layer="[00011:07]sha256:c44de33f5c800c4dc1fe0c961db0b672bbd850a7b12681baf41b45407189f0b6-12408281" shared=true
time="Jan 24 08:12:22.267684508" level=info msg="fetched layer" layer="[00016:20]sha256:94b84a62a5e359745ad76c367bdca9bf2ce3bdc8bb116a2393259a9921908dc0-2165" shared=true
time="Jan 24 08:12:22.267615599" level=info msg="fetched layer" layer="[00019:17]sha256:b52220c829a31f50a974358e881e6a48dd3ae0f58d86371cf3fc3c4b0b1a41f5-30327" shared=true
time="Jan 24 08:12:22.267623206" level=info msg="fetched layer" layer="[00020:10]sha256:19d82ac87a32ab4fd87c6fca68f16ebd95cd52364050cee12863514cce3ab3ff-4708" shared=true
time="Jan 24 08:12:22.267703593" level=info msg="fetched layer" layer="[00007:19]sha256:7a532a96a66f44d9e9071c5991b4a259398a08c127f5ffba13b858260427118e-2814" shared=true
time="Jan 24 08:12:22.267638445" level=info msg="fetched layer" layer="[00017:12]sha256:44688d4bc6c8fe47d7564c2417852a138ccfc0d2f193a735fd87075fe4500027-1335" shared=true
time="Jan 24 08:12:22.267611169" level=info msg="fetched layer" layer="[00010:09]sha256:d131dca527404f803ca14286a282ff54ae7d8cb4b2425f39a694e1f8b8d5f9ca-11584821" shared=true
time="Jan 24 08:12:22.267597678" level=info msg="fetched layer" layer="[00021:08]sha256:f8207675971b9f0e4bbe665b6d4df7585c6b6a3fd0bcaf6047245d1ef9f99693-939" shared=true
time="Jan 24 08:12:22.267601147" level=info msg="fetched layer" layer="[00005:16]sha256:28904abea0a638a029ae1ada96adf7e378d352750f0845eb620b1dedf74b9333-848" shared=true
time="Jan 24 08:12:22.267905689" level=info msg="find existing file contents" _step=1 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" total=0 unique=0
time="Jan 24 08:12:22.414975846" level=info msg="find requested file contents" _step=2 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" total=20685 unique=15511
time="Jan 24 08:12:22.419693100" level=info msg="find the best file content references" _step=3 builder="Builder ()->(mydomain:5000/wordpress:latest-sl)" compressedSize=258944523 content=15511
time="Jan 24 08:12:22.889638557" level=info msg="generated response header" _digest="sha256:ad85b3baebd712667d58fd199e28f4b72bc0764840002b222fc4c12a629315e0" body=258944523 config=5207 header=2231476 manifest=2279
time="Jan 24 08:12:22.914611437" level=debug msg="header sent" action=delta ip="172.31.37.107:49294"
2024/01/24 08:12:22 http: panic serving 172.31.37.107:49294: runtime error: invalid memory address or nil pointer dereference
goroutine 258 [running]:
net/http.(*conn).serve.func1()
        /usr/local/go/src/net/http/server.go:1854 +0xbf
panic({0x952420, 0xe2acf0})
        /usr/local/go/src/runtime/panic.go:890 +0x263
io.(*SectionReader).ReadAt(0xc00630ee28?, {0xc005cae840?, 0x95c560?, 0xc00630ee20?}, 0x40a853?)
        /usr/local/go/src/io/io.go:540 +0x25
io.(*SectionReader).ReadAt(0x7fa265c51288?, {0xc005cae840?, 0x60?, 0xc000450000?}, 0xc005cae840?)
        /usr/local/go/src/io/io.go:552 +0x9f
io.(*SectionReader).Read(0xc006312330, {0xc005cae840?, 0x927d60?, 0xc006066801?})
        /usr/local/go/src/io/io.go:513 +0x54
io.(*LimitedReader).Read(0xc005ce32a8, {0xc005cae840?, 0xc0060668b0?, 0x10?})
        /usr/local/go/src/io/io.go:477 +0x45
io.copyBuffer({0xabd000, 0xc0060668b0}, {0xabc2a0, 0xc005ce32a8}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:427 +0x1b2
io.Copy(...)
        /usr/local/go/src/io/io.go:386
net.genericReadFrom({0xabc4c0?, 0xc0004b8000?}, {0xabc2a0, 0xc005ce32a8})
        /usr/local/go/src/net/net.go:675 +0x6a
net.(*TCPConn).readFrom(0xc0004b8000, {0xabc2a0, 0xc005ce32a8})
        /usr/local/go/src/net/tcpsock_posix.go:54 +0x78
net.(*TCPConn).ReadFrom(0xc0004b8000, {0xabc2a0?, 0xc005ce32a8?})
        /usr/local/go/src/net/tcpsock.go:130 +0x36
net/http.(*response).ReadFrom(0xc000d72380, {0xabc2a0?, 0xc005ce32a8})
        /usr/local/go/src/net/http/server.go:599 +0x36e
io.copyBuffer({0xabc760, 0xc000d72380}, {0xabc2a0, 0xc005ce32a8}, {0x0, 0x0, 0x0})
        /usr/local/go/src/io/io.go:413 +0x14b
io.Copy(...)
        /usr/local/go/src/io/io.go:386
io.CopyN({0xabc760, 0xc000d72380}, {0xabc300?, 0xc006312330}, 0x57)
        /usr/local/go/src/io/io.go:362 +0x9a
github.com/mc256/starlight/proxy.(*Builder).WriteBody(0xc00017d030?, {0xabfb90?, 0xc000d72380}, 0x2?)
        /go/src/app/proxy/builder.go:160 +0x3a5
github.com/mc256/starlight/proxy.(*Server).delta(0xc0000cb0e0, {0xabfb90, 0xc000d72380}, 0xc000d66d00)
        /go/src/app/proxy/server.go:154 +0x9f2
net/http.HandlerFunc.ServeHTTP(0xc0000ac180?, {0xabfb90?, 0xc000d72380?}, 0x40d90a?)
        /usr/local/go/src/net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc003720475?, {0xabfb90, 0xc000d72380}, 0xc000d66d00)
        /usr/local/go/src/net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0xc00049a0c0?}, {0xabfb90, 0xc000d72380}, 0xc000d66d00)
        /usr/local/go/src/net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc000098000, {0xac0010, 0xc00007a510})
        /usr/local/go/src/net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3089 +0x5ed

And there's no doubt that the creation would fail.

Seems the proxy is trying to connect to the registry by HTTPS. But there's no way to change to HTTP:

  1. I couldn't find a flag such as --plain-http for ctr-starlight pull.
  2. There's no way to configure snapshotter or daemon. It seems that only the protocol used to connect to the proxy is configurable.
  3. It seems like there's a way to configure proxy by setting the environment variable ``REGISTRY=http://mydomain:5000", but there's no instructions on how to use the default backend. I set this environment variable as I described at the beginning, but the error still occurred.

Any tips or guidance would be greatly appreciated.

What kind of hardward you have?

mc256 commented 7 months ago

We mainly use Kubernetes in production. But I will update the quick start guide! ETA is 1 week

mc256 commented 7 months ago

Hi @hangvane, Could you try this guide instead?

https://github.com/mc256/starlight/blob/4dec5a91cb83be7c68e052c82e15128818dcb9bb/docs/newbie.md#tldr-all-in-one-quick-start-guide

The quickstart 1, 2 are for 0.1.2 and are indeed outdated. I am in the process of creating a Terraform for AWS deployment, so we don't need to write CLI in the prompt.

Please let me know if you any further questions!