search

dragonflyoss / Dragonfly

This repository has be archived and moved to the new repository https://github.com/dragonflyoss/Dragonfly2.
https://d7y.io
Apache License 2.0
6k stars 774 forks source link

Unable to use harbor p2p distribution feature to preheat images with dragonfly #1537

Open bmmadhuprasad opened 3 years ago

bmmadhuprasad commented 3 years ago

I have deployed Harbor v2.0 and dragonfly latest version trying to use harbor p2p image distribution to dragonfly but getting below when applied the preheat policy to preheat image Check preheat progress: Task is failing and distribution is not happening

2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'dragonfly/test:0.1@sha256:54c9b521ae50156029ea48394041885f33414c1dad2a5deefcaa46ddf22079a3' to the target preheat provider: dragonfly newdragon:http://dragonflynew-supernode.dfn.svc.cluster.local:8002 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done 2021-03-31T13:19:39Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m) 2021-03-31T13:19:49Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:19:59Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:09Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:19Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z 2021-03-31T13:20:29Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [55b2dd4d29556dbffae842f1fefa7d7b05f7dd34c565c9febcbc99dfda187dd6] is failed, start time=2021-03-31T13:19:39.918Z

Below is logs from dragonfly

2021-03-31 14:17:50.496 DEBU sign:9 : GET /_ping remote:192.168.71.183:40028 cost:2.901µs err: 2021-03-31 14:17:50.498 DEBU sign:9 : create preheat: Type[image] URL[https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1] Filter[] Identifier[] Headers[map[Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTcyMDIwNjcsImp0aSI6Imc3TFhnUmJidmkzOFV4bFUiLCJpYXQiOjE2MTcyMDAyNjcsImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTcyMDAyNjcsInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.DFeugff4V8VplCOGrW1AqAUhUTRo8F4pCwPAKa06zP8A67Euls4GC6MidfXpzAazU5QndtHBMkKdh-CBAUgzKLv73F3Kz1tdhXBAcyGXJHX7r8otQ4J8wNf0TwOHesmoW1sKc0IUBmlsA_a4fXR_7DAspzJmFiBchNin5qGCvhvn6449Tr_kF5HyoknOrw1NoK5_mG0mcBWLSC6E_16P1YIeAWaisfbuPurfnd-Zs2MLfSltWDjgibPXU02csiBmfGgy1mES0icDKkgeuqiLlFz90pgV61cZcQieqAeB5k3quMN20HWC3EaYTqPyspo0ZMjc_w1WItldMFDusncH8kqak7iKfQYSISf2MlD7WSmHO77_jgePWsEYKvdQIuLOcY-PZ-dswISb1DE72hJ6ZvNEnOna7LKe67VfFgfBvar4UEtwCzMCyJt0aAUzepgvFbdA1ajUQagpeAlco5o-oZmvmbfay_xiBRxEsBa0n9KIKrxQ3ceLhx5xCFau3OH9Poa9ch2QeMI7bYe5BAkL2xQKzZdk4LNkG6hpQtucLNlCzMTOZ_HO7E4lFXe2LQlYcNy6XSSv16g1R5qjM2W4UBf_pmGGo1E4wTC0WSVlgIXdlCkVo8P1h6xfVhmOfnZWiZgVmPynR21urj1pW-M0FEbarBefU-JTvQU4Qgt25Q0]] 2021-03-31 14:17:50.498 DEBU sign:9 : POST /preheats remote:192.168.71.183:40028 cost:193.662µs err: 2021-03-31 14:18:00.502 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.646µs err: 2021-03-31 14:18:10.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:24.602µs err: 2021-03-31 14:18:20.499 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:37.729µs err: 2021-03-31 14:18:30.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:25.405µs err: 2021-03-31 14:18:40.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:23.134µs err: 2021-03-31 14:18:50.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:28.289µs err: 2021-03-31 14:19:00.500 DEBU sign:9 : GET /preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 remote:192.168.71.183:40028 cost:33.542µs err:

Can anybody let me know anything i am missing what may be the cause for the above issue

bmmadhuprasad commented 3 years ago

even i have deployed dragonfly supernode in VM directly from source code without using docker as per document and tried to use that as end point in harbor but i see same error when i created p2p preheat policy and apply to initiate image preheat to dragonfly Below is the error i see from harbor 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:93]: Preheating image 'test/mem:0.1@sha256:9871d56e046e38e9daaef6b592e67282c1e286a85e84797ed18b942a8263eadf' to the target preheat provider: dragonfly vm:http://54.215.243.61:8002 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:120]: Get preheat provider driver: dragonfly 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:134]: Check health of preheat provider instance: Healthy 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:146]: Sending preheat request is successfully done 2021-04-05T13:49:00Z [INFO] [/pkg/p2p/preheat/job.go:170]: Start to loop check the preheating status until it's success or timeout(30m) 2021-04-05T13:49:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:49:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:50:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:10Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:20Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:30Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:40Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:51:50Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check preheat progress: Task [3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735] is failed, start time=2021-04-05 21:49:00 2021-04-05T13:52:00Z [INFO] [/pkg/p2p/preheat/job.go:186]: Check p below is the logs from dragonfly supernode 1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhayenlv3u999yAnLnAgjGaYijNWv6r2IMeO6mFnI6AZSzlKH2ig3N25gPbxh1UT7PC7jd-mJEuxVd_BBMFhS2BttRbLqyJf6Bkr5T4gKOYyVjAm0tBG0zdhulrjakAt97BW6v3IFbP2cI0aL1-vk"},"type":"image","url":"https://harbor.madhu.mwpdemos.com/v2/test/mem/manifests/0.1"}, res:{"body":{"id":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735"},"headers":{},"statusCode":"OK","statusCodeValue":200} 2021-04-05 13:49:00.958 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:c1c1ba809094891c97d8ca9f34817f54c72cfab868b6914d85f2a156a6ac26a9 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.969 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.970 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:79f39971f65f5d879d2ed9e4402b5fba9a825803c4c76b8a09e4926ea58185c2 parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.973 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e72295e4b7adb87d61408b4101c0e4f8ccc15c84ccbfdbfe756e410f83b0fcaf parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.974 INFO [preheat-10] c.d.d.s.s.p.ImagePreheater - preheat layer:https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:e38d1a0197a41b27b23313a6ae0ec7f4219c31fdb55e5f5b9c43ee781237312f parentId:3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 2021-04-05 13:49:00.980 INFO [preheat-12] c.d.d.s.s.p.PreheatServiceImpl - command: /usr/local/bin/dfget -u https://harbor.madhu.mwpdemos.com/v2/test/mem/blobs/sha256:ff59de7222ecb2f8dd85ac954ca687d3b63b726d7a9631964f00535e65339584 -o /home/admin/supernode/repo/preheat/87626317-ef0f-448a-8a6e-78e60f72cc6c --callsystem dragonfly_preheat --totallimit 90M -s 90M --notbs --header Authorization:Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjdXTUE6S1ZYTzpLRjI1OjM1TkE6R0ozNTpaR0tZOlBHWVQ6STVVVDpUR1dZOlVVSTQ6UFQ3WjpYM0tXIiwidHlwIjoiSldUIn0.eyJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE2MTc2MzIzMzksImp0aSI6IkNobzlxcXFhMmFGTWJKeDMiLCJpYXQiOjE2MTc2MzA1MzksImlzcyI6ImhhcmJvci10b2tlbi1pc3N1ZXIiLCJuYmYiOjE2MTc2MzA1MzksInN1YiI6ImRpc3RyaWJ1dG9yIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L21lbSIsImFjdGlvbnMiOlsicHVsbCJdfV19.RXfGEpeMUpNYQA2v2F60IPSnSy4h_X4WKFEIlYAd0H363oVE5omj25faYccMBEU3XFPeFUWx2tGaHCdrbgnSa5SnQ5qJgerJ4FmGYCNgi1Pvr-eFjEjC8hqNAREoiZ25xsF4lg90t7-YQ_UZSxgmwMKQNRfaREseJw1emTXDLAulJSxzJVO6egEt2_jWAWsK58zyTBu40zTFGwD6fUr27vAXn2xf3KSPETf4Hf0HunDpzoZFTqMqAmHSp5w3loPuHvGF9pt9f4_blkT1JULmFFc3_zcSkuF-NDcqT2UAxN7ZQHaYJpG6uhuQmEc0bJ8fR69v1Ag_CKppOzUZbl0fGVMEBn0PDJLn67leEo5UwBn-knydMtPozfYC1fg20PxVAASo9i964ZS1jkbOoDfjpG5QKf-al5DgaCge8O6ULFo2PKPUZaip-6opnpcdG65tRAP64UZvCCYBZ8l1maecZHK2MLgQYiD5lGInfjcUWEYaENtw4FXrRltRteUiv1GYN8jBktuoAKZevLsia0ULMhaye Please help us to fix the issue and let us know anything i am missing

ansinlee commented 3 years ago

Is the image public?

bmmadhuprasad commented 3 years ago

yes project created in harbor where this image pushed is public image

bmmadhuprasad commented 3 years ago

dragonfly supernode image i am using is dragonflyoss/supernode:1.0.6,I have also tried with latest image of supernode but same issue.

jim3ma commented 3 years ago

Thanks for report this issue. I will verify it ASAP.

ansinlee commented 3 years ago

query the detail error message from dragonfly server of the task as the following command curl http://{dragonfly-server-address}/preheat/{task-id}

for example: curl http://192.168.0.180:8002/preheats/51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127

{"ID":"51e4eec494ff56f5eeb1e2dfcc96b1a65783b110e54671d45dc4b98f030b6127","finishTime":"2021-04-07T08:42:52.682Z","startTime":"2021-04-07T08:42:49.681Z","status":"FAILED","errorMsg":"Get http://xx.xx.xx.xx/v2/library/nginx/manifests/latest: dial tcp xx.xx.xx.xx:80: i/o timeout"}

bmmadhuprasad commented 3 years ago

@ansinlee curl http://13.56.114.60:8002/preheats/3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735 {"ID":"3e7f24c514a47091cf0b917bbb93eabea49feac440e52f93eb5ca9d586426735","finish Time":"2021-04-08T09:28:33.656Z","startTime":"2021-04-08T09:28:33.400Z","status" :"FAILED","errorMsg":"Get https://harbor.madhu.mwpdemos.com/v2/test/mem/manifest s/0.1: x509: certificate signed by unknown authority"}

jim3ma commented 3 years ago

@bmmadhuprasad It seems your harbor cert is not valid for dragonfly supernode.

liang-junwei commented 3 years ago

I also met the same problem, how to solve it?

huangpxs commented 3 years ago

I also met the same problem，my harbor version is v2.2.1

jim3ma commented 3 years ago

Follow https://blog.jim.plus/blog/post/jim/selfsign-ca-and-update-certficates to update ca chains.

zkking commented 2 years ago

@bmmadhuprasad Obviously, this is a bug of Dragonfly 1.x. In the file https://github.com/dragonflyoss/Dragonfly/tree/master/supernode/daemon/mgr/preheat/image_preaheater.go, the getLayers method does not support TLS to obtain image layer information.

Therefore, before fixing the problem, you need to manually set the certificate trust for the supernode container. The operation is as follows: docker exec -it supernode bash cp /certs/ca.crt /usr/local/share/ca-certificates/ update-ca-certificates docker restart supernode