img push --insecure-registry does not seem to work

[Kelvin-M]

Hello, I'm running into an issue by trying to push an image to a local registry on kubernetes

Steps to reproduce the behavior

Here what I'm doing :

1. Setup local registry

   helm install -n registry stable/docker-registry

2. Create a local pod with img docker image, which reproduce a push with insecure-registry option

   apiVersion: v1
   kind: Pod
   metadata:
   name: img-test
   spec:
   containers:
   - name: img-test
   image: r.j3ss.co/img:v0.5.9
   command: ["/bin/sh", "-c"]
   args: ["img pull busybox:1.31.1; img tag busybox:1.31.1 registry-docker-registry:5000/busybox:1.31.1; img push --insecure-registry=true registry-docker-registry:5000/busybox:1.31.1"]
   restartPolicy: Never

You can also do

apiVersion: v1
kind: Pod
metadata:
  name: img-test
spec:
  containers:
  - name: img-test
    image: r.j3ss.co/img:v0.5.9
    command: ["/bin/sh", "-c"]
    args: ["sleep 3600"]
  restartPolicy: Never

And launch the command manually :

kubectl exec img-test -- img pull busybox:1.31.1
kubectl exec img-test -- img tag busybox:1.31.1 registry-docker-registry:5000/busybox:1.31.1
kubectl exec img-test -- img push --insecure-registry=true registry-docker-registry:5000/busybox:1.31.1

Results :

Pushing registry-docker-registry:5000/busybox:1.31.1...
Error: failed to do request: Head https://registry-docker-registry:5000/v2/busybox/blobs/sha256:759344404bffa09286afe8f43530d0e6e0f59ff025ea9d671e899ceb06666537: http: server gave HTTP response to HTTPS client

The registry is accessible

kubectl exec img-test  -- wget -q -O- registry-docker-registry:5000/v2/_catalog

I tried with

 --insecure-registry=true #  but also with TRUE, True, 1
 --insecure-registry true #  does not work because it is not taken into account, true is considered as the image to push ^^
 --insecure-registry # blank

Questions:

• Am I missing something in my example that prevent it from working ?
• Is the insecure-registry option is not working at all ?

Thank you for your help !

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.62. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[Kelvin-M]

@jessfraz and @AkihiroSuda Sorry to pin you directly, but do you have any idea on this issue ?

If I look into the code : https://github.com/genuinetools/img/blob/f860bdb559a2eb840f4f62859cd4fc9106529002/push.go#L83

with c a client instance which forward the push insecure-registry value to https://github.com/genuinetools/img/blob/f860bdb559a2eb840f4f62859cd4fc9106529002/client/push.go#L37 and use moby/buildkit/util/push https://github.com/moby/buildkit/blob/214aa5dbcf68f3eb5fc66b1f5198a84c6e2505e2/util/push/push.go#L27

But insecure option does not seem to be used in the function :/

Am I missing something ?

[Kelvin-M]

I found a dirty fix to make it work. It seems that using docker.ConfigureDefaultRegistries() from https://github.com/genuinetools/img/blob/f860bdb559a2eb840f4f62859cd4fc9106529002/client/workeropt.go#L135 return RegistryHosts with https scheme see https://github.com/containerd/containerd/blob/master/remotes/docker/registry.go#L150-L157

diff --git a/client/push.go b/client/push.go
index fd0cc1a4..564122a4 100644
--- a/client/push.go
+++ b/client/push.go
@@ -3,9 +3,11 @@ package client
 import (
    "context"
    "fmt"
+   "net/http"

    "github.com/docker/distribution/reference"
    "github.com/moby/buildkit/util/push"
+   "github.com/containerd/containerd/remotes/docker"
 )

 // Push sends an image to a remote registry.
@@ -34,5 +36,35 @@ func (c *Client) Push(ctx context.Context, image string, insecure bool) error {
    if err != nil {
        return err
    }
-   return push.Push(ctx, sm, opt.ContentStore, imgObj.Target.Digest, image, insecure, opt.RegistryHosts, false)
+
+   registriesHosts := opt.RegistryHosts
+   if insecure {
+       registriesHosts = configurePushRegistries("http")
+   }
+
+
+   return push.Push(ctx, sm, opt.ContentStore, imgObj.Target.Digest, image, insecure, registriesHosts, false)
+}
+
+func configurePushRegistries(scheme string) docker.RegistryHosts {
+   return func(host string) ([]docker.RegistryHost, error) {
+           config := docker.RegistryHost{
+               Client:       http.DefaultClient,
+               Authorizer:   nil,
+               Host:         host,
+               Scheme:       scheme,
+               Path:         "/v2",
+               Capabilities: docker.HostCapabilityPull | docker.HostCapabilityResolve | docker.HostCapabilityPush,
+           }
+
+           if config.Client == nil {
+               config.Client = http.DefaultClient
+           }
+
+           if host == "docker.io" {
+               config.Host = "registry-1.docker.io"
+           }
+
+           return []docker.RegistryHost{config}, nil
+       }
 }

[gavan1]

@Kelvin-M any solutions for this? I am trying to push to a local bare metal microk8s still getting the

http: server gave HTTP response to HTTPS client

error.

[Kelvin-M]

@Kelvin-M any solutions for this? I am trying to push to a local bare metal microk8s still getting the

http: server gave HTTP response to HTTPS client

error.

I've just done a "dirty" patch (https://github.com/genuinetools/img/issues/297#issuecomment-660120365) to test during a benchmark between image builder in a k8s context, but I did not spend more time on this issue :)

[Constantin07]

any plans to merge the patch with the fix ?

[Kelvin-M]

Close as I did not use img anymore in my project !

[keypointt]

I have the same error as I filed in https://github.com/kubernetes-sigs/kind/issues/2604

Seems Kelvin's solution is:

Close as I did not use img anymore in my project !

lol

[Kelvin-M]

I have the same error as I filed in kubernetes-sigs/kind#2604

Seems Kelvin's solution is:

Close as I did not use img anymore in my project !

lol

Yep, we were benchmarking different solutions to build and push images inside a k8s cluster into a local registry. You can use Docker in Docker (Dind) or kaniko (https://github.com/GoogleContainerTools/kaniko). We also tried Mmakisu (https://github.com/uber-archive/makisu) but it's not maintained anymore. Because of this issue in img (bypassed by my "dirty" patch) we prefer to use kaniko