I can not pull images from private registry

[smiklosovic]

I have image in private registry, I delete that image locally so I expect that Cube will download it. I get this exception:

Caused by:
com.github.dockerjava.api.NotFoundException: Error parsing HTTP response: invalid character 'p' after top-level value: "404 page not found\n"
at com.github.dockerjava.core.util.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:48)
at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:134)
at org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:123)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:171)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:251)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:667)
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:664)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:228)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:424)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:664)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:424)
at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:333)
at com.github.dockerjava.jaxrs.PullImageCmdExec.execute(PullImageCmdExec.java:36)
at com.github.dockerjava.jaxrs.PullImageCmdExec.execute(PullImageCmdExec.java:17)
at com.github.dockerjava.jaxrs.AbstrDockerCmdExec.exec(AbstrDockerCmdExec.java:57)
at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:29)
at com.github.dockerjava.core.command.PullImageCmdImpl.exec(PullImageCmdImpl.java:15)
at org.arquillian.cube.docker.impl.docker.DockerClientExecutor.pullImage(DockerClientExecutor.java:550)
at org.arquillian.cube.docker.impl.docker.DockerClientExecutor.createContainer(DockerClientExecutor.java:331)
at org.arquillian.cube.docker.impl.model.DockerCube.create(DockerCube.java:74)
... 24 more

In arquillian.xml I have in dockerContainers this line:

image: docker.some.company:5000/some/image:x.y.z

[smiklosovic]

This is my workflow:

1) docker tag docker.some.company:5000/some/images:x.y.z 2) docker push docker.some.company:5000/some/images:x.y.z

When I remove that image locally by docker rmi , Cube is not able to pull it, even I have correct dockerRegistry set in arquillian.xml (to http://docker.some.company:5000) and username and password set.

I can login to docker registry by

docker login -p password -u user -e docker@some.company docker.some.company:5000

[aslakknutsen]

@smiklosovic What happens if you remove the image prefix, and just set the registry host in arquillian.xml ?

[lordofthejars]

@smiklosovic you are the first one to try this feature. We are using directly the Docker-Java feature but I think that wrong because we are setting username and password but not the serverAddress property: https://github.com/docker-java/docker-java#configuration

Can you try to set this property docker.io.serverAddress as system property and check if it works?

[smiklosovic]

I was not able to pull it neither way ... I dont know what is the problem. I can pull them normally without problem.

[smiklosovic]

docker's tag documentation says that I have to tag it like

docker tag container_hash repository:port/some/image:tag

after that I am going to push it - if I left repository:port prefix, it would push it to docker.io instead of a private registry. So I do:

docker push repository:port/some/image:tag

which starts to push it there and once it is done, I delete that with

docker rmi -f repository:port/some/image:tag

Now I expect that when I put it to image: in arquillian.xml - with or without server prefix, it would download it but when I add there that repository prefix, it results in above exception. If I omit that prefix, it says that it does not exist.

GET on that endpoint in registry says that some/image:tag exists.

I set that serverAddress but it has not changed a thing ...

[smiklosovic]

@aslakknutsen @lordofthejars

Maybe I have something ... I wiresharked it and when that Docker client do POST to docker daemon that it should create a container with some name, after that daemon responses that it does not exist (404 not found) so client POST that daemon should create that image and client posts it like this:

/v1.21/images/create?tag=5000/some/image:1.0.0&fromImage=docker.some.company&registry=http://docker.some.company:5000

And daemon responses that it does not exist (404 not found)

But when you notice, that tag is very weird, it looks like it took all after the first semicolon in "image" string. So when I do it without that all docker.some.company:5000 - image: is only some/image:1.0 I get that it does not exist - which is weird. Maybe the problem is that "fromImage" SHOULD include that whole path because I have tagged it like that.

[lordofthejars]

@smiklosovic can you try to use docker-java alone to reproduce the problem? In this way we would be able to isolate the problem.

[lordofthejars]

I have checked and yes we are setting private repo in pull operation: https://github.com/arquillian/arquillian-cube/blob/b0b33b307b260eafb97be6331ac4faca26248c89/docker/docker/src/main/java/org/arquillian/cube/docker/impl/docker/DockerClientExecutor.java#L539

[smiklosovic]

This is my test with pure docker client which downloads it without problems

https://gist.github.com/smiklosovic/0addd29979b5dae906bb

I can see in the output that it downloaded new image locally.

I think you have it wrong somewhere along these lines:

https://github.com/arquillian/arquillian-cube/blob/b0b33b307b260eafb97be6331ac4faca26248c89/docker/docker/src/main/java/org/arquillian/cube/docker/impl/docker/DockerClientExecutor.java#L535-L551

[smiklosovic]

You do not basically count with the case when imageName has there two semicolons because this line (1) will parse tag from this imageName "docker.some.company:5000/some/image:latest" as "5000/some/image:latest" which is obviously wrong.

Everything which goes to pullImageCmd(argument) should go there WITHOUT tag, that tag will be parsed afterwards and it will be put here in withTag method on that command.

(1) https://github.com/arquillian/arquillian-cube/blob/b0b33b307b260eafb97be6331ac4faca26248c89/docker/docker/src/main/java/org/arquillian/cube/docker/impl/docker/DockerClientExecutor.java#L543

[smiklosovic]

(i closed it by accident)

[lordofthejars]

true, so you mean that lastIndexOf would be the best one right? Does this fix everything?

[smiklosovic]

Yeah plus, as I said above, you should not pass string into pullImageCmd() with tag included. Argument of that method is named "repository", that is not "image with tag" but "repository". In Docker case and in this, "repository" is docker.some.company:5000/some/image. I believe in case you do not have custom registry, it would be "some/image", again without the tag.

[lordofthejars]

ok, we have then some problems because yo might have tag and not host or tag and host, ... so we will need to parse/play with indexOf('/')

[smiklosovic]

yeah, parse it according to how many semicolons you got

[lordofthejars]

well not the number of semicolons but the position of them. Ok I am going to talk with @aslakknutsen to stop the release of next version :)

[smiklosovic]

please could you handle that docker.io.serverAddress if it is still applicable as well?

[smiklosovic]

I think you will ever get only one or two of them. You can not use semicolon in tag nor in image name. The only use cases of semicolon are 1) repository/image and tag separator 2) port in registry address.

[lordofthejars]

if you don't have a tag and you push to a server running on 80 port then you have 0 right? Or this is not a real scenario?

[smiklosovic]

ha :D yes ... or zero :)

[lordofthejars]

https://github.com/arquillian/arquillian-cube/pull/264

[smiklosovic]

Hi @lordofthejars

I tried what is merged now and it does not work. I will investigate further if I am pulling it from right Docker registry to double check it.

It timeouts and writes this:

    Caused by:
    com.github.dockerjava.api.DockerClientException: Could not pull image: Error: image company/service:1.0.0 not found
        at com.github.dockerjava.core.command.PullImageResultCallback.awaitSuccess(PullImageResultCallback.java:49)
        at org.arquillian.cube.docker.impl.docker.DockerClientExecutor.pullImage(DockerClientExecutor.java:551)
        at org.arquillian.cube.docker.impl.docker.DockerClientExecutor.createContainer(DockerClientExecutor.java:338)
        at org.arquillian.cube.docker.impl.model.DockerCube.create(DockerCube.java:84)

Maybe I need to setup that server property as you mentioned. Can not be that done in the code and handled by Cube already?

I dunno why it says that such image can not be pulled. It seems to me that it cut the docker.some.company:5000 string from the name.

[smiklosovic]

I have here some weird network timeouts and unstable internet ... I will keep you posted.

[lordofthejars]

Yes please, take a look at the code because IIRC I am only getting image and tag, only two strings El 25/1/2016 7:58 p. m., "Štefan Miklošovič" notifications@github.com escribió:

I have here some weird network timeouts and unstable internet ... I will keep you posted.

— Reply to this email directly or view it on GitHub https://github.com/arquillian/arquillian-cube/issues/257#issuecomment-174621445 .

[smiklosovic]

URLs seem to be ok and it parses it correctly. The thing is that requests from JUnit test I posted and from Cube differ.

In JUnit test case, the request has X-Registry-Auth header and Cube sends it without it. On registry side, I can clearly see that the request with that header from test pulls that image without any problems and authentization passes.

On the other hand, I see that Cube does not attach that header to the request and from the registry point of view, client fails to authenticate because of that.

Could you somehow add there that header with (hashed) value as well? Why you do not set set it already?

[smiklosovic]

I think it boils down to the docker logging thing (1)

(1) http://stackoverflow.com/questions/25560333/what-is-content-of-x-registry-auth-for-docker-push-to-private-registry

I can login from the console to private registry without problems. It seems to me that Cube kind of does not pick this as well.

Or maybe I am just missing something?

[smiklosovic]

Oh man this was ordeal ...

I did it. The issue was that I have not set email property and without that - X-Registry-Auth was not added as a header. This is super weird but nevermind ...

So it works :)

:+1: :+1: :+1: :clap: :clap: :clap:

[lordofthejars]

yes, you need to set all the parameters or it doesn't work hehehe, Cool that it worked.

[aslakknutsen]

Is there something to detect in configuraiton here and possible log a warning to the user? if != docker.hub and email == null? "you mgiht want to configure credentials" ?

[smiklosovic]

that would definitely help in debugging. I would check if whole username-password-email triple is set. I can imagine that a lot of people would stop to try to set it as my frustration level was quite high yesterday :D (but motivation even bigger)

[lordofthejars]

Yes check your investigations and if you need the triplet, then let's add an error log line