ECR repo configuration requires username password

[Hraesvelg]

Hi every one !

All is in the title, for more detail :

[go] The plugin sent a response that could not be understood by Go. Plugin returned with code '500' and the following response: 'Failed to publish Artifact[id=my-registry, storeId=my-registry, artifactPlanConfig={"Image":"test-nginx","Tag":"latest"}]: java.lang.RuntimeException: unauthorized: incorrect username or password'

RegistryType : ecr RegistryURL : (Not specified) RegistryID : 12345678912 AWSAccessKeyId : AWSSecretAccessKey: AWSRegion : eu-west-1 Username : (Not specified) Password : (Not specified)

[chadlwilson]

I think you can put a bit more effort into explaining what you did and what isn't working.

What's in the plugin specific log file? How is your EC$ configured?

[Hraesvelg]

Hi @chadlwilson thanks for your help !

in server log nothing particular... :

/var/log/go-server# ll
total 320
drwxr-x---  2 go   go      4096 Jul  4 11:05 ./
drwxrwxr-x 11 root syslog  4096 Jul  1 00:00 ../
-rw-r--r--  1 go   go      2038 Jul  3 21:00 go-server-wrapper.log
**-rw-r--r--  1 go   go      4360 Jul  4 11:06 go-server.log**
-rw-r--r--  1 go   go      5018 Jul  3 21:00 plugin-cd.go.artifact.docker.registry.log
-rw-r--r--  1 go   go      4967 Jul  3 21:00 plugin-cd.go.authentication.ldap.log
-rw-r--r--  1 go   go      3630 Jul  4 10:36 plugin-cd.go.authentication.passwordfile.log
-rw-r--r--  1 go   go     32448 Jun 30 21:47 plugin-yaml.config.plugin.log

The error mentioned in plugin-cd.go.artifact.docker.registry.log is too old, not linked here...

in agent log :

2024-07-04 11:07:18,953 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Using docker registry server ''.
2024-07-04 11:07:18,978 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Pushing docker image `test-nginx:latest` to docker registry ''.
2024-07-04 11:07:19,854 ERROR [pool-13-thread-1] DockerRegistryArtifactPlugin:122 - Failure: unauthorized: incorrect username or password
2024-07-04 11:07:19,856 ERROR [scheduler-1] DockerRegistryArtifactPlugin:127 - Failed to publish Artifact[id=amazon-registry, storeId=amazon-registry, artifactPlanConfig={"Image":"test-nginx","Tag":"latest"}]: java.lang.RuntimeException: unauthorized: incorrect username or password
com.spotify.docker.client.exceptions.DockerException: java.lang.RuntimeException: unauthorized: incorrect username or password
        at com.spotify.docker.client.DefaultDockerClient.tailResponse(DefaultDockerClient.java:2773)
        at com.spotify.docker.client.DefaultDockerClient.requestAndTail(DefaultDockerClient.java:2802)
        at com.spotify.docker.client.DefaultDockerClient.push(DefaultDockerClient.java:1391)
        at com.spotify.docker.client.DefaultDockerClient.push(DefaultDockerClient.java:1374)
        at cd.go.artifact.docker.registry.executors.PublishArtifactExecutor.execute(PublishArtifactExecutor.java:66)
        at cd.go.artifact.docker.registry.DockerRegistryArtifactPlugin.handle(DockerRegistryArtifactPlugin.java:79)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:204)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:163)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
        at com.thoughtworks.go.plugin.access.artifact.ArtifactExtension.publishArtifact(ArtifactExtension.java:135)
        at com.thoughtworks.go.remote.work.artifact.ArtifactsPublisher.publishPluggableArtifact(ArtifactsPublisher.java:128)
        at com.thoughtworks.go.remote.work.artifact.ArtifactsPublisher.publishPluggableArtifacts(ArtifactsPublisher.java:105)
        at com.thoughtworks.go.remote.work.artifact.ArtifactsPublisher.publishArtifacts(ArtifactsPublisher.java:63)
        at com.thoughtworks.go.remote.work.BuildWork.completeJob(BuildWork.java:208)
        at com.thoughtworks.go.remote.work.BuildWork.build(BuildWork.java:142)
        at com.thoughtworks.go.remote.work.BuildWork.doWork(BuildWork.java:92)
        at com.thoughtworks.go.agent.JobRunner.run(JobRunner.java:50)
        at com.thoughtworks.go.agent.AgentHTTPClientController.doWork(AgentHTTPClientController.java:136)
        at com.thoughtworks.go.agent.AgentHTTPClientController.tryDoWork(AgentHTTPClientController.java:115)
        at com.thoughtworks.go.agent.AgentController.performWork(AgentController.java:89)
        at com.thoughtworks.go.agent.AgentWorkRetrievalScheduler.run(AgentWorkRetrievalScheduler.java:60)
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.lang.RuntimeException: unauthorized: incorrect username or password
        at cd.go.artifact.docker.registry.DockerProgressHandler.progress(DockerProgressHandler.java:39)
        at com.spotify.docker.client.ProgressStream.tail(ProgressStream.java:77)
        at com.spotify.docker.client.DefaultDockerClient$ResponseTailReader.call(DefaultDockerClient.java:2754)
        at com.spotify.docker.client.DefaultDockerClient$ResponseTailReader.call(DefaultDockerClient.java:2738)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        ... 3 common frames omitted

about my configuration what can i tell... hum ... the endpoint to amazonaws is more like : 12345678912.dkr.ecr.eu-west-1.amazonaws.com/test-app may be here ? dkr.ecr ... ?

also my pipeline config :

artifacts:
    - external:
        id: amazon-registry
        store_id: amazon-registry
        configuration:
            options:
                Image: "test-nginx"
                Tag: "latest"

[chadlwilson]

First, make sure you are on the latest version of the plugin, and let me know your GoCD version.

The plugin does the equivalent of what is documented at https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html#registry_auth_http to get a token to be used with HTTP authentication. You could check yourself that you can correctly use your AWS access key to talk to the ECR endpoint using HTTP authentication.

However after quickly scanning through the code, I am not convinced the plugin works correctly when configured for ECR when determining the registry URL. This may sound a bit weird, but can you try switching to "others", entering your registry URL in the field there (leave username/pass blank), then switch back to ECR and save? And then try with that?

[Hraesvelg]

seems better in log, now, he see the endpoint :

2024-07-04 17:02:43,476 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Using docker registry server **12345678912.dkr.ecr.eu-west-1.amazonaws.com**.
2024-07-04 17:02:43,496 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Pushing docker image `test-nginx:latest` to docker registry 12345678912.dkr.ecr.eu-west-1.amazonaws.com.
2024-07-04 17:02:44,357 ERROR [pool-22-thread-1] DockerRegistryArtifactPlugin:122 - Failure: unauthorized: incorrect username or password

i also checked my AK/SK and should be good :

aws ecr get-login-password

-> eyJwYXlsb2FkIjoiZzZRL3NIZUtwRzg2....

aws ecr get-authorization-token

{
    "authorizationData": [
        {
            "authorizationToken": "QVdTOmV5SndZWGxzYjJGa0l....

Note :

Artifact plugin for docker Version 1.4.0-719 GoCD Version: 24.2.0 (19076-1406870fc6e121194028e55c4facc0c638d70007).

[chadlwilson]

I think you may be missing the /test-app from your registry, but I can't recall off the top of my head the norms for ECR.

After getting the token, you could try the curl command suggested by the ECR docs to the registry endpoint via the command line to validate the credentials work all together?

[Hraesvelg]

the curl work perfectly but build still failed :

curl -i -H "Authorization: Basic $TOKEN" https://12345678912.dkr.ecr.eu-west-1.amazonaws.com/v2/test-app/tags/list

HTTP/1.1 200 OK Docker-Distribution-Api-Version: registry/2.0 Sizes: Date: Fri, 05 Jul 2024 11:22:04 GMT Content-Length: 35 Content-Type: text/plain; charset=utf-8

{"name":"test-app","tags":[]}

[cd.go.artifact.docker.registry] Pushing docker image test-app:latest to docker registry https://12345678912.dkr.ecr.eu-west-1.amazonaws.com/test-app/. [cd.go.artifact.docker.registry] unauthorized: incorrect username or password

[chadlwilson]

What's happens if the URL matches exactly? With the V2?

Anyway, I honestly don't know. Clearly this plugin has some problems with ECR and itd need some detailed debugging to figure out what's going on.

Its not a plugin that is commonly used with normal docker registries either so it probably hasn't had much love.

If you're not able to go deeper to debug the problem it might be worth stepping back and thinking about whether you need to use a docker artifact plugin for your use case, rather than just using basic docker or aws CLI commands to pull and push mages.

[Hraesvelg]

Very sad that plugin is not maintained ... I touch a start of solution :

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 12345678912.dkr.ecr.eu-west-1.amazonaws.com

AWS seems a common username, i don't know why but ok and for the password, it is the "token" from get-login-password

-> WARNING! Your password will be stored unencrypted in /home/debian/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

Do you think that someone will fix this point ?

[chadlwilson]

Unfortunately the above doesn't tell us the problem of or confirm the command line is doing the same thing as the plugin.

As I said, it needs more detailed debugging by turning on plugin debug logging, digging into the Spotify docker client behaviour and making sure it is sending the right thing to the right endpoint.

I don't have time/energy to do this as I've never heard from anyone using this plugin with ECR, don't have an existing AWS account with ECR available etc, so doesn't seem worth my effort. I am happy to merge PRs if people get to the bottom of a problem and have an idea of how to fix it or make it easy for me to replicate.

But I'm personally not going to do first principles investigations for cloud provider integrations as that costs time, energy and money and funding that I don't have. People who work for companies that use these tools (for free) do tend to have those resources, I need for them to use them 😃

[Hraesvelg]

ok i followed instructions here to activate debug log (on server and agent...) : https://github.com/gocd/docker-registry-artifact-plugin?tab=readme-ov-file#enable-debug-logs

i didn't see big effect, i am waiting to see "DEBUG" message in log but except this error, nothing more :

2024-07-06 10:02:49,323 INFO  [goPluginLocationMonitor] DockerRegistryArtifactPlugin:72 - Loading plugin Artifact plugin for docker[null].
2024-07-06 10:02:50,590 ERROR [goPluginLocationMonitor] DockerRegistryArtifactPlugin:127 - Error while executing request go.plugin-settings.get-configuration
java.lang.NullPointerException: Cannot invoke "cd.go.artifact.docker.registry.Request.ordinal()" because the return value of "cd.go.artifact.docker.registry.Request.fromString(String)" is null
        at cd.go.artifact.docker.registry.DockerRegistryArtifactPlugin.handle(DockerRegistryArtifactPlugin.java:55)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:204)
        at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:163)
        at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
        at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
        at com.thoughtworks.go.plugin.access.common.AbstractExtension.getPluginSettingsConfiguration(AbstractExtension.java:56)
        at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.lambda$findSettingsAndViewOfAllExtensionsIn$2(PluginSettingsMetadataLoader.java:84)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.findSettingsAndViewOfAllExtensionsIn(PluginSettingsMetadataLoader.java:89)
        at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.fetchPluginSettingsMetaData(PluginSettingsMetadataLoader.java:59)
        at com.thoughtworks.go.plugin.access.common.settings.PluginSettingsMetadataLoader.pluginLoaded(PluginSettingsMetadataLoader.java:49)
        at com.thoughtworks.go.plugin.infra.PluginLoader.lambda$doPostBundleInstallActivities$0(PluginLoader.java:112)
        at org.apache.commons.collections4.IteratorUtils.forEach(IteratorUtils.java:1224)
        at org.apache.commons.collections4.IterableUtils.forEach(IterableUtils.java:585)
        at com.thoughtworks.go.plugin.infra.PluginLoader.lambda$doPostBundleInstallActivities$1(PluginLoader.java:112)
        at org.apache.commons.collections4.IteratorUtils.forEach(IteratorUtils.java:1224)
        at org.apache.commons.collections4.IterableUtils.forEach(IterableUtils.java:585)
        at com.thoughtworks.go.plugin.infra.PluginLoader.doPostBundleInstallActivities(PluginLoader.java:112)
        at com.thoughtworks.go.plugin.infra.PluginLoader.loadPlugin(PluginLoader.java:70)
        at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.refreshBundle(DefaultPluginJarChangeListener.java:180)
        at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.addPlugin(DefaultPluginJarChangeListener.java:123)
        at com.thoughtworks.go.plugin.infra.listeners.DefaultPluginJarChangeListener.pluginJarAdded(DefaultPluginJarChangeListener.java:72)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.lambda$pluginJarAdded$0(DefaultPluginJarLocationMonitor.java:237)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.doOnAllPluginJarChangeListener(DefaultPluginJarLocationMonitor.java:258)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread$DoOnAllListeners.pluginJarAdded(DefaultPluginJarLocationMonitor.java:237)
        at com.thoughtworks.go.plugin.infra.monitor.PluginChangeNotifier.lambda$notify$0(PluginChangeNotifier.java:33)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at com.thoughtworks.go.plugin.infra.monitor.PluginChangeNotifier.notify(PluginChangeNotifier.java:30)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.loadAndNotifyPluginsFrom(DefaultPluginJarLocationMonitor.java:204)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.oneShot(DefaultPluginJarLocationMonitor.java:191)
        at com.thoughtworks.go.plugin.infra.monitor.DefaultPluginJarLocationMonitor$PluginLocationMonitorThread.run(DefaultPluginJarLocationMonitor.java:177)

[chadlwilson]

Youd need to look in the plugin specific log file plugin-cd.go.artifact.docker.registry.log for the debug logs, not the server log. But if it's still not working might be doing something wrong.

[Hraesvelg]

i confirm that debug mode is well enabled :

but nothing more in log always the same error :

2024-07-06 11:35:17,767 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Using docker registry server https://12345678912.dkr.ecr.eu-west-1.amazonaws.com.
2024-07-06 11:35:17,791 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Pushing docker image test-nginx:latest to docker registry https://12345678912.dkr.ecr.eu-west-1.amazonaws.com.
2024-07-06 11:35:18,370 ERROR [pool-7-thread-1] DockerRegistryArtifactPlugin:122 - Failure: unauthorized: incorrect username or password
2024-07-06 11:35:18,371 ERROR [scheduler-1] DockerRegistryArtifactPlugin:127 - Failed to publish Artifact[id=amazon-registry, storeId=amazon-registry, artifactPlanConfig={"Image":"test-nginx","Tag":"latest"}]: java.lang.RuntimeException: unauthorized: incorrect username or password
com.spotify.docker.client.exceptions.DockerException: java.lang.RuntimeException: unauthorized: incorrect username or password
        at com.spotify.docker.client.DefaultDockerClient.tailResponse(DefaultDockerClient.java:2773)

i am not java developer but...

is it here where plugin handle authentication https://github.com/gocd/docker-registry-artifact-plugin/blob/master/src/main/java/cd/go/artifact/docker/registry/RegistryAuthSupplierChain.java ?

may be line 33 / 34 the content of payload is not like the plugin wait.. i mean a list ["username", "password"]

it will be interesting to have here a debug message to see value in username and password !

[chadlwilson]

That still doesn't look like it's from the correct log file (plugin-cd.go.artifact.docker.registry.log) to me.

[Hraesvelg]

yes i ensure you ^^

root@fobar:/var/log/go-agent# cat plugin-cd.go.artifact.docker.registry.log

2024-07-06 09:45:21,085 INFO  [main] DockerRegistryArtifactPlugin:72 - Loading plugin Artifact plugin for docker[null].
2024-07-06 09:45:46,706 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Using docker registry server `https://12345678912.dkr.ecr.eu-west-1.amazonaws.com`.
2024-07-06 09:45:47,316 INFO  [scheduler-1] DockerRegistryArtifactPlugin:72 - Pushing docker image `test-nginx:latest` to docker registry `https://12345678912.dkr.ecr.eu-west-1.amazonaws.com`.
2024-07-06 09:45:48,045 ERROR [pool-4-thread-1] DockerRegistryArtifactPlugin:122 - Failure: unauthorized: incorrect username or password
2024-07-06 09:45:48,052 ERROR [scheduler-1] DockerRegistryArtifactPlugin:127 - Failed to publish Artifact[id=amazon-registry, storeId=amazon-registry, artifactPlanConfig={"Image":"test-nginx","Tag":"latest"}]: java.lang.RuntimeException: unauthorized: incorrect username or password
com.spotify.docker.client.exceptions.DockerException: java.lang.RuntimeException: unauthorized: incorrect username or password
        at com.spotify.docker.client.DefaultDockerClient.tailResponse(DefaultDockerClient.java:2773)
        at com.spotify.docker.client.DefaultDockerClient.requestAndTail(DefaultDockerClient.java:2802)
        at com.spotify.docker.client.DefaultDockerClient.push(DefaultDockerClient.java:1391)
...

i am ready to open a new ticket for bug with log file / debug mode not working... if you want ;)

Edit : i add a suggestion, if possible, to provide me a jar with log.INFO everywhere it's necessary i am ready to test ... i will do myself but as i said i am not java developer i don''t know how to compile your extension...