Using JIB to push images works, but images don't show up in the UI

[markusheiden]

Description

When I push via JIB (e.g. mvn jib:build) to our docker-ce registry, the image shows as successfully pushed to the registry. But Portus does not show it. If I try pulling it, I do get the correct image, which means the registry got the image.

Pulling it via docker and pushing it via docker make it appear in Portus.

An analysis of a colleague showed that the events of the registry are in a slightly different order (sorry, but I don't know the details) when pushing with JIB instead of the docker client.

Steps to reproduce

1. Build and push image with JIB.
2. Take a look in Portus.

• Expected behavior: Image is shown in Portus.
• Actual behavior: Image is not visible in Portus.

Sorry, but the other requested information is not available for me.

[Jean-Baptiste-Lasselle]

hi @markusheiden I encountered same problem, and I'm pretty sure where your problem is : It's in the backgound

Indeed, as you will find in the portus awful documentation (I mean it, it's awful, awfully inconsistent yet) :

the background process is required for a production setup.

Why required? I'll tell you why: because if it isn't there, well you will miss something.

I encountered the same problem : the Web UI would never show any images in the namespace listing page, though I did sucesfully pushed several images to the registry. What is really important to understand, is that :

• the background container in the docker-compose.yml Im' sure you know about, receives the webhook HTTP request sent by the docker registry to send notifications about regsitry events , like "Oh a [docker push] just completed successfully, you can display it in your web ui now" .
• without the background, I surely verified that I stil can sucessfully push new images to the docker registry, but the Portus WebUI Portus at https://portus.mycompany.io:3000/ won't ever display it...

What happened for em is that I messed up with the background container configuration, so it just stopped its execution at runtime, and therefore...byebye UI. So OK. I thought then that I could definitely check that webhooks are important ONLY to the background container, the portus container does not need to receive them (actually he already discards them, if I believe its logs in my current environment), with the following experience :

• I create a new env. variable export PORTUS_BACKGROUND_SERVICE_FQDN=portus-backservice.mycompany.io
• Then I change the configuration of the notifications for the private docker registry, in the docker-compose.yml, so that it becomes :

      # REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus-oci-registry.crt
      REGISTRY_HTTP_TLS_KEY: /secrets/portus-oci-registry.key
      # Portus endpoint
      # So here https://${PORTUS_BACKGROUND_SERVICE_FQDN}/v2/webhooks/events  should point to portus background service
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://${PORTUS_BACKGROUND_SERVICE_FQDN}:3000/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s

• Instead of :

      # REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
      REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus-oci-registry.crt
      REGISTRY_HTTP_TLS_KEY: /secrets/portus-oci-registry.key
      # Portus endpoint
      # So here https://${PORTUS_SERVICE_FQDN}/v2/webhooks/events  should point to portus
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://${PORTUS_SERVICE_FQDN}:3000/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s

I'll do those tests tomorrow, n give u the answer if you ask.

All in all, I would be glad to find out if this helped, while I'm pretty sure here you just got rid of the background container, or mis-configured it.

JB

[dleborgne]

We have the same problem here using Docker Registry 2.7.1 and Portus 2.4.3. Everything works as expected with the Docker 19.03 client but when Jib 2.0.0 client sends the manifest PUT request, the Portus backgound container log contains a "Cannot find user" message, which is probably caused by the empty "actor" field in the JSON message received by the background Portus container.

Using Docker client, the push event sent by Docker Registry to Portus, with non-empty "actor" field

Handling 'push' event:
{
  "id": "e4f8ba20-c7e1-4916-919c-a90803854ccc",
  "timestamp": "2020-02-04T11:00:56.708989056Z",
  "action": "push",
  "target": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "size": 1785,
    "digest": "sha256:49433a8241d5e44ca711191b978055f1d2ab5ccc862acf7c4d8d73c274342e91",
    "length": 1785,
    "repository": "testjib/hello-world",
    "url": "https://registry/v2/testjib/hello-world/manifests/sha256:49433a8241d5e44ca711191b978055f1d2ab5ccc862acf7c4d8d73c274342e91",
    "tag": "latest"
  },
  "request": {
    "id": "b1579a99-7f2b-454a-aeb3-f1bf611b320f",
    "addr": "10.255.0.2",
    "host": "registry",
    "method": "PUT",
    "useragent": "docker/19.03.4 go/go1.12.10 git-commit/9013bf5 kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 \\(windows\\))"
  },
  "actor": {
    "name": "testjib"
  },
  "source": {
    "addr": "0564da668715:5000",
    "instanceID": "4396b935-e643-4384-8526-2e85839e5427"
  }
}

Using Jib client, the push event sent by Docker Registry to Portus, with empty "actor" field

{
  "id": "412c19bc-fe91-4e81-a735-b2ac76f97bf8",
  "timestamp": "2020-02-04T10:46:20.116168774Z",
  "action": "push",
  "target": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "size": 1403,
    "digest": "sha256:7d13e143448e78c7f81c75aeda012513b7f898bfb5f8cc2a6541a13c5d831a16",
    "length": 1403,
    "repository": "testjib/hello-world",
    "url": "https://registry/v2/testjib/hello-world/manifests/sha256:7d13e143448e78c7f81c75aeda012513b7f898bfb5f8cc2a6541a13c5d831a16",
    "tag": "latest"
  },
  "request": {
    "id": "aba2e75f-68fd-4f69-ba85-8a11cd10f828",
    "addr": "10.255.0.2",
    "host": "registry",
    "method": "PUT",
    "useragent": "jib 2.0.0 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)"
  },
  "actor": {
  },
  "source": {
    "addr": "0564da668715:5000",
    "instanceID": "4396b935-e643-4384-8526-2e85839e5427"
  }
}
Cannot find user

[Jean-Baptiste-Lasselle]

hi @markusheiden @dleborgne , I today add a comment to update you with my last test results, which are :

• in my docker-compose.yml, I have 3 different network names / FQDN for portus, background, and registry (also therefore, I now have 3 different SSL/TLS Certificates)
• with this setup Igive you infos about, Ihave a runing portus, I can docker push and new images are updated in the Portus Web UI
• My main result I wanna give you here, is that I made sure, that it is the portus service, which expects and must receive webhooks from the registry service, which I configured like this :

      # Portus endpoint
      # So here https://${PORTUS_SERVICE_FQDN}/v2/webhooks/events  should point to portus
      REGISTRY_NOTIFICATIONS_ENDPOINTS: >
        - name: portus
          url: https://${PORTUS_SERVICE_FQDN}:3000/v2/webhooks/events
          timeout: 2000ms
          threshold: 5
          backoff: 1s

Does it help ? Did you already solved the problem ?

The only thing to help, that cross my mind, is that I think very likely the problem is unrelated with the maven jib plugin

For more details on the mechanism there's also https://github.com/SUSE/Portus/issues/2275#issuecomment-586648423 I recently wrote.

[dleborgne]

As I wrote before, the issue only happens when using Jib. Images pushed with the docker client are updated in the Portus web UI. I assume everything is well configured regarding the REGISTRY_NOTIFICATIONS_ENDPOINTS registry environment variable.

Today, after some digging in the Portus documentation and Github issues, I changed the background sync strategy to update using the PORTUS_BACKGROUND_SYNC_STRATEGY environment variable, and... all the images pushed with Jib appeared in the UI. But their author is "portus" instead of the real authenticated user that pushed them to the registry, so this looks more like a workaround than an solution to the Jib/Registry/Portus issue 😢

[Jean-Baptiste-Lasselle]

Hi @dleborgne answering quite a few weeks later :

• thank you so much for this digging more into the issue.
• So alright, I note those infos, and will feed back next time I have time to work on Portus. That might takes weeks, since I am currently involed in a new project on a professionnal side.

Never the less, I wan to tell you that I am almost sure I have seen people having a very similar problem, related to GUNs and notary : the guys were complaining the webhooks received from notary, were rejected because portus does not support GUNs in its webhooks endpoint.

Never the less, reading that issue about notary, I wanted to test the webhook, and see for myself : does portus receives from the registry, the info of the pusher id ? Answer is yes, see https://github.com/SUSE/Portus/issues/2293#issuecomment-612449096

So Okay, I'm knackered, but could not resist, I know how frustating your team feels, being in a team myself, n after succeeding , now you get new problem... . So I worked a little , and wrote a test, you can run against your private portus infra :

• assuming you used portus WebUI to create a user named obelix : use the WebUI to create a portus token, and docker login as obelix, and provide your token wheln prompted for password by the docker client.
• docker push any image
• now, in my personal infra, if I use docker client, I have the obelix mentioned as pusher : which means the webhook payload, from registry, to portus, does convey the info of the portus username of the pusher.
• Good, so first test is : I think yes, since you stated "issue happens only using Jib", but do you confirm that push with docker client, using portus token auth, you do get in portus webui, the username of the pusher, and not portus ?
• Can you run this test against your private portus ? (you have to modify some Env var. values, obvious things only)
• Now, I have a question :
  • You, your developer colleagues, and your jenkins, do you all use a docker credential helper, for authenticating to registry through portus ? Or do you use ~/.docker/config.json, or specific credentials plugin config like the following :

<configuration>
  ...
  <from>
    <image>aws_account_id.dkr.ecr.region.amazonaws.com/my-base-image</image>
    <auth>
      <username>my_username</username>
      <password>my_password</password>
    </auth>
  </from>
  <to>
    <image>gcr.io/my-gcp-project/my-app</image>
    <auth>
      <username>${env.REGISTRY_USERNAME}</username>
      <password>${env.REGISTRY_PASSWORD}</password>
    </auth>
  </to>
  ...
</configuration>

?

• Can you show me one example config of your jib maven plugin, when you run your mvn goals? (obfuscate secrets if necerssary)

[Jean-Baptiste-Lasselle]

:) but you speak french ?! :) ok, we'll keep on in english for the use of community

[Jean-Baptiste-Lasselle]

also, @dleborgne about your SYNC stategy, won't write the whole explanation, but you need to configure also PORTUS_DELETE_XXX variables. just clikc this to search with the PORTUS_DELETE string in opened and closed issues, and you will find a lot of explanations discussions I had on the subject, with others, before we finally repaired things like the garbage collector.

SYNC and DELETE are closely related , along with the garbage collector . And don't forget to set same config with env. vars. to both portus and background, except obvious things like PORTUS_BACKGROUND=true

I'll follow up the issue, cause I'll get a working jib after that, which is a tool I never used before, and I wanna springboot a little soon. So thank you for your questions and feedback.

[Jean-Baptiste-Lasselle]

Oh, oh : @dleborgne I just noticed : you are using portus 2.4.3

Myself along with a few community users, dissected completely portus, to understand what the problem was. I'll give you the short answer :

• There are many bugs you will keep facing, until you run a very specific version of portus : the last commit on master.
• I have writen down a complete proof of that, thanks to unified efforts with @diranged , @astonian @robgiovanardi and many others : https://github.com/SUSE/Portus/issues/2241#issuecomment-586972528
• And you will check that there exists, an official opensuse/portus:2.5, though there exists no 2.5 tag on git@github.com:SUSE/Portus
• worse, you will also checked that official opensuse/portus:2.5 crashes
• so I completely re-build portus from scratch, from last commit on master, and debian based, instead of opensuse based. See my screenshots showing witness that I get the exact Portus source code version here : https://github.com/SUSE/Portus/issues/2241#issuecomment-593370789

So, before anything, you have to upgrade from 2.4.3 to 2.5 , and you can do that from inside your orgasnisation, building your own docker image.

If you are willing to test using https://github.com/pokusio/portus-build-from-source/releases/tag/0.0.1 , I will be interested in results of the tests, after upgrade. And I'll take any issue you open on https://github.com/pokusio/portus-build-from-source/issues

Note that I also rebuilt from source the portusctl utility, to ease people upgrading to my image definition. (some people use portusctl inside entrypoints and init scripts, or to live manage users see this issue comment, though there are reasons why I do not recommend using portusctl )

[Jean-Baptiste-Lasselle]

just pops out my mind : portusctl portusctl authenticates as portus user, using the boot token (which should ever be used). I don't know if it's gonna be of any use.

[Jean-Baptiste-Lasselle]

@dleborgne thank you for making me read about jib, very interesting indeed, this ability it has to build a docker image, without requiring docker installed, or a base image, and layers by layers... And.. by the way, do run mvn from within inside of a container ? think you do, since you talked about it because of your work on a CI CD chain

Is it drone CI ?

[stale[bot]]

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[ehsanrocket]

https://github.com/SUSE/Portus/issues/2243#issuecomment-593288139 this works for me setting the PORTUS_BACKGROUND_SYNC_STRATEGY=update-delete in background section of the docker-compose file