Closed markusheiden closed 4 years ago
hi @markusheiden I encountered same problem, and I'm pretty sure where your problem is :
It's in the backgound
Indeed, as you will find in the portus
awful documentation (I mean it, it's awful, awfully inconsistent yet) :
the background process is required for a production setup.
Why required? I'll tell you why: because if it isn't there, well you will miss something.
I encountered the same problem : the Web UI would never show any images in the namespace listing page, though I did sucesfully pushed several images to the registry
.
What is really important to understand, is that :
background
container in the docker-compose.yml Im' sure you know about, receives the webhook HTTP request sent by the docker registry
to send notifications about regsitry events , like "Oh a [docker push] just completed successfully, you can display it in your web ui now" .background
, I surely verified that I stil can sucessfully push new images to the docker registry
, but the Portus
WebUI Portus at https://portus.mycompany.io:3000/ won't ever display it...What happened for em is that I messed up with the background container configuration, so it just stopped its execution at runtime, and therefore...byebye UI.
So OK.
I thought then that I could definitely check that webhooks are important ONLY to the background
container, the portus
container does not need to receive them (actually he already discards them, if I believe its logs in my current environment), with the following experience :
export PORTUS_BACKGROUND_SERVICE_FQDN=portus-backservice.mycompany.io
private docker registry
, in the docker-compose.yml
, so that it becomes : # REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus-oci-registry.crt
REGISTRY_HTTP_TLS_KEY: /secrets/portus-oci-registry.key
# Portus endpoint
# So here https://${PORTUS_BACKGROUND_SERVICE_FQDN}/v2/webhooks/events should point to portus background service
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://${PORTUS_BACKGROUND_SERVICE_FQDN}:3000/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
# REGISTRY_HTTP_TLS_KEY: /secrets/portus.key
REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/portus-oci-registry.crt
REGISTRY_HTTP_TLS_KEY: /secrets/portus-oci-registry.key
# Portus endpoint
# So here https://${PORTUS_SERVICE_FQDN}/v2/webhooks/events should point to portus
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://${PORTUS_SERVICE_FQDN}:3000/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
I'll do those tests tomorrow, n give u the answer if you ask.
All in all, I would be glad to find out if this helped, while I'm pretty sure here you just got rid of the background container, or mis-configured it.
JB
We have the same problem here using Docker Registry 2.7.1 and Portus 2.4.3. Everything works as expected with the Docker 19.03 client but when Jib 2.0.0 client sends the manifest PUT request, the Portus backgound container log contains a "Cannot find user" message, which is probably caused by the empty "actor" field in the JSON message received by the background Portus container.
Using Docker client, the push event sent by Docker Registry to Portus, with non-empty "actor" field
Handling 'push' event:
{
"id": "e4f8ba20-c7e1-4916-919c-a90803854ccc",
"timestamp": "2020-02-04T11:00:56.708989056Z",
"action": "push",
"target": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1785,
"digest": "sha256:49433a8241d5e44ca711191b978055f1d2ab5ccc862acf7c4d8d73c274342e91",
"length": 1785,
"repository": "testjib/hello-world",
"url": "https://registry/v2/testjib/hello-world/manifests/sha256:49433a8241d5e44ca711191b978055f1d2ab5ccc862acf7c4d8d73c274342e91",
"tag": "latest"
},
"request": {
"id": "b1579a99-7f2b-454a-aeb3-f1bf611b320f",
"addr": "10.255.0.2",
"host": "registry",
"method": "PUT",
"useragent": "docker/19.03.4 go/go1.12.10 git-commit/9013bf5 kernel/4.9.184-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.4 \\(windows\\))"
},
"actor": {
"name": "testjib"
},
"source": {
"addr": "0564da668715:5000",
"instanceID": "4396b935-e643-4384-8526-2e85839e5427"
}
}
Using Jib client, the push event sent by Docker Registry to Portus, with empty "actor" field
{
"id": "412c19bc-fe91-4e81-a735-b2ac76f97bf8",
"timestamp": "2020-02-04T10:46:20.116168774Z",
"action": "push",
"target": {
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 1403,
"digest": "sha256:7d13e143448e78c7f81c75aeda012513b7f898bfb5f8cc2a6541a13c5d831a16",
"length": 1403,
"repository": "testjib/hello-world",
"url": "https://registry/v2/testjib/hello-world/manifests/sha256:7d13e143448e78c7f81c75aeda012513b7f898bfb5f8cc2a6541a13c5d831a16",
"tag": "latest"
},
"request": {
"id": "aba2e75f-68fd-4f69-ba85-8a11cd10f828",
"addr": "10.255.0.2",
"host": "registry",
"method": "PUT",
"useragent": "jib 2.0.0 jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)"
},
"actor": {
},
"source": {
"addr": "0564da668715:5000",
"instanceID": "4396b935-e643-4384-8526-2e85839e5427"
}
}
Cannot find user
hi @markusheiden @dleborgne , I today add a comment to update you with my last test results, which are :
docker-compose.yml
, I have 3 different network names / FQDN for portus
, background
, and registry
(also therefore, I now have 3 different SSL/TLS Certificates)portus
service, which expects and must receive webhooks from the registry
service, which I configured like this : # Portus endpoint
# So here https://${PORTUS_SERVICE_FQDN}/v2/webhooks/events should point to portus
REGISTRY_NOTIFICATIONS_ENDPOINTS: >
- name: portus
url: https://${PORTUS_SERVICE_FQDN}:3000/v2/webhooks/events
timeout: 2000ms
threshold: 5
backoff: 1s
Does it help ? Did you already solved the problem ?
The only thing to help, that cross my mind, is that I think very likely the problem is unrelated with the maven
jib
plugin
For more details on the mechanism there's also https://github.com/SUSE/Portus/issues/2275#issuecomment-586648423 I recently wrote.
As I wrote before, the issue only happens when using Jib. Images pushed with the docker client are updated in the Portus web UI. I assume everything is well configured regarding the REGISTRY_NOTIFICATIONS_ENDPOINTS registry environment variable.
Today, after some digging in the Portus documentation and Github issues, I changed the background sync strategy to update
using the PORTUS_BACKGROUND_SYNC_STRATEGY
environment variable, and... all the images pushed with Jib appeared in the UI. But their author is "portus" instead of the real authenticated user that pushed them to the registry, so this looks more like a workaround than an solution to the Jib/Registry/Portus issue 😢
Hi @dleborgne answering quite a few weeks later :
Never the less, I wan to tell you that I am almost sure I have seen people having a very similar problem, related to GUNs and notary : the guys were complaining the webhooks received from notary, were rejected because portus does not support GUNs in its webhooks endpoint.
Never the less, reading that issue about notary, I wanted to test the webhook, and see for myself : does portus receives from the registry, the info of the pusher id ? Answer is yes, see https://github.com/SUSE/Portus/issues/2293#issuecomment-612449096
So Okay, I'm knackered, but could not resist, I know how frustating your team feels, being in a team myself, n after succeeding , now you get new problem... . So I worked a little , and wrote a test, you can run against your private portus infra :
obelix
: use the WebUI to create a portus token, and docker login
as obelix
, and provide your token wheln prompted for password by the docker client.obelix
mentioned as pusher : which means the webhook payload, from registry
, to portus
, does convey the info of the portus username of the pusher.portus
?registry
through portus
? Or do you use ~/.docker/config.json
, or specific credentials plugin config like the following : <configuration>
...
<from>
<image>aws_account_id.dkr.ecr.region.amazonaws.com/my-base-image</image>
<auth>
<username>my_username</username>
<password>my_password</password>
</auth>
</from>
<to>
<image>gcr.io/my-gcp-project/my-app</image>
<auth>
<username>${env.REGISTRY_USERNAME}</username>
<password>${env.REGISTRY_PASSWORD}</password>
</auth>
</to>
...
</configuration>
?
maven
plugin, when you run your mvn
goals? (obfuscate secrets if necerssary):) but you speak french ?! :) ok, we'll keep on in english for the use of community
also, @dleborgne about your SYNC stategy, won't write the whole explanation, but you need to configure also PORTUS_DELETE_XXX
variables. just clikc this to search with the PORTUS_DELETE
string in opened and closed issues, and you will find a lot of explanations discussions I had on the subject, with others, before we finally repaired things like the garbage collector.
SYNC and DELETE are closely related , along with the garbage collector . And don't forget to set same config with env. vars. to both portus and background
, except obvious things like PORTUS_BACKGROUND=true
I'll follow up the issue, cause I'll get a working jib after that, which is a tool I never used before, and I wanna springboot a little soon. So thank you for your questions and feedback.
2.4.3
Myself along with a few community users, dissected completely portus
, to understand what the problem was. I'll give you the short answer :
portus
: the last commit on master
.opensuse/portus:2.5
, though there exists no 2.5
tag on git@github.com:SUSE/Portus opensuse/portus:2.5
crashesSo, before anything, you have to upgrade from 2.4.3 to 2.5 , and you can do that from inside your orgasnisation, building your own docker image.
If you are willing to test using https://github.com/pokusio/portus-build-from-source/releases/tag/0.0.1 , I will be interested in results of the tests, after upgrade. And I'll take any issue you open on https://github.com/pokusio/portus-build-from-source/issues
Note that I also rebuilt from source the portusctl
utility, to ease people upgrading to my image definition. (some people use portusctl inside entrypoints and init scripts, or to live manage users see this issue comment, though there are reasons why I do not recommend using portusctl
)
just pops out my mind : portusctl
portusctl authenticates as portus
user, using the boot token (which should ever be used). I don't know if it's gonna be of any use.
@dleborgne thank you for making me read about jib
, very interesting indeed, this ability it has to build a docker image, without requiring docker installed, or a base image, and layers by layers... And.. by the way, do run mvn
from within inside of a container ? think you do, since you talked about it because of your work on a CI CD chain
Is it drone CI ?
Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
https://github.com/SUSE/Portus/issues/2243#issuecomment-593288139 this works for me setting the PORTUS_BACKGROUND_SYNC_STRATEGY=update-delete in background section of the docker-compose file
Description
When I push via JIB (e.g.
mvn jib:build
) to our docker-ce registry, the image shows as successfully pushed to the registry. But Portus does not show it. If I try pulling it, I do get the correct image, which means the registry got the image.Pulling it via docker and pushing it via docker make it appear in Portus.
An analysis of a colleague showed that the events of the registry are in a slightly different order (sorry, but I don't know the details) when pushing with JIB instead of the docker client.
Steps to reproduce
Sorry, but the other requested information is not available for me.