Error writing events with webhook

[mitesh-devops]

I am using wildcard ssl and using the .crt with a combination of CA bundle file + certificate. The SSL works correctly when I add in portus UI. The push is also working but the webhook call is somehow not working. the registry container has the following error.

registry_1 | time="2020-04-21T15:09:30Z" level=error msg="retryingsink: error writing events: httpSink{https://registry.domain.com/v2/webhooks/events}: error posting: Post https://registry.domain.com/v2/webhooks/events: x509: certificate is valid for *.kinsta.com, kinsta.com, not registry.domain.com, retrying"

We have added appropriate entries with extra_hosts and it works on internal IP correctly but the above error somehow indicated that webhook is being called on a public network.

compose.txt

I have tried the solution given on https://github.com/SUSE/Portus/issues/1496. Please advice.

[Jean-Baptiste-Lasselle]

ok :

• this is ths registry container, trying to connect to portus, sending exactly https://registry.domain.com/v2/webhooks/events (he confirms to registry "Yes, I got your notification, thank you, you can stop resending the webhook")
• Well, first :
  • I personnally think this is a monster configuration you have here,
  • because the only reason why the https://registry.domain.com/v2/webhooks/events HTTP request ends up to portus instead of registry , is because of a very special configuration you have in your nginx.conf, which you found in Portus git repo. Do you agree it is weird to call https://registry.domain.com/v2/webhooks/events , while you actually want to connect to https://portus.domain.com/v2/webhooks/events ?
• Oh, and actually, ou don't want to connect your portus with https://portus.domain.com/v2/webhooks/events , but with https://portus.kinsta.com/v2/webhooks/events , don't you ?
• Yes you do, and TLS agrees with you, and so prevents you from connecting to https://registry.domain.com/ while the certificate is for *.kinsta.com
• but your wildcard TLS Cert, is *.kinsta.com
• So, the fix is that :
  • you don't bother about nginx
  • you change your configuration, so that the hostname for your registry is no longer registry.domain.com, but registry.kinsta.com, and same thing for portus.kinsta.com. Other possibility is that they all have hostname docker.kinsta.com, but it's weird, .think of hub.docker.com and docker.io/yourname/portus:2.4.3 .... (ping -c 4 docker.io). Use : PORTUS_MACHINE_FQDN=portus.kinsta.com on portus (must match portus.crt TLS cert CN Common Name), and docker config (you're devops)
  • then also don't forget :

    
    auth:
    token:
    realm: https://PORTUS_SERVICE_FQDN_JINJA2_VAR:3000/v2/token
    service: OCI_REGISTRY_SERVICE_FQDN_JINJA2_VAR
    issuer:  PORTUS_SERVICE_FQDN_JINJA2_VAR
    rootcertbundle: /secrets/portus.crt
    notifications:
    endpoints:

  • name: portus url: https://PORTUS_SERVICE_FQDN_JINJA2_VAR:3000/v2/webhooks/events

    https://docs.docker.com/registry/notifications/

    headers:

    Authorization: Bearer bs2M5RmbspjzLkh4GrXX

    timeout: 500ms threshold: 5 backoff: 1s secret: SECRET_PORTUS_API_KEY_JINJA2_VAR

• And come back with ur results

[mitesh-devops]

Hi, Thank you for your quick response! I couldn't figure out the need for different hostnames prior as couldn't find anything mentioned in a document to use the different hostname for portus and registry.

I have now changed hostnames with portus.domain.com and regisrty.domain.com(I am using the actual wildcard SSL issued to *.domain.com for both containers). and .env hostname to portus.domain.com as you suggested. However, now I am getting requested access to the resource is denied when uploading to registry.domain.com/admin/.

I didn't get you for the PORTUS_SERVICE_FQDN_JINJA2_VAR changes. Can you please explain a bit about it? I don't have this variable in .env.

Thank you once again!

[Jean-Baptiste-Lasselle]

@mitesh-devops please give :

• your docker-compose.yml file
• your .env
• and I want content of both of those files neatly formated, so I do not get tired reading :

# So just asking you to take special care to markdown format the 
# content of those two files, like this

# So just asking you to take special care to markdown format the 
# content of those two files, like this

• PORTUS_SERVICE_FQDN : look into your docker-compose.yml, and I will require that you work more than that, or advise you to forget about using portus or harbor, immediately, because you can't afford the work time required for that.
• So, you did not read the docker-compose.yml, and probably not a lot in the docs. Take some time on that, or you will have some questions I will not answer : bear in mind I am not form the Portus Team, I am answering just for the pleasure.
• And set PORTUS_SERVICE_FQDN=registry.domain.com in your docker-compose.yml

When I have that, I will ask more questions, and then answer.

[mitesh-devops]

Hello @Jean-Baptiste-Lasselle,

Thank you for sparing time to look into it!

Let me give you a short summary of what setup I am doing.

I have one Ubuntu machine at my office premises where I am setting up this. I just cloned this project and started working on this file https://github.com/SUSE/Portus/tree/master/examples/compose I required to perform few changes in the compose file to make things work.

I have wildcard certificate which I am using in portus.domain.com and registry.domain.com.

What is working:

1. UI is accessible with SSL at https://portus.domain.com:3000
2. I am able to create admin user and docker login registry.domain.com from CLI with that admin credentials.
3. Add registry from UI is working with green reachable status.
4. Push from CLI registry.domain.com/admin/image

What is not working:

1. visibility of the images on UI. Getting the same webhook error.

registry_1 | time="2020-04-23T05:32:36Z" level=error msg="retryingsink: error writing events: httpSink{https://portus.domain.com/v2/webhooks/events}: error posting: Post https://portus.domain.com/v2/webhooks/events: x509: certificate is valid for *.kinsta.com, kinsta.com, not portus.domain.com, retrying" registry_1 | time="2020-04-23T05:32:36Z" level=warning msg="httpSink{https://portus.domain.com/v2/webhooks/events} encountered too many errors, backing off"

Note: Kinsta.com has nothing to do here as it is not our domain name. It seems that it is trying to go outside the public network and check for SSL. But, the SSL is working within an internal network only as it we are not exposing it to a public network by reissue of the ssl with 2 new common names.

I have added both compose and .env file.

docker-compose.txt env.txt

[mitesh-devops]

More on this. The following curl request is successful from inside all the containers which seem that ssl is correctly Identified.

'curl -v -X POST https://portus.domain.com/v2/webhooks/events'

curl-logs.txt

Also, the first time pushed image is visible on UI but after that, it doesn't sync any new images to Portus.

I hope this information helps to debug. Can you please provide me any POST request example with header,body which I can try from registry container console and see if any clue is identified?

[Jean-Baptiste-Lasselle]

@mitesh-devops Just a quick remark before answser :

• in :

  registry_1 | time="2020-04-23T05:32:36Z" level=error msg="retryingsink: error writing events: httpSink{https://portus.domain.com/v2/webhooks/events}: error posting: Post https://portus.domain.com/v2/webhooks/events: x509: certificate is valid for *.kinsta.com, kinsta.com, not portus.domain.com, retrying" registry_1 | time="2020-04-23T05:32:36Z" level=warning msg="httpSink{https://portus.domain.com/v2/webhooks/events} encountered too many errors, backing off" 

• linux tells you, and me, that *.kinsta.com has to do with what you (and noone else) configured
• So trust me, Linux is right, and you are wrong if u think there is no relation.

ttyl im at work

[Jean-Baptiste-Lasselle]

Hi again @mitesh-devops So, I asked you :

• your docker-compose.yml file (ok I have it)

• your .env (ok I have it)

• and it is fine if you give them as files, instaed of content of both of those files markdown formated

• Additionnally, I will also need :

  • your nginx.conf,
  • and for every file, I need :
  • the path of the file on your computer, like /home/mitesh/vscode/myproject/docker-compose.yml
  • in addition to the content of the file, well formated as requested before.

Ok, this is stupid : we are on github, and we are exchanging files with http link... : could you just commit and push to a git repository all the files in the folder where you ran docker-compose up ?

One remark :

• It seems like you are in a hurry
• It also seems ("I run portus on my compouter ay work"), that you are alone, working on having a private registry
• If no one is aksing you to work on an infrastructure, and you just need a private regsitry, to do something, I am advising you :
  • To stop right now working on portus, and look for a much simpler solution (find a "your docker private regsitry" tutorial, until you find one where you succeed)
  • Or be prepared to work a lot, and for months, befopre you get any close to production.

You are informed. Unless you provide the requested files, I will assume you chose the first option.

JB.

[Jean-Baptiste-Lasselle]

hi @mitesh-devops I update my last message,

Treating portus issue https://github.com/SUSE/Portus/issues/2297

• Execute this :

export OPS_HOME=$(pwd)/jblanswerabout
export DESIRED_VERSION=feature/preparing-first-release
git clone https://github.com/Jean-Baptiste-Lasselle/mitesh-dialog ${OPS_HOME}
cd ${OPS_HOME}
git checkout ${DESIRED_VERSION}

cd mitesh
cat README.md | head -n 1

docker-compose config |grep image:
echo ""
echo "So you used portus image [opensuse/portus:head]"

• So you used the docker image opensuse/portus:head : which does not exists on any public docker registry
• For a good reason : that is the developement setup.
• And now I know you docker built yourself the image of portus: another thing that I miss, to reproduce your case.
• Ok, those files having txt extension do not give me any kind of information I can reliably work on.

I now do not need any more work to tell you :

• That you are not able to understand what reproduceability is,
• So I will definitely advise to forget working with Portus,
• And when you think you understand this explanation, and reproduceability
• You can try again and publicly ask a questionh to an engineer.

The good news, is now you can have a nice restful week-end.

JB.

[stale[bot]]

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.