Suggested approach for custom CA certificates for new JFrog Platform achitecture

[erick-duran-tfs]

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request

Version of Helm and Kubernetes: Helm v3.1.0 Kubernetes v1.14

Which chart: All

What happened: Context: Our company uses internal CAs to sign our SSL certificates for all our services. In the new platform version/charts, we now just use the internal signed certificate for Artifactory (HA). Therefore, there is a need for us to provide the root CA to all the services (Xray, Distribution, Mission Control) so they can trust the main Artifactory service(s), as we have set up HTTP->HTTPS redirection in our PROD instances (currently in Artifactory 6.x version and compatible).

Result: In previous versions, we leveraged the customVolumeMounts params available for some of the charts, and it worked fine for at least Artifactory + Xray.

However, looking at your new architecture, I see that the router microservices now perform the communication between the JFrog products. We noticed that the previous approach does not work as the communication is relayed to the router container, therefore the router container logs show the "certificate signed from unknown authority" error.

We couldn't find a customVolumeMounts in the router values or similar approach in your charts. Is there a recommended way to trust certificates in the router container?

Also, we couldn't reach an Artifactory Edge instance from our main Artifactory HA instance via HTTPS because of the same problem. Which is the service that performs the connection with other instances in the "Platform Deployments" tab? When trying to add the custom CA certificate to Artifactory's keytool we've got the same "Permission Denied" error in the logs as we pointed out here. Does it make sense to try to trust it from keytool or do we need to somehow access Artifactory's router (I don't see a router definition in the artifactory-ha chart)?

What you expected to happen: All the services to communicate.

How to reproduce it (as minimally and precisely as possible):

1. Launch the Artifactory service with a not-trusted cert
2. Try to deploy Xray/MC/Distribution with a HTTPS JFrog URL

Anything else we need to know: How are we making it work right now? The way we got it to work was by creating a custom Docker image, taking docker.bintray.io/jfrog/router latest version as the base image and just appending the custom CA certificate at /etc/ssl/certs/ca-certificates.crt. Then, we just:

router:
  image: 
    repository: <custom-image-repository>
    version: 1.3.0
    imagePullPolicy: IfNotPresent

We have tested that with the Xray chart only, it works fine (Xray -> Artifactory via HTTPS URL). However, we would really appreciate a more maintainable solution.

Thanks in advance!

[erick-duran-tfs]

An update for this on the Artifactory side: We were trying to reach out our internal Edge node instance via HTTPS (using the platform deployments UI). We are getting a "Test connection failed" error and I can't find anything relevant in the logs, tests fine via HTTP (same endpoint). We assume this is related to our internal CA...

Here's what we have tried:

• Trusting the CA cert using keytool. Looks like the keytool used is now located here: /opt/jfrog/artifactory/app/third-party/java/bin/keytool. So we ran in all nodes:

  /opt/jfrog/artifactory/app/third-party/java/bin/keytool -importcert -keystore /opt/jfrog/artifactory/app/third-party/java/lib/security/cacerts -storepass changeit -file /PATH/TO/CERT -alias mycert -noprompt

• Appending the CA cert using to /etc/ssl/certs/ca-certificates.crt so it is generally reachable by the tools in the container.

We tried this steps in all nodes from both the Artifactory and Artifactory Edge deployments (all use the same root CA). No luck so far.

[chukka]

@erick-duran Thanks for raising this. This feature is currently under development (no tentative release date yet) .Custom CA can be provided in Access (since Artifactory > 7.4.0), please refer - https://www.jfrog.com/confluence/display/JFROG/Managing+TLS+Certificates

[chukka]

Duplicate of #617

[evannorstrand]

Duplicate of #759

@chukka fix the code the code broken in the PR and CA Certs will once again work :)

[evannorstrand]

@chukka since you decided to close my ticket claiming the issue was fixed by "going a new way" and saying I just needed to do a reinstall.

I have reinstalled, the issue still exists, and you have multiple customers broken.

The fix is as follows:

Update the charts to re-allow the customVolumeMounts as I had them working before. The customVolumeMount needs to be attached to the "router" service.

Do this and we are all back in business.

[evannorstrand]

@chukka for your tracking, I have opened and escalated an official case id as I am an enterprise customer. https://support.jfrog.com/jFrogCasePage?id=5001r00002rKnzsAAC

[chukka]

@evannorstrand Thanks for feedback , Can you please review this PR (work in progress) - https://github.com/jfrog/charts/pull/943

Update :

https://github.com/jfrog/charts/pull/943 is Merged

[chukka]

@erick-duran customvolumeMounts has been added under router in latest xray chart 3.4.1

[erick-duran-tfs]

Thank you @chukka, I will review this next week and get back to you to let you know how it goes. Is implementing this for your other products also in the roadmap? We are currently using the above workaround with Xray, Mission Control and Distribution.

[chukka]

@erick-duran Sure, Please try this workaround and share feedback 👍

yes, We have an internal tracking issue for supporting custom CA certificates for all products . Sorry. No ETA

[evannorstrand]

I am trying to deploy now, and I am getting this error

Failed to install app xray. Error: UPGRADE FAILED: YAML parse error on xray/templates/xray-statefulset.yaml: error converting YAML to JSON: yaml: line 348: mapping values are not allowed in this context

The values yaml looks like:

common: customVolumes: |

• name: my-ca configMap: name: my-ca

[evannorstrand]

@chukka Here is the issue:

{{- if .Values.indexer.customVolumeMounts }} {{ tpl .Values.indexer.customVolumeMounts . | indent 8 }} {{- end }}

This was the original code, you need to move the NEW code to the left with no indents.

[evannorstrand]

@danielezer - bringing this to your attention.

[chukka]

@evannorstrand Thanks for reporting - Fix is available in PR https://github.com/jfrog/charts/pull/946. Can you please try and share feedback

[evannorstrand]

@chukka The PR fixed the mounting issue.

I can now mount the ca certificate, but I still get the following.

2020-06-23T00:19:53.558Z [jfrou] [INFO ] [6a7cc0c1f49417bb] [join_executor.go:152 ] [main ] - Cluster join: Retry 50: Service registry ping failed, will retry. Error: Get "https://artifactory.my.domain/access/api/v1/system/ping": x509: certificate signed by unknown authority

[chukka]

@evannorstrand Thanks for the feedback, Could you please help us with the reproducible (detailed) steps. I hope you are using custom CA certificates ( if possible can you share them as well)

[evannorstrand]

1) Deploy the application with the following config

---
  unifiedUpgradeAllowed: true
  xray: 
    jfrogUrl: "https://artifactory.my.domain"
    joinKey: "xyz"
  common:
    customVolumes: |
      - name: my-ca
        configMap: 
          name: my-ca
    customVolumeMounts: |
      - name: my-ca
        mountPath: /etc/pki/tls/certs/ca-bundle.crt
        subPath: ca-bundle.crt

2. The router throws the x509 error, xray fails

I will email the custom ca cert to the ticket i've got open on the internal system.

[maxyor6040]

please provide access-service.log output

[evannorstrand]

The access service restarts constantly because the router fails.

[evannorstrand]

The issue seems to be that the bundle isn't being used by the Router service.

[erick-duran-tfs]

Sorry for the delay I have finished testing this. Custom mounts work fine, but just putting the file into /etc/ssl/certs/ does not work and /etc/pki/tls/certs/ca-bundle.crt as @evannorstrand suggests might not work for this Linux distro. Please provide the exact path you want us to mount the certificate to.

Using the workaround that I initially posted via the prestart command didn't work either as running

cat /etc/ssl/certs/my-cert.crt >> /etc/ssl/certs/ca-certificates.crt

returns a permission denied since the pre-start command is not running as root.

So I've got this to work by a slightly different approach that we were doing (overrinding the router with our own router image), by getting the original /etc/ssl/certs/ca-certificates.crt from the router/indexer/analysis/persist/server images and appending our private certs to the end. Then we just:

common:
  customVolumes: |
    - name: ca-bundle
      configMap:
        name: ca-bundle
  customVolumeMounts: |
    - name: ca-bundle
      mountPath: /etc/ssl/certs/ca-certificates.crt
      subPath: ca-certificates.crt
      readOnly: true

Which will overwrite the original file with the new file that is exactly the same but includes our cert. I used the common mount as I verified and all microservices use the same ca-certificates.crt file.

Even though I think this is better than our initial workaround, this is not ideal as we will have to maintain that file to keep up-to-date with the latest certificates if those ever change in the base images. It would be way better to just simply mount this to a known trusted certificate directory that can be writable by the user and call it a day.

Kudos

[palko444]

Hello chart developers, would it be possible to speed up little bit development of this feature. I guess lot of people is waiting for this and I'm one of them. I changed this already 3 times since I'm using artifactory/xray helm charts. I do not want to repackage docker images every time as this is only working solution all the time. Having certs as volumes and init script loading them to java key store was working solution and it was easy to implement. I guess same can be done also for other products( init script running java keytool or update-ca-trust or what ever else is required). It can be done same way (for customers) in all products and it should be easy to implement even for developers. I'm paying customer of jfrog and these changes are not helping really. If it will take a long time to implement this feature, can you please return it back as its done in init-script for artifactory helm chart v6 and if you later decide to have new way, please keep also old way for some amount of time. Not just cut it like it happens now. Thank you in advance

[erick-duran-tfs]

We also have an internal support ticket for this, #130966 for reference.

[palko444]

@erick-duran, is it possible to see ticket if i have account here? https://support.jfrog.com/

Thank you in advance

[evannorstrand]

@palko444 - Tickets are private I believe and can't be viewed by normal users. I feel your frustration, I was the original contributor who fixed this in the previous version of the xray charts. To help push this issue, please create a support ticket and reference this github ticket. The more paying customers who put in support tickets, the faster this issue will be resolved.

@erick-duran - I agree, creating a custom image is an unnecessary burden on users, and proves that this is a bug in the latest implementation of xray.

@chukka & @danielezer - As you guys can see, we have at least three paying customers blocked due to this bug. Any updates on when we can finally get a resolution?

[chukka]

@evannorstrand @palko444 @erick-duran Thanks for follow up, We are currently working (tracking issue internally) on simplifying the approach to use certificates across the charts. ETA ~ 2 Weeks from now.

[evannorstrand]

@chukka & @danielezer any updates?

[palko444]

@chukka & @danielezer any updates please?

[chukka]

@palko444 Thanks for patience, As you can see we have 2 PRs open #1020 and #1023 , new charts will be available this week :)

[chukka]

@evannorstrand @erick-duran @palko444 please check artifactory and xray changes. Will add support for distribution and Mission-control in the coming days

Please try below charts and share feedback

• artifactory-10.0.6
• artifactory-ha-3.0.6
• xray-4.1.2

Update : Below charts have the support too

• distribution-6.1.3
• missioncontrol-4.2.0

[palko444]

Hello, Artifactory as CA

so from what I saw, you want access to be CA for artifactory/xray/.. deployment. But what about use case, when our S3, LDAP and other dependencies are using custom certificate. Anyway i do not like idea to have another CA. Our CA will give me certificate, i just need to make trust with our CA, by importing certificates as it was done before. If I will use artifactory as CA, all users/tools (gitlab/k8s/..) have to import my custom CA. This is NOGO. If i understand correctly, this change does not solve anything for my use case, just making it much more complicated for users using artifactory.

I was just checking changes you pointed in link, If I miss something please let me know. I need an easy way how to import certificates from our CA, so artifactory trust all other component signed with this CA.

XRAY/ARTIFACTORY - cert import

I think this is unnecessary complicated. I liked way how its done in artifactory chart artifactory-ha-1.6.0 (app version 6.19). All i have to do is to mount my certificates to _/artifactory_extracerts directory and init container will take this to javakeystore. (all files there) With this approach i can actually have more than one certificate.(This is my use case)

With current solution i have to do this:

1. create volume/volumeount somethere - this is the only thing i had to do for artifactory-ha-1.6.0 chart
2. create some directory
3. copy file from that directory to {{ .Values.xray.persistence.mountPath }}/etc/security/keys/trusted/ca.crt
4. Im not sure if javakeystore allows to have more than one cert in file

I guess same think can be done in artifactory chart.

Thank you in advance Br Palo

[chukka]

@palko444 if you are using external CA, please refer https://github.com/jfrog/charts/issues/242#issuecomment-600753714

[palko444]

hello @chukka , thx for this. Can you please put this also to helm documentation for artifactory, I guess this is useful also for other people.

Another question, is this same also for xray? Or I have to use way you suggest in here.

Thank you in advance Br Palo

[chukka]

@palko444 Sure, will plan to update helm documentation in upcoming chart releases .

For Xray, please use as suggested