Self-Signed/Additional trusted certificate support in AppFramework

[Akhalaka]

When starting up a new splunk resource with the app framework configured, the init container fails with:

fatal error: SSL Validation failed for https://XXXXXXXXX [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:  self signed certificate in certificate chain (_ssl.c:1125)

The appRepo configuration is as follows:

appRepo:
  appsRepoPollIntervalSeconds: 600
  defaults:
    volumeName: test-apps
    scope: local
  appSources:
  - name: configApps
    location: common-config-apps/
  volumes:
  - name: test-apps
    storageType: s3
    provider: aws
    path: splunk-dev-apps/
    endpoint: https://XXXXXXXXXXXXXXX
    secretRef: s3

The endpoint URL points to a ceph s3 endpoint that I have trusted certificates available, but would need to be able to configure the amazon/aws-cli image with. Based on looking at the code this all looks to be hardcoded right now.

Other notes: I did hack the underlying statefulset to add --no-verify-ssl option in the runtime args of the init container. This did allow the pod to start and the 1 test app to download to /init-apps/configApps/test.tgz. This clearly is not a sustainable solution because the next time the splunk definition is changed, this does get wiped out. That said, this trusted certificate also needs to be available in the splunk-operator itself. The splunk operator started producing the error

{"level":"info","ts":1629738397.8384712,"logger":"splunk.client.GetAppsList","msg":"Got an object error","object.Err":"get \"https://XXXXXXXXXXXXXXXXXXXXX\": x509: certificate signed by unknown authority","client.BucketName":"splunk-dev-apps"}

Some potential solutions would be:

• to allow the configuration of the amazon/aws-cli image to use. This would allow me to build the identical image but with the additional trust built in (I can do the same thing with the actual splunk container...have not tested this yet though). This would also be useful/necessary to continue to support the full use of Private registries (ie with the RELATED_IMAGE_SPLUNK_ENTERPRISE piece of config)
• allow for additional configuration in the referenced secret to include additional trusted CA certificates and take appropriate actions to add this to the necessary trust.

[magguns]

Any news on this? We are using rook-ceph object store, and thus provider: minio . Since TLS is required and the http connection is force-upgraded to https there should be some means of supplying the CA certificate ourselves.

[magguns]

Small update: I upgraded to splunk-operator 2.1.0 yesterday (from 1.0.5) and noticed that the Minio client session no longer force-upgrades http to https, so you can indeed use the provider: minio with http s3 endpoints now!

[PrymalInstynct]

I have run into the same issue working in a network that is not connected to the internet. so using tools like certmanager and lets encrypt to get a globally trusted cert just is not possible. We are limited to using a Certificate Authority without any outside trust.

I had success as described by @magguns by configuring Minio to use insecure http, which is fine for development but that ultimately will not fly in production.

What is odd to me is that the SmartStore configuration does not have the same issue as the AppFramework does.

[yaroslav-nakonechnikov]

yes, support of self-signed certificates is needed.

and i've checked splunk-ansible and splunk-docker projects - nothing supports it. But it is really needed.