[BUG] TLS CONFIGURATION FAILS: Failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]

[Siradjedd]

What is the bug?

logs :

`java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: OpenSearchException[Unable to read /usr/share/opensearch/config/tls-transport/ca.crt (/usr/share/opensearch/config/tls-transport/ca.crt). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemtrustedcas_filepath]`

Configuration files

i'am using cert-manager and let'sencrypt for the certs generation Certifications:

`
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: ca-certificate
  namespace: default
spec:
  secretName: ca-cert
  duration: 9000h # ~1year
  renewBefore: 360h # 15d
  isCA: true
  privateKey:
    size: 2048
  dnsNames:
    - '*.{{.Values.domain}}.adeiz.com'
  usages:
    - digital signature
    - key encipherment
  issuerRef:
    name: letsencrypt-{{.Release.Namespace}}-prod
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-certs
  namespace: default
spec:
  secretName: opensearch-certs
  duration: 9000h # ~1year
  renewBefore: 360h # 15d
  isCA: false
  privateKey:
    size: 2048
    algorithm: RSA
    encoding: PKCS8
  dnsNames:
    - '*.{{.Values.domain}}.adeiz.com'
  usages:
    - signing
    - key encipherment
    - server auth
    - client auth
  issuerRef:
    name: letsencrypt-{{.Release.Namespace}}-prod
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-admin-certs
  namespace: default
spec:
  secretName: opensearch-admin-certs
  duration: 9000h # ~1year
  renewBefore: 360h # 15d
  isCA: false
  privateKey:
    size: 2048
    algorithm: RSA
    encoding: PKCS8
  dnsNames:
    - '*.{{.Values.domain}}.adeiz.com'
  usages:
    - signing
    - key encipherment
    - server auth
    - client auth
  issuerRef:
    name: letsencrypt-{{.Release.Namespace}}-prod
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-dashboards-certs
  namespace: default
spec:
  secretName: opensearch-dashboards-certs
  duration: 9000h # ~1year
  renewBefore: 360h # 15d
  isCA: false
  privateKey:
    size: 2048
    algorithm: RSA
    encoding: PKCS8
  dnsNames:
    - '*.labadeiz.adeiz.com'
  usages:
    - signing
    - key encipherment
    - server auth
    - client auth
  issuerRef:
    name: letsencrypt-{{.Release.Namespace}}-prod
    kind: ClusterIssuer`

ClusterIssuer:

`apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-{{.Release.Namespace}}-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: contact@adeiz.com
    privateKeySecretRef:
      name: letsencrypt-{{.Values.domain}}-prod-account-key
    solvers:
    - selector:
        dnsNames:
        - '*.{{.Values.domain}}.adeiz.com'
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key-prod
              key: key
              secret: secret
            production: true
            ttl: 600
          groupName: acme.mycompany.com
          solverName: godaddy`

Opensearch-cluster.yaml:

apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: my-first-cluster
  namespace: default
spec:
  general:
    serviceName: my-first-cluster
    version: 2.3.0
    setVMMaxMapCount: true
  dashboards:
    enable: true
    version: 2.3.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
    tls:
      enable: true
      generate: false
      secret:
        name: opensearch-dashboards-certs
  security:
    config:
      adminSecret:
        name: opensearch-admin-certs
    tls:
      transport:
        generate: false
        perNode: false
        secret:
          name:  opensearch-certs
        nodesDn: [CN="'*.{{.Values.domain}}.adeiz.com'", ] 
        adminDn: ["CN='*.{{.Values.domain}}.adeiz.com'", ]
      http:
        generate: false
        secret:
          name: opensearch-certs
  nodePools:
    - component: masters
      replicas: 2
      diskSize: "3Gi"
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
        - "master"

What is the expected behavior?

OpenSearch works normally with the generated certifications.

[prudhvigodithi]

[Triage] Adding @swoehrl-mw to please take a look and guide @Siradjedd. Thanks