[Bitnami/MongoDB] External access with SSL enabled not working

bitnami / charts

Bitnami Helm Charts

https://bitnami.com

Other

9k stars 9.21k forks source link

[Bitnami/MongoDB] External access with SSL enabled not working #4182

Closed mnmami closed 3 years ago

mnmami commented 4 years ago

Which chart: Bitnami/MongoDB Chart version 9.1.2, App version 4.4.1

Describe the bug The external access with SSL enabled not working after enabling SSL encryption.

To Reproduce Steps to reproduce the behavior:

Install the Chart with external access enabled, for example, using the following settings:

architecture: replicaset
replicaCount: 4
externalAccess.enabled: true
externalAccess.service.type: LoadBalancer
externalAccess.service.port: 27017
externalAccess.autoDiscovery.enabled: true
rbac.create: true

This will generate a LoadBalancer service for every pod with external IP addresses, say IP1, IP2 ,IP3 ,IP4.

Generate or simply copy mongodb.pem and mongo-ca-cert files from one of the generated pods and start mongo shell with the full list of external service IPs, like so:
```
mongo --host "IP1,IP2,IP3,IP4" --ssl --sslPEMKeyFile=ssl_keys/mongodb.pem --sslCAFile=ssl_keys/mongodb-ca-cert 
```

You get: The server certificate does not match the host name. Hostname: {IP1} does not match SAN(s): {the list of CNs configured} (DNSs list see under Additional context below).

This since because the external IP addresses are only known later after the chart has been installed.

Expected behavior Client (e.g., mongo shell) successfully starts by providing the PEM file and CA file as arguments.

Version of Helm and Kubernetes:

Output of helm version:

version.BuildInfo{Version:"v3.3.0-rc.1", GitCommit:"5c2dfaad847df2ac8f289d278186d048f446c70c", GitTreeState:"dirty", GoVersion:"go1.14.4"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15+", GitVersion:"v1.15.12-gke.20", GitCommit:"0ac5f81eecab42bff5ef74f18b99d8896ba7b89b", GitTreeState:"clean", BuildDate:"2020-09-09T00:48:20Z", GoVersion:"go1.12.17b4", Compiler:"gc", Platform:"linux/amd64"}

Additional context I tried to add the external LoadBalancer service names (ending "-external") to the existing list of CNs https://github.com/bitnami/charts/blob/2ab9b6e14babbb48ee65357efef88580769b9f3a/bitnami/mongodb/templates/replicaset/statefulset.yaml#L129-L133

... like so:

...
DNS.6 = ${my_hostname}-external
DNS.7 = ${my_hostname}-external.$svc.$MY_POD_NAMESPACE.svc.cluster.local

Did not solve it.

SSL feature was recently introduced (in https://github.com/bitnami/charts/issues/3365), so the external access feature should be adapted accordingly.

FraPazGal commented 3 years ago

Hi @mnmami,

Thanks for bringing up this issue! As far as I understand, we would need to include an alt_name for the new LoadBalancer IPs, is that right? A solution that should allow you to configure the DNSs before chart installation is using statics IPs and pass them to the LoadBalancers using externalAccess.service.loadBalancerIPs. Would that work for you?

mnmami commented 3 years ago

Assuming I have a static IP address for every replicaset member, shall I configure the certificate of every member with all the IPs (IP.0,IP.1,...)?

FraPazGal commented 3 years ago

Hi @mnmami,

I would say so, as the certificate is created to be used by all of the pods created by the statefulset.

mnmami commented 3 years ago

Hi @FraPazGal,

I tried and it worked 👍 .

We need to make this configurable though, adding the static IPs to the certificate creation script under alt_name (https://github.com/bitnami/charts/blob/master/bitnami/mongodb/templates/replicaset/statefulset.yaml#L128).

mnmami commented 3 years ago

One idea could be to take the list of IPs from the variable loadBalancerIPs https://github.com/bitnami/charts/blob/master/bitnami/mongodb/values-production.yaml#L447 when the external access is enabled, read it inside the Statefulset as env. variable and from there into the script inside the initContainers.

I'll try this approach and if you like it, I could push it as a PR.

FraPazGal commented 3 years ago

Hi @mnmami,

I'm glad the issue was solved! And a PR would indeed be greatly appreciated!! Similar to what you propose, I was thinking about adding the list of loadBalancerIPs directly to the initContainer inside a {{ - if .Values.externalAccess.service.loadBalancerIPs }}.

mnmami commented 3 years ago

Hi @FraPazGal,

Here it is. Please have a look.

Should the same be done to the Arbiter Statefulset?

franklin432 commented 3 years ago

A fix should be applied to when NOT using LoadBalancer and only using NodePorts as well. I am using NodePorts instead of LoadBalancers for my setup and am getting the same error in regards to The server certificate does not match the remote host name. Hostname: {IP1} does not match SAN(s): {the list of CNs configured}. In my setup to connect to my database node from outside I need the primary and secondary node name/IP in addition to the nodePort. I was able to resolve this by including the Pod's IP, which you can simply get from kubectl describe , into the altNames of the certificate creation in the initContainer. I passed the following as an env variable in the generate-tls-certs initContainer AND the mongodb container:

env: 
    - name: MY_POD_IP
      valueFrom: 
          fieldRef: 
             fieldPath: status.podIP

I then added as an extra SAN in the altNames of the cert creation, $MY_POD_IP.

               DNS.1 = $svc 
               DNS.2 = $my_hostname 
               DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local 
               DNS.4 = localhost 
               DNS.5 = 127.0.0.1 
               DNS.6 = $MY_POD_IP

$MY_POD_IP represents the IP of the pod, which is the same IP that was showing up in the logs saying the IP did not match the SAN. With this, I am now able to start up the replicaset with no errors because the server certificate, which includes $MY_POD_IP as a SAN, now matches the host name. Also when I bash into the mongo primary pod and attempt to connect internally, I am able to connect using the Pod IP as the host. However when I attempt to connect to my cluster externally via a mongo client on my local desktop, the mongo connection (including the Pod IP as the host, the node port and TLS settings) does not work. The error seen:

"The server certificate does not match the remote host name, "attr":{"remoteHost": "nodeName.xxx.com", "certificateNames":"SAN(s):{the list of CNs configured SANs}.

In my environment, when attempting to connect externally, it instead requires the node name or node IP of the node that is hosting the mongodb-0 pod, to be specified as the host and not the actual IP of the pod as stated before. You can retrieve this from the kubectl describe command, under the Node section. In order to get each pod in the replicaset to contain this, I included it as an env variable just as I had done before with the IP env.

I passed the following as an env variable in the generate-tls-certs initContainer and the mongodb container:

env: 
    - name: MY_NODE_NAME
      valueFrom: 
          fieldRef: 
             fieldPath: spec.nodeName

kubectl describe gives me the name/IP of the Node that is hosting my mongo primary pod. I retrieve the other pods name/IP in my replicaset this way then attempt to connect to it with: ./mongo -u root -p mypassword --host , --tls --tlsCertificateKeyFile /tls/mongodb.pem --tlsCAFile /tls/mongodb-ca-cert

This works for me and I am able to connect externally because the server cert SAN now contains the podIP and the NodeName of the node hosting my mongodb-0 primary pod and the same for my mongodb-1 secondary pod.

FraPazGal commented 3 years ago

Hello!

Thanks for the PR @mnmami, it looks good! I would say that it may be not necessary to include the arbiters as we are not assigning a IP to the arbiter node.

And thank you @franklin432 for the input, it is true that we should also take into account the case when externalAccess.service.type=NodePort. Lets please continue this discussion on the PR!

mnmami commented 3 years ago

Hello,

@franklin432 thanks a lot for taking the time and explaining your approach. I would definitely return to it if we ever decide to use NodePort instead.

@FraPazGal thanks for your response regarding the Arbiter.

binnythomas-1989 commented 3 years ago

Hello Guys! I ran the helm chart from master, with external and i got the below error. It works when i disable tls.

On the values file added this related to tls

tls:
 ## Enable or disable MongoDB TLS Support 
  enabled: true

externalAccess:
  ## Enable Kubernetes external cluster access to MongoDB nodes
  ##
  enabled: true
  autoDiscovery:
    ## Enable external IP/ports auto-discovery
    ##
    enabled: true

{"t":{"$date":"2020-11-11T16:36:20.792+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn94","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"The server is configured to only allow SSL connections"},"remote":"127.0.0.1:48334","connectionId":94}}

I have a couple of questions.

Is this helm chart missing something? Or am i missing some configurations? Why do we need 3-4 loadbalancer per node? How do we access the loadbalancers with a single domain? Like
ExternalDNS(DomainCreation) ---> ServicePort or DNS -- > Loadbalancer Type --> Service Port

Please explain.

binnythomas-1989 commented 3 years ago

I tried the same helm chart without tls the pods did not fail. But got these errors on the logs

{"t":{"$date":"2020-11-12T09:09:57.489+00:00"},"s":"I",  "c":"REPL",     "id":21313,   "ctx":"initandlisten","msg":"Did not find local replica set configuration document at startup","attr":{"error":{"code":47,"codeName":"NoMatchingDocument","errmsg":"Did not find replica set configuration document in local.system.replset"}}}
{"t":{"$date":"2020-11-12T09:09:57.490+00:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2020-11-12T09:09:57.490+00:00"},"s":"I",  "c":"CONTROL",  "id":20711,   "ctx":"LogicalSessionCacheReap","msg":"Failed to reap transaction table","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}

{"t":{"$date":"2020-11-12T09:10:08.874+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn7","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"19790c83-c91e-48fa-8751-0e231baa7aef\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-12T09:10:18.863+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn12","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"ff7a0efb-6160-4a06-808c-5fca8e9b2fa5\") } ], $db: \"admin\" }"}}}

mnmami commented 3 years ago

@binnythomas-1989, as of now, the TLS encryption cannot be used with external access and dynamic (auto-discovered) IP addresses. As discussed in this issue and the following PR, we are suggesting to use static IP addresses. Can you obtain static IP addresses? Also, the number of replicaset members is up to you.

As per your other issue, could you share the command you are using and possibly custom values files. Although, that's a different issue than what is being discussed in this one.

binnythomas-1989 commented 3 years ago

I have retried using static ips from the pull request you created as you have suggested. I still get this errror

externalAccess:
  ## Enable Kubernetes external cluster access to MongoDB nodes
  ##
  enabled: true
  ## External IPs auto-discovery configuration
  ## An init container is used to auto-detect LB IPs or node ports by querying the K8s API
  ## Note: RBAC might be required
  ##
  autoDiscovery:
    ## Enable external IP/ports auto-discovery
    ##
    enabled: false
    ## Bitnami Kubectl image
    ## ref: https://hub.docker.com/r/bitnami/kubectl/tags/
    ##

  service:
    ## Service type. Allowed values: LoadBalancer or NodePort
    ##
    type: LoadBalancer
    ## Port used when service type is LoadBalancer
    ##
    port: 27017
    ## Array of load balancer IPs for each MongoDB node. Length must be the same as replicaCount
    ## Example:
    ## loadBalancerIPs:
    ##   - X.X.X.X
    ##   - Y.Y.Y.Y
    ##
    loadBalancerIPs: 
      - 10.xx.246.xx
      - 10.xx.246.xxx

ERROR:

{"t":{"$date":"2020-11-12T14:22:31.286+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn177","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"The server is configured to only allow SSL connections"},"remote":"127.0.0.1:58088","connectionId":177}}

Do i require to enable some configurations on mongodb for it to work?

FraPazGal commented 3 years ago

Hi @binnythomas-1989,

Could you please share your custom values.yaml and the steps you followed in your chart installation?

binnythomas-1989 commented 3 years ago

I have followed the following steps.

I have updated the values_production.yaml with the following values. the external ips where pre-created and added to the LoadBalancer_ips array.

Then ran the following command to install helm install code-signing ./ -f values-production.yaml -n code-signing-mig


image:
  registry: docker.io
  repository: bitnami/mongodb
  tag: 4.4.1-debian-10-r61
  pullPolicy: IfNotPresent

  debug: false

architecture: replicaset

useStatefulSet: false

auth:
  enabled: true
  rootPassword: "tuvwxyz"
  username: codesigning
  password: abcdef
  database: signer_service_audit
  replicaSetKey: "xxxxxxxxxxxxxxxxx"

tls:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/nginx
    tag: 1.19.4-debian-10-r13
    pullPolicy: IfNotPresent

replicaSetName: rs0

replicaSetHostnames: false

enableIPv6: false

directoryPerDB: false

systemLogVerbosity: 0
disableSystemLog: false

configuration: ""

initdbScripts: {}

extraFlags: []

extraEnvVars: []

annotations: {}

labels: {}

replicaCount: 2

strategyType: RollingUpdate

podManagementPolicy: OrderedReady

affinity: {}

nodeSelector: {}

tolerations: []

podLabels: {}

podAnnotations: {}

podSecurityContext:
  enabled: true
  fsGroup: 1001
  sysctls: []

containerSecurityContext:
  enabled: true
  runAsUser: 1001
  runAsNonRoot: true

resources:
  limits: 
    cpu: 2000m
    memory: 2Gi
  requests:
    cpu: 2000m
    memory: 2Gi

livenessProbe:
  enabled: true
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1
readinessProbe:
  enabled: true
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 6
  successThreshold: 1

customLivenessProbe: {}

customReadinessProbe: {}

initContainers: {}

sidecars: {}

extraVolumeMounts: []
extraVolumes: []

pdb:
  create: true
  minAvailable: 1

persistence:
  enabled: true
  storageClass: "standard"
  accessModes:
    - ReadWriteOnce
  size: 30Gi
  annotations: {}
  mountPath: /bitnami/mongodb
  subPath: ""

service:
  type: ClusterIP
  port: 27017
  portName: mongodb
  nodePort: ""
  externalIPs: []

  loadBalancerSourceRanges: []
  annotations: {}

externalAccess:
  enabled: true
  autoDiscovery:
    enabled: false
    image:
      registry: docker.io
      repository: bitnami/kubectl
      tag: 1.18.10-debian-10-r21
      pullPolicy: IfNotPresent
      pullSecrets: []
    resources:
      limits: {}
      requests: {}
  service:
    type: LoadBalancer
    port: 27017
    loadBalancerIPs: 
      - xx.58.xxx.xx5
      - xx.58.xx.xx0
    loadBalancerSourceRanges: []
    nodePorts: []
    annotations:
      external-dns.alpha.kubernetes.io/hostname: codesign-test-mongodb.eu02.dsg.arm.com

arbiter:
  enabled: true

  configuration: ""

  extraFlags: []

  extraEnvVars: []

  annotations: {}

  labels: {}

  affinity: {}

  nodeSelector: {}

  tolerations: []

  podLabels: {}

  podAnnotations: {}

  podSecurityContext:
    enabled: true
    fsGroup: 1001
    sysctls: []

  containerSecurityContext:
    enabled: true
    runAsUser: 1001

  resources:
    limits:
      cpu: 1000m
      memory: 1Gi
    requests:
      cpu: 1000m
      memory: 1Gi

  livenessProbe:
    enabled: true
    initialDelaySeconds: 30
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 6
    successThreshold: 1
  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 6
    successThreshold: 1

  customLivenessProbe: {}

  customReadinessProbe: {}

  initContainers: {}

  sidecars: {}

  extraVolumeMounts: []
  extraVolumes: []

  pdb:
    create: false
    minAvailable: 1

serviceAccount:
  create: true

rbac:
  create: true

volumePermissions:
  enabled: false
  image:
    registry: docker.io
    repository: bitnami/minideb
    tag: buster
    pullPolicy: Always
    pullSecrets: []
  resources:
    limits: {}
    requests: {}
  securityContext:
    runAsUser: 0

metrics:
  enabled: true
  image:
    registry: docker.io
    repository: bitnami/mongodb-exporter
    tag: 0.11.2-debian-10-r39
    pullPolicy: IfNotPresent

  extraFlags: ""

  extraUri: ""

  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 500m
      memory: 512Mi

  service:
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "{{ .Values.metrics.service.port }}"
      prometheus.io/path: "/metrics"
    type: ClusterIP
    port: 9216

  livenessProbe:
    enabled: true
    initialDelaySeconds: 15
    periodSeconds: 5
    timeoutSeconds: 5
    failureThreshold: 3
    successThreshold: 1
  readinessProbe:
    enabled: true
    initialDelaySeconds: 5
    periodSeconds: 5
    timeoutSeconds: 1
    failureThreshold: 3
    successThreshold: 1

  serviceMonitor:
    enabled: false

    interval: 30s
    additionalLabels: {}

  prometheusRule:
    enabled: false
    additionalLabels: {}
    rules: {}

binnythomas-1989 commented 3 years ago

any updates @FraPazGal

binnythomas-1989 commented 3 years ago

I have managed to get it running but with errors on arbiter, but while trying to connect get the following error. I used kubectl cp to download from /cacerts folder

bintho01@C02YH0BNJHD5 mongodb-test (master) $ kubectl get pods
NAME                             READY   STATUS             RESTARTS   AGE
code-signing-mongodb-0           2/2     Running            0          63m
code-signing-mongodb-1           2/2     Running            1          63m
code-signing-mongodb-arbiter-0   0/1     CrashLoopBackOff   10         63m
bintho01@C02YH0BNJHD5 mongodb-test (master) $

bintho01@C02YH0BNJHD5 mongodb-test (master) $ kubectl logs code-signing-mongodb-arbiter-0
mongodb 12:11:12.17
mongodb 12:11:12.17 INFO  ==> ** Starting MongoDB setup **
mongodb 12:11:12.19 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 12:11:12.21 INFO  ==> Initializing MongoDB...
mongodb 12:11:12.24 INFO  ==> Deploying MongoDB from scratch...
mongodb 12:11:13.31 INFO  ==> Creating users...
mongodb 12:11:13.31 INFO  ==> Users created
mongodb 12:11:13.31 INFO  ==> Writing keyfile for replica set authentication...
mongodb 12:11:13.35 INFO  ==> Configuring MongoDB replica set...
mongodb 12:11:13.38 INFO  ==> Stopping MongoDB...
mongodb 12:11:15.71 INFO  ==> Trying to connect to MongoDB server code-signing-mongodb-0.code-signing-mongodb-headless.code-signing-mig.svc.cluster.local...
mongodb 12:11:15.74 INFO  ==> Found MongoDB server listening at code-signing-mongodb-0.code-signing-mongodb-headless.code-signing-mig.svc.cluster.local:27017 !
mongodb 12:14:16.28 ERROR ==> Node code-signing-mongodb-0.code-signing-mongodb-headless.code-signing-mig.svc.cluster.local did not become available
mongodb 12:14:16.28 INFO  ==> Stopping MongoDB...

bintho01@C02YH0BNJHD5 mongodb-test (master) $ mongo --host 1x.58.xxx.xxx,1x.58.xxx.xx --tls --tlsCertificateKeyFile  mongodb.pem --tlsCAFile  mongodb-ca-cert
MongoDB shell version v4.2.6
connecting to: mongodb://1x.58.xxx.xx:27017,1x.58.xx.xxx:27017/?compressors=disabled&gssapiServiceName=mongodb
2020-11-16T12:07:22.494+0000 E  NETWORK  [js] SSL peer certificate validation failed: Certificate trust failure: CSSMERR_TP_CERT_SUSPENDED; connection rejected
2020-11-16T12:07:22.550+0000 E  NETWORK  [js] SSL peer certificate validation failed: Certificate trust failure: CSSMERR_TP_CERT_SUSPENDED; connection rejected
2020-11-16T12:07:22.551+0000 E  QUERY    [js] Error: couldn't connect to server 10.58.246.170:27017, connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: Certificate trust failure: CSSMERR_TP_CERT_SUSPENDED; connection rejected :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2020-11-16T12:07:22.555+0000 F  -        [main] exception: connect failed
2020-11-16T12:07:22.555+0000 E  -        [main] exiting with code 1
bintho01@C02YH0BNJHD5 mongodb-test (master) $

Pod 0 logs

{"t":{"$date":"2020-11-16T12:09:12.893+00:00"},"s":"I",  "c":"CONTROL",  "id":20711,   "ctx":"LogicalSessionCacheReap","msg":"Failed to reap transaction table","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}}
{"t":{"$date":"2020-11-16T12:14:12.893+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"rs0","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}}
{"t":{"$date":"2020-11-16T12:14:12.893+00:00"},"s":"I",  "c":"NETWORK",  "id":4333208, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host selection timeout","attr":{"replicaSet":"rs0","error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}}
{"t":{"$date":"2020-11-16T12:14:12.893+00:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}}
{"t":{"$date":"2020-11-16T12:14:12.893+00:00"},"s":"I",  "c":"CONTROL",  "id":20711,   "ctx":"LogicalSessionCacheReap","msg":"Failed to reap transaction table","attr":{"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}}

binnythomas-1989 commented 3 years ago

I have fixed the connection issues too. The problem was that the mongo client should be of the similar version to the mongo server. The current error outstanding is the arbiter failing with the following errors seen on the primary node as pasted on the above thread. Kindly help!

"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"primary\" } for set rs0"}

bintho01@C02YH0BNJHD5 mongodb-test (master) $ mongo --host 10.xx.2xx.xxx --tls --tlsCAFile mongodb-ca-cert --tlsCertificateKeyFile mongodb.pem
**MongoDB shell version v4.4.1**
connecting to: mongodb://10.xx.2xx.xxx:27017/?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2020-11-16T14:28:32.128Z"},"s":"W",  "c":"NETWORK",  "id":23208,   "ctx":"js","msg":"You have an IP Address in the DNS Name field on your certificate. This formulation is deprecated."}
Implicit session: session { "id" : UUID("90ef1c94-d57e-4e02-ab06-15a7c9a9c28d") }
**MongoDB server version: 4.4.1**

bintho01@C02YH0BNJHD5 mongodb-test (master) $ mongo --host 10.xx.2xx.xxx --tls --tlsCAFile mongodb-ca-cert --tlsCertificateKeyFile mongodb.pem
MongoDB shell version v4.4.1
connecting to: mongodb://10.xx.2xx.xxx:27017/?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2020-11-16T14:47:57.158Z"},"s":"W",  "c":"NETWORK",  "id":23208,   "ctx":"js","msg":"You have an IP Address in the DNS Name field on your certificate. This formulation is deprecated."}
Implicit session: session { "id" : UUID("7526f5e0-ce8c-487c-a446-e3467609ebf7") }
MongoDB server version: 4.4.1
> conn = new Mongo("10.xx.2xx.xxx")
{"t":{"$date":"2020-11-16T14:48:01.724Z"},"s":"W",  "c":"NETWORK",  "id":23208,   "ctx":"js","msg":"You have an IP Address in the DNS Name field on your certificate. This formulation is deprecated."}
connection to 10.xx.2xx.xxx:27017
> db=conn.getDB("signer_service_audit")
signer_service_audit
> collection = db.getCollection("dev_code_signing_audit")
signer_service_audit.dev_code_signing_audit
> collection.find()
Error: error: {
    "ok" : 0,
    "errmsg" : "command find requires authentication",
    "code" : 13,
    "codeName" : "Unauthorized"
}
> db.auth("codesigning","xxxxxx")
1
> collection = db.getCollection("dev_code_signing_audit")
signer_service_audit.dev_code_signing_audit
> collection.find()
Error: error: {
    "topologyVersion" : {
        "processId" : ObjectId("5fb25eefbdb76ed3dccbc75b"),
        "counter" : NumberLong(2)
    },
    "ok" : 0,
    "errmsg" : "not master and slaveOk=false",
    "code" : 13435,
    "codeName" : "NotMasterNoSlaveOk"
}
>

Also found Pod 1 Logs

Continuous not authorized errors.

{"t":{"$date":"2020-11-16T15:14:10.352+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn8643","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"d8e18f47-849b-4821-8f0e-984ff7ee43d4\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-16T15:14:17.884+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn8646","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"796b506a-33aa-4e32-92a5-65e3134a441a\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-16T15:14:20.360+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn8649","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"7a15d15b-c52f-44f7-932a-63d6845dd74c\") } ], $db: \"admin\" }"}}}

FraPazGal commented 3 years ago

Hi @binnythomas-1989,

If I understood correctly, your current issue is that your arbiter pod is not connecting to the primary one because of a SSLHandshakeFailed, is that right? Are you including the arbiter hostname in the certificate? Also, could you please run the arbiter pod with image.debug=true and share its logs?

I'm not sure these errors are related to using external access, is your configuration working with externalAccess.enabled=false?

binnythomas-1989 commented 3 years ago

@FraPazGal Those errors happened when i had 2 nodes running. When i added 3 nodes instead with externalAccess.enabled=true. I dont see those errors.

Now i see the following errors.

Pod-0,Pod-1 and Pod-2 logs

{"t":{"$date":"2020-11-17T11:55:57.287+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn1733","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"172.26.71.244:65173","connectionId":1733}}
{"t":{"$date":"2020-11-17T11:55:57.817+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn1734","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"172.26.70.88:47847","connectionId":1734}}
{"t":{"$date":"2020-11-17T11:55:58.347+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn1735","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"no SSL certificate provided by peer; connection rejected"},"remote":"172.24.0.27:60585","connectionId":1735}}

Is there a way i could reach out to you? binnythomas@gowithusnow.co.uk this is my email id

Im able to connect to the mongo db and access the tables. But i would like to understand these errors on the logs and also understand if my certificates are valid?

Every 2.0s: kubectl get pods                                                                                                          C02YH0BNJHD5: Tue Nov 17 12:34:22 2020

NAME                            READY   STATUS    RESTARTS   AGE
code-signer-mongodb-0           2/2     Running   0          168m
code-signer-mongodb-1           2/2     Running   0          167m
code-signer-mongodb-2           2/2     Running   1          167m
code-signer-mongodb-arbiter-0   1/1     Running   0          168m

franklin432 commented 3 years ago

@binnythomas-1989 So lets get a few things understood first. You are trying to run this mongodb helm chart with tls enabled, external access enabled and the arbiter enabled correct? Currently this chart can run fine with TLS enabled and the arbiter enabled. As @mnmami said above, as of now, the TLS encryption cannot be used with external access and dynamic (auto-discovered) IP addresses. In his PR, he suggest to use static IP addresses to put in the loadBalancerIPs: which it looks like you have done in your config. However are you including that in the cert creation? By adding the static IPs to the certificate creation script under alt_name? The arbiter's name should also be added as well since it needs to have TLS running to connect to the TLS enabled cluster. The arbiter statefulset is already set up to handle TLS once TLS is enabled in values-production.yaml @mnmami PR proposed this idea to get it running if I am correct. Read his fix here and apply it to the cert creation section in the values-production.yaml file https://github.com/bitnami/charts/pull/4249/files

I am using this chart with external access enabled but am using NodePorts since i do not have a loadBalancer option in our environment. It seems to work fine for me with the proposed change/additions I mentioned in the beginning of this request somwhere in the messages.

binnythomas-1989 commented 3 years ago

@franklin432 thanks for the response. I found @mnmami change and thats how i implemented what i did till now.

To answer your questions this is how the initContainer looks

        - name: generate-tls-certs
          image: {{ include "mongodb.tls.image" . }} 
          imagePullPolicy: {{ .Values.tls.image.pullPolicy | quote }}
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: certs-volume
            mountPath: /certs/CAs
          - name: certs
            mountPath: /certs
          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless
              cp /certs/CAs/* /certs/
              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL
              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server certificate
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server certificate with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication. 
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF
        {{- end }}

I presume this would add the ips to the alt_names.

Questions: You are also saying apart from the ips we should add the arbiter service name to the alt_names?

Please refer to this link where @FraPazGal says otherwise https://github.com/bitnami/charts/issues/4182#issuecomment-725444858

Why is it that we should access the mongo client using the .pem and the ca-cert.?

Reason being i could see you do the concat of the cert and private key.

          openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
          #Signing the server certificate with the CA cert and key
          openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
          rm /certs/mongo.csr
          #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication. 
          cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem

binnythomas-1989 commented 3 years ago

Out of Topic here! We use the below link to connect to mongo. So im trying to understand how this would fall into place with the ca-file coming into the picture. https://www.codeproject.com/Tips/1163375/Using-Mongo-with-SSL-and-GO

Reason being: It Works with ca file:

bintho01@C02YH0BNJHD5 mongodb-test (master) $ mongo --host 10.xxx.2x6.xxx,10.xxx.2x0.xxx,10.xxx.2x3.xxx --tls --tlsCAFile mongodb-ca-cert --tlsCertificateKeyFile mongodb.pem
MongoDB shell version v4.4.1
connecting to: mongodb://10.xxx.2x6.xxx:27017,10.xxx.2x0.xxx:27017,10.xxx.2x3.xxx:27017/?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2020-11-17T13:31:42.917Z"},"s":"W",  "c":"NETWORK",  "id":23208,   "ctx":"js","msg":"You have an IP Address in the DNS Name field on your certificate. This formulation is deprecated."}
Implicit session: session { "id" : UUID("65289be8-dad6-4ecc-9067-a61ebf3cd2ca") }
MongoDB server version: 4.4.1
rs0:PRIMARY> exit
bye

WithoutCa file

Error:

bintho01@C02YH0BNJHD5 mongodb-test (master) $ mongo --host 10.xxx.2x6.xxx,10.xxx.2x0.xxx,10.xxx.2x3.xxx --tls --tlsCertificateKeyFile mongodb.pem
MongoDB shell version v4.4.1
connecting to: mongodb://10.xxx.2x6.xxx:27017,10.xxx.2x0.xxx:27017,10.xxx.2x3.xxx:27017/?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2020-11-17T13:37:32.120Z"},"s":"E",  "c":"NETWORK",  "id":23212,   "ctx":"js","msg":"SSL peer certificate validation failed; connection rejected","attr":{"error":"Certificate trust failure: CSSMERR_TP_NOT_TRUSTED"}}
{"t":{"$date":"2020-11-17T13:37:32.162Z"},"s":"E",  "c":"NETWORK",  "id":23212,   "ctx":"js","msg":"SSL peer certificate validation failed; connection rejected","attr":{"error":"Certificate trust failure: CSSMERR_TP_NOT_TRUSTED"}}
{"t":{"$date":"2020-11-17T13:37:32.199Z"},"s":"E",  "c":"NETWORK",  "id":23212,   "ctx":"js","msg":"SSL peer certificate validation failed; connection rejected","attr":{"error":"Certificate trust failure: CSSMERR_TP_NOT_TRUSTED"}}
Error: couldn't connect to server 10.58.246.200:27017, connection attempt failed: SSLHandshakeFailed: Certificate trust failure: CSSMERR_TP_NOT_TRUSTED :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6

binnythomas-1989 commented 3 years ago

@FraPazGal and @franklin432 So it seems you guys are right! We would require the arbiter host added to the alt_names as well for that error to disappear which is not part of @mnmami change or the current helm chart.

Would you be able to confirm. I have done some changes to the helm to get it working. If its already not a part of the helm should i raise a pull request?

franklin432 commented 3 years ago

@binnythomas-1989 Your intiContainer cert setup looks fine just as @mnmami has it. No, i was not saying apart from the IPs you should be adding the arbiter service name to the alt_names. I was saying the arbiter's statefulset.yaml file already contains the same intiContainer as the replicaSet statefulset, so when you have arbiter.enabled and tls.enabled set to true it will generate automatically a signed arbiter cert just as the rest of the pods, to connect to the TLS enabled cluster.

The cluster as a whole is set to "requireTLS" from the flag MONGODB_EXTRA_FLAGS located in /templates/replicaset/statefulset.yaml as well as in /templates/arbiter/statefuslet. Therefore when set it requires the CA cert file (mongodb-ca-cert) and the mongo signed certificate (mongodb.pem) to connect to. As a result, your mongo-client or even when you manually bash into the container will need that CA cert and signed Cert file in order to authenticate to it. The signed CA cert will also contain the SAN or alt_names you had specified in the initContainer. Without those certs it as you showed above, (mongo --host 10.xxx.2x6.xxx,10.xxx.2x0.xxx,10.xxx.2x3.xxx --tls --tlsCertificateKeyFile mongodb.pem) it will not allow you to connect. You can easily change this by changing the TLS mode as you see below. Setting it to preferTLS will both allow you to connect without specifying the .pem and the ca-cert at the same time it will also connect you if you presented the .pem and ca-cert.

Value | Description
-- | --
disabled | The server does not use TLS.
allowTLS | Connections between servers do not use TLS. For incoming connections, the server accepts both TLS and non-TLS.
preferTLS | Connections between servers use TLS. For incoming connections, the server accepts both TLS and non-TLS.
requireTLS | The server uses and accepts only TLS encrypted connections.

Make sure the templates/replicaset/statefulset.yaml, templates/arbiter/statefuslet and /templates/standalone/dep-sts.yaml file all contain the same parameters in the section of the generate-tls-cert initContainer.

binnythomas-1989 commented 3 years ago

@franklin432 so you mean i should not have added the mongo arbiter host to the alt_names of the replicaset statefulset?

Also when you say

templates/replicaset/statefulset.yaml, templates/arbiter/statefuslet and /templates/standalone/dep-sts.yaml file all contain the same parameters

What do you mean by parameters, the command section or the environment variables? If i suppose its the command section of the initContainer. Would i require to add the same logic of @mnmami over onto the arbiters statefulset?

mongodb replicaset command

          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless
              arbiter_svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-arbiter-headless
              cp /certs/CAs/* /certs/
              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = $MYPOD_ARBITER_NAME.$arbiter_svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.5 = localhost
              DNS.6 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL

              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server certificate
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server certificate with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication. 
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF

arbiter statefulset command

          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless

              cp /certs/CAs/* /certs/

              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              EOL

              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server cert
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server cert with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication.
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF

franklin432 commented 3 years ago

@binnythomas-1989 I'm not sure if @mnmami logic needs to be added to the arbiter statefulset. You will have to play with that and test it. But to be on the safe side until we get confirmation, I would go ahead and just include the following in each of the templates/replicaset/statefulset.yaml, templates/arbiter/statefuslet and /templates/standalone/dep-sts.yaml

- name: generate-tls-certs
          image: {{ include "mongodb.tls.image" . }} 
          imagePullPolicy: {{ .Values.tls.image.pullPolicy | quote }}
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: certs-volume
            mountPath: /certs/CAs
          - name: certs
            mountPath: /certs
          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless
              cp /certs/CAs/* /certs/
              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL
              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server certificate
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server certificate with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication. 
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF
        {{- end }}

Forgive my choice of words with the parameters. I simply meant the code from this initContainer above to just match all 3 statefulset.yaml files .

binnythomas-1989 commented 3 years ago

@franklin432 i have just updated the statefulset the way you had requested. Updated the commands to all the statefulsets mongo replicaset and mongo arbiter. But i get the below errors

pod-2

{"t":{"$date":"2020-11-18T11:30:37.157+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn25","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"e7c06fb8-f915-4f08-8ea4-0eb3db384f93\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-18T11:30:45.369+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn28","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"3357fbdc-1f75-4ff8-91b3-98b2e81338f3\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-18T11:30:47.140+00:00"},"s":"I",  "c":"ACCESS",   "id":20436,   "ctx":"conn31","msg":"Checking authorization failed","attr":{"error":{"code":13,"codeName":"Unauthorized","errmsg":"not authorized on admin to execute command { endSessions: [ { id: UUID(\"c47f812e-385b-4afb-b2e3-eef9afc48127\") } ], $db: \"admin\" }"}}}
{"t":{"$date":"2020-11-18T11:30:51.732+00:00"},"s":"I",  "c":"NETWORK",  "id":20125,   "ctx":"ReplCoordExtern-0","msg":"DBClientConnection failed to receive message","attr":{"connString":"10.58.246.215:27017","error":"SocketException: stream truncated"}}
{"t":{"$date":"2020-11-18T11:30:51.734+00:00"},"s":"I",  "c":"NETWORK",  "id":20121,   "ctx":"ReplCoordExtern-0","msg":"Reconnect attempt failed","attr":{"connString":"10.58.246.215:27017 failed","error":""}}
{"t":{"$date":"2020-11-18T11:35:51.732+00:00"},"s":"I",  "c":"CONNPOOL", "id":22572,   "ctx":"ReplCoordExternNetwork","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.xxx.2x6.xxx:27017","error":"ShutdownInProgress: Pool for 10.xxx.2x6.xxx:27017 has expired."}}

replicaset-mognodb

        - name: generate-tls-certs
          image: {{ include "mongodb.tls.image" . }} 
          imagePullPolicy: {{ .Values.tls.image.pullPolicy | quote }}
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: certs-volume
            mountPath: /certs/CAs
          - name: certs
            mountPath: /certs
          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless
              cp /certs/CAs/* /certs/
              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL

              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server certificate
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server certificate with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication. 
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF
        {{- end }}

arbiter

        - name: generate-client
          image: {{ include "mongodb.tls.image" . }}
          imagePullPolicy: {{ .Values.tls.image.pullPolicy | quote }}
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: certs-volume
            mountPath: /certs/CAs
          - name: certs
            mountPath: /certs
          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless

              cp /certs/CAs/* /certs/

              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL

              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server cert
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server cert with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication.
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF
        {{- end }}

@FraPazGal and @franklin432 please check what this error is about?

franklin432 commented 3 years ago

I'm not sure what this error means sorry. Unfortunately I cannot test on my end due to me not using LoadBalancers in our environment nor having the ability to do so. I have been testing with NodePorts and works. There was a change applied to the master bitnami/mongodb chart. Maybe the fix is in there?

binnythomas-1989 commented 3 years ago

@franklin432 a qq since you had asked to use the commands across all templates/replicaset/statefulset.yaml, templates/arbiter/statefuslet and /templates/standalone/dep-sts.yaml.

I would like to understand when i use a standalone architecture, do we use externalAccess. I suppose not.

Correct me if im wrong. If we dont use externalAccess. Then how would we expose the loadbalancer ip and get the certs to work?

Please find the first line of external-access-svc

{{- if and (eq .Values.architecture "replicaset") .Values.externalAccess.enabled }}

Im trying to implement standalone architecture with exposes loadbalancers ip with tls enabled. I would say the documentation is too hard to follow.

binnythomas-1989 commented 3 years ago

@FraPazGal would you be able to test this. Would be really helpful. https://github.com/bitnami/charts/issues/4182#issuecomment-729615908

We still get back to back errors and the arbiter fails. The helm chart doesn't look stable enough using it from master.

FraPazGal commented 3 years ago

Hello @binnythomas-1989,

I don't see what the error could be at first glance, let me try to reproduce it and come back with my findings.

Also, thanks to @franklin432 for helping out with this issue!

binnythomas-1989 commented 3 years ago

@franklin432 thanks a lot buddy. So i have found that we dont have the ssl validation errors if we just use the Lodabalancer ip and integrating @mnmami changes on the initContainer.

@FraPazGal we then used the following changes to the helm templates to enable dns names to the loadbalancer ips.

on the external-access-svc.yaml

{{- if and (eq .Values.architecture "replicaset") .Values.externalAccess.enabled }}
{{- $fullName := include "mongodb.fullname" . }}
{{- $replicaCount := .Values.replicaCount | int }}
{{- $root := . }}

{{- range $i, $e := until $replicaCount }}
{{- $targetPod := printf "%s-%d" (printf "%s" $fullName) $i }}
---
apiVersion: v1
kind: Service
metadata:
  name: {{ $fullName }}-{{ $i }}-external
  namespace: {{ include "mongodb.namespace" $ }}
  labels: {{- include "common.labels.standard" $ | nindent 4 }}
    app.kubernetes.io/component: mongodb
    pod: {{ $targetPod }}
  {{- if $root.Values.externalAccess.service.loadBalancerHosts }}
  annotations: 
    external-dns.alpha.kubernetes.io/hostname: {{ index $root.Values.externalAccess.service.loadBalancerHosts $i }}
  {{- end }}
  {{- if $root.Values.externalAccess.service.annotations }}
  annotations: {{- include "common.tplvalues.render" ( dict "value" $root.Values.externalAccess.service.annotations "context" $) | nindent 4 }}
  {{- end }}

spec:
  type: {{ $root.Values.externalAccess.service.type }}
  {{- if eq $root.Values.externalAccess.service.type "LoadBalancer" }}
  {{- if not (empty $root.Values.externalAccess.service.loadBalancerIPs) }}
  loadBalancerIP: {{ index $root.Values.externalAccess.service.loadBalancerIPs $i }}
  {{- end }}
  {{- if $root.Values.externalAccess.service.loadBalancerSourceRanges }}
  loadBalancerSourceRanges: {{- toYaml $root.Values.externalAccess.service.loadBalancerSourceRanges | nindent 4 }}
  {{- end }}
  {{- end }}
  publishNotReadyAddresses: true
  ports:
    - name: {{ $root.Values.service.portName }}
      port: {{ $root.Values.externalAccess.service.port }}
      {{- if not (empty $root.Values.externalAccess.service.nodePorts) }}
      nodePort: {{ index $root.Values.externalAccess.service.nodePorts $i }}
      {{- else }}
      nodePort: null
      {{- end }}
      targetPort: mongodb
  selector: {{- include "common.labels.matchLabels" $ | nindent 4 }}
    app.kubernetes.io/component: mongodb
    statefulset.kubernetes.io/pod-name: {{ $targetPod }}
---
{{- end }}
{{- end }}

on the values-production.yaml add

    loadBalancerIPs: 
      - 1.2.3.4
      - 1.2.3.5
      - 1.2.3.6

    loadBalancerHosts:
      - codesign-0-mongodb.abc.com
      - codesign-1-mongodb.abc.com
      - codesign-2-mongodb.abc.com

We updated the arbiters and mogodb replicasets initContainer have the following

        - name: generate-client
          image: {{ include "mongodb.tls.image" . }}
          imagePullPolicy: {{ .Values.tls.image.pullPolicy | quote }}
          env:
            - name: MY_POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: certs-volume
            mountPath: /certs/CAs
          - name: certs
            mountPath: /certs
          command:
            - sh
            - "-c"
            - |
              /bin/bash <<'EOF'
              my_hostname=$(hostname)
              svc=$(echo -n "$my_hostname" | sed s/-[0-9]*$//)-headless

              cp /certs/CAs/* /certs/

              cat >/certs/openssl.cnf <<EOL
              [req]
              req_extensions = v3_req
              distinguished_name = req_distinguished_name
              [req_distinguished_name]
              [ v3_req ]
              basicConstraints = CA:FALSE
              keyUsage = nonRepudiation, digitalSignature, keyEncipherment
              subjectAltName = @alt_names
              [alt_names]
              DNS.1 = $svc
              DNS.2 = $my_hostname
              DNS.3 = $my_hostname.$svc.$MY_POD_NAMESPACE.svc.cluster.local
              DNS.4 = localhost
              DNS.5 = 127.0.0.1
              {{- if .Values.externalAccess.service.loadBalancerIPs }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerIPs }}
              IP.{{ $key }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              {{- if .Values.externalAccess.service.loadBalancerHosts }}
              {{- range $key, $val := .Values.externalAccess.service.loadBalancerHosts }}
              DNS.{{ add $key 6 }} = {{ $val | quote }}
              {{- end }}
              {{- end }}
              EOL

              export RANDFILE=/certs/.rnd && openssl genrsa -out /certs/mongo.key 2048
              #Create the client/server cert
              openssl req -new -key /certs/mongo.key -out /certs/mongo.csr -subj "/C=US/O=My Organisations/OU=IT/CN=$my_hostname" -config /certs/openssl.cnf
              #Signing the server cert with the CA cert and key
              openssl x509 -req -in /certs/mongo.csr -CA /certs/mongodb-ca-cert -CAkey /certs/mongodb-ca-key -CAcreateserial -out /certs/mongo.crt -days 3650 -extensions v3_req -extfile /certs/openssl.cnf
              rm /certs/mongo.csr
              #Concatenate to a pem file for use as the client PEM file which can be used for both member and client authentication.
              cat /certs/mongo.crt /certs/mongo.key > /certs/mongodb.pem
              cd /certs/
              shopt -s extglob
              rm -rf !(mongodb-ca-cert|mongodb.pem|CAs|openssl.cnf)
              EOF
        {{- end }}
      containers:

With the ips it runs perfectly fine and we tested it. But when adding a domain name using external dns is when we again see the ssl validation errors as below.

e failure"},"remote":"172.26.70.88:40961","connectionId":265}}
{"t":{"$date":"2020-11-19T12:43:46.345+00:00"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"conn266","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: certificate signature failure"}}
{"t":{"$date":"2020-11-19T12:43:46.345+00:00"},"s":"I",  "c":"NETWORK",  "id":22988,   "ctx":"conn266","msg":"Error receiving request from client. Ending connection from remote","attr":{"error":{"code":141,"codeName":"SSLHandshakeFailed","errmsg":"SSL peer certificate validation failed: certificate signature failure"},"remote":"172.26.75.250:54347","connectionId":266}}

Do let us know what we did is a possibility? How could we map the ips to a host? @FraPazGal

Off topic, the standalone architecture can not be used with externalAccess as mentioned on the the previous thread.

FraPazGal commented 3 years ago

Hi @binnythomas-1989,

I have been looking into this and your case may not be contemplated with the current logic. AFAIK, the error may not be related with tls but with using hostnames with externalAccess. I was also able to connect using IPs but it didn't using hostnames. I configured the loadBalancerIPs and used the following ingress:

ingress.yaml

``` $ helm template apiVersion: extensions/v1beta1 kind: Ingress metadata: name: mongodb labels: app.kubernetes.io/name: mongodb helm.sh/chart: mongodb-10.0.5 app.kubernetes.io/instance: mongodb app.kubernetes.io/managed-by: Helm app.kubernetes.io/component: mongodb annotations: spec: rules: - host: mongo0.domainname.com http: paths: - path: / backend: serviceName: mongodb-0-external servicePort: 27017 - host: mongo1.domainname.com http: paths: - path: / backend: serviceName: mongodb-1-external servicePort: 27017 ```

The problem may be in the database setup script created in scripts-configmap.yaml:

...
setup.sh: |-
    #!/bin/bash

    {{- if .Values.externalAccess.enabled }}
    {{- if eq .Values.externalAccess.service.type "LoadBalancer" }}
    {{- if .Values.externalAccess.autoDiscovery.enabled }}
    export MONGODB_ADVERTISED_HOSTNAME="$(<${SHARED_FILE})"
    {{- else }}
    ID="${MY_POD_NAME#"{{ $fullname }}-"}"
    export MONGODB_ADVERTISED_HOSTNAME=$(echo '{{ .Values.externalAccess.service.loadBalancerIPs }}' | tr -d '[]' | cut -d ' ' -f "$(($ID + 1))")
    {{- end }}
    {{- else if eq .Values.externalAccess.service.type "NodePort" }}
    {{- if .Values.externalAccess.service.domain }}
    export MONGODB_ADVERTISED_HOSTNAME={{ .Values.externalAccess.service.domain }}
    {{- else }}
    export MONGODB_ADVERTISED_HOSTNAME=$(curl -s https://ipinfo.io/ip)
    {{- end }}
...

Mongodb needs to configure it advertised hostname, when externalAccess is enabled this env's values are the IPs (existing or auto-discovered) or the domain if we are using NodePort. Adding your loadBalancerHosts parameter as the value, like is done with .Values.externalAccess.service.loadBalancerIPs may be a solution. Could you confirm whether you are able to connect using hostnames with tls.enabled=false and if changing the values of MONGODB_ADVERTISED_HOSTNAME would help?

stale[bot] commented 3 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

stale[bot] commented 3 years ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

kodeine commented 2 years ago

@FraPazGal you are right, i am getting this error

mongodb 03:22:23.96 WARN  ==> Problem initiating replica set
            request: rs.initiate({"_id":"rs0", "members":[{"_id":0,"host":"174.138.119.151:27017","priority":5}]})
            response: MongoDB shell version v4.4.13
connecting to: mongodb://127.0.0.1:27017/admin?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2022-03-22T03:22:23.917Z"},"s":"W",  "c":"NETWORK",  "id":23237,   "ctx":"js","msg":"You have an IP Address in the DNS Name field on your certificate. This formulation is deprecated."}
Implicit session: session { "id" : UUID("4b586d6d-f3e3-410f-9938-c4ad9b844b79") }
MongoDB server version: 4.4.13
{
    "ok" : 0,
    "errmsg" : "No host described in new configuration with {version: 1, term: 0} for replica set rs0 maps to this node",
    "code" : 93,
    "codeName" : "InvalidReplicaSetConfig"
}
bye

i am using following config for external access. also this all works fine without tls.

tls:
  enabled: true
  autoGenerated: true
externalAccess:
  enabled: true
  autoDiscovery:
    enabled: true
  service:
    type: LoadBalancer
    port: 27017

Can we please re-open this issue?