opensearch-project / security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields
https://opensearch.org/docs/latest/security-plugin/index/
Apache License 2.0
197 stars 275 forks source link

[BUG] Error after updating security details #4238

Open malayh opened 7 months ago

malayh commented 7 months ago

What is the bug? I am trying to install opensearch (using the 2.13.0 docker image). The setup works with the demo security configs. I have done the following steps to update the security configs

Generate keys and certs Using the following makefile to generate keys.

keys/root-ca.key:
    mkdir -p keys;
    openssl genrsa -out keys/root-ca.key 2048;
keys/ca.crt: keys/root-ca.key
    openssl req -new -x509 -sha256 -key keys/root-ca.key -out keys/ca.crt -days 730 -subj "/CN=ca.local";

keys/admin.key:
    mkdir -p keys;
    openssl genrsa -out keys/admin-temp.key 2048;
    openssl pkcs8 -inform PEM -outform PEM -in keys/admin-temp.key -topk8 -nocrypt -v1 PBE-SHA1-3DES -out keys/admin.key
    rm keys/admin-temp.key; 
keys/admin.crt: keys/admin.key keys/ca.crt keys/root-ca.key
    openssl req -new -key keys/admin.key -out keys/admin.csr -subj "/CN=admin";
    openssl x509 -req -in keys/admin.csr -CA keys/ca.crt -CAkey keys/root-ca.key -CAcreateserial -sha256 -out keys/admin.crt -days 730;
    rm keys/admin.csr;

keys/tls.key:
    openssl genrsa -out keys/tls-temp.key 2048;
    openssl pkcs8 -inform PEM -outform PEM -in keys/tls-temp.key -topk8 -nocrypt -v1 PBE-SHA1-3DES -out keys/tls.key
    rm keys/tls-temp.key;
keys/tls.crt: keys/tls.key keys/ca.crt keys/root-ca.key
    openssl req -new -key keys/tls.key -out keys/tls.csr -subj "/CN=node";
    openssl x509 -req -in keys/tls.csr -CA keys/ca.crt -CAkey keys/root-ca.key -CAcreateserial -sha256 -out keys/tls.crt -days 730;
    rm keys/tls.csr;
removeoldkeys:
    rm -rf keys;
makekeys: removeoldkeys keys/admin.key keys/admin.crt keys/tls.key keys/tls.crt keys/ca.crt
    @echo "Keys are generated.";

Update opensearch.yml as follows

cluster:
  name: opensearch
  initial_cluster_manager_nodes:
    - opensearch-manager-0
http:
  port: 9200
transport:
  port: 9300
path:
  data: /usr/share/opensearch/data
network:
  host: opensearch-manager-0.testopensearch-opensearch-manager-hl.testopensearch.svc.cluster.local
  publish_host: opensearch-manager-0.testopensearch-opensearch-manager-hl.testopensearch.svc.cluster.local
  bind_host: 0.0.0.0
node:
  name: opensearch-manager-0
  max_local_storage_nodes: 1
  roles:
    - cluster_manager
discovery:
  seed_hosts:
    - testopensearch-opensearch-data-hl.testopensearch.svc.cluster.local
    - testopensearch-opensearch-manager-hl.testopensearch.svc.cluster.local
  initial_state_timeout: 10m
plugins:
  security:
    disabled: false

plugins.security.ssl.transport.pemcert_filepath: tls.crt
plugins.security.ssl.transport.pemkey_filepath: tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls.crt
plugins.security.ssl.http.pemkey_filepath: tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: ca.crt
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn: ['CN=admin']
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [all_access, security_rest_api_access]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices:
    - .plugins-ml-agent
    - .plugins-ml-config
    - .plugins-ml-connector
    - .plugins-ml-controller
    - .plugins-ml-model-group
    - .plugins-ml-model
    - .plugins-ml-task
    - .plugins-ml-conversation-meta
    - .plugins-ml-conversation-interactions
    - .plugins-ml-memory-meta
    - .plugins-ml-memory-message
    - .plugins-ml-stop-words
    - .opendistro-alerting-config
    - .opendistro-alerting-alert*
    - .opendistro-anomaly-results*
    - .opendistro-anomaly-detector*
    - .opendistro-anomaly-checkpoints
    - .opendistro-anomaly-detection-state
    - .opendistro-reports-*
    - .opensearch-notifications-*
    - .opensearch-notebooks
    - .opensearch-observability
    - .ql-datasources
    - .opendistro-asynchronous-search-response*
    - .replication-metadata-store
    - .opensearch-knn-models
    - .geospatial-ip2geo-data*
    - .plugins-flow-framework-config
    - .plugins-flow-framework-templates
    - .plugins-flow-framework-state
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.nodes_dn:
  - 'CN=node'

Ran security_admin.sh as follows:

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -icl -nhnv \
-cd "/usr/share/opensearch/config/opensearch-security" \
-key "/usr/share/opensearch/config/kirk-key.pem" \
-cert "/usr/share/opensearch/config/kirk.pem" \
-cacert "/usr/share/opensearch/config/root-ca.pem"

Following is the error I am getting after that

[2024-04-12T10:27:59,220][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-manager-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:310) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1445) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1864) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]
        ... 27 more
[2024-04-12T10:27:59,222][WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-manager-0] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9200, remoteAddress=/[0:0:0:0:0:0:0:1]:44858}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.107.Final.jar:4.1.107.Final]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:310) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1445) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        ... 16 more
Caused by: javax.crypto.BadPaddingException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1864) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:239) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:196) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:159) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:310) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1445) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.107.Final.jar:4.1.107.Final]
        ... 16 more

What am I missing?

Possible related issue: https://github.com/opensearch-project/security/issues/3299

stephen-crawford commented 7 months ago

[Triage] Hi @malayh, can you confirm whether this behavior was consistent across multiple attempts?

The issue you linked is definitely similar and a really challenging one we have faced for some time. That said, it tends to be intermittent so if you are seeing this behavior consistently you may be able to submit a PR to help fix it!

malayh commented 6 months ago

@scrawfor99 It has been consistent across attempts.

Although I got it working with a different approach.

Here is my current opensearch.yml. With DISABLE_INSTALL_DEMO_CONFIG=true

cluster:
  name: "opensearch"
path:
  data: /usr/share/opensearch/data
network:
  bind_host: 0.0.0.0
node:
  max_local_storage_nodes: 1
discovery:
  seed_hosts:
    - opensearch-master-0.opensearch-cluster-master-headless.logging.svc.cluster.local
    - opensearch-data-0.opensearch-cluster-data-headless.logging.svc.cluster.local
    - opensearch-data-1.opensearch-cluster-data-headless.logging.svc.cluster.local
plugins:
  security:
    disabled: false
    nodes_dn:
      - CN=opensearch
    ssl:
      transport:
        pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
        pemkey_filepath: /usr/share/opensearch/config/c`erts/tls.key
        pemtrustedcas_filepath: /usr/share/opensearch/config/certs/ca.crt
        enforce_hostname_verification: false
      http:
        enabled: true
        pemcert_filepath: /usr/share/opensearch/config/certs/tls.crt
        pemkey_filepath: /usr/share/opensearch/config/certs/tls.key
        pemtrustedcas_filepath: /usr/share/opensearch/config/certs/ca.crt
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=admin
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]

And since my setup is in k8s, I am using cert-manager to create the certificates and follow, then mounting the files

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch
  namespace: {{ .Release.Namespace }}
spec:
  dnsNames:
  - "localhost"
  - "*.{{ .Release.Namespace }}.svc.cluster.local"
  - "*.{{ .Release.Namespace }}.pod.cluster.local"
  commonName: "opensearch"
  issuerRef:
    kind: ClusterIssuer
    name: local-ca-issuer
  secretName: cert-manager
  privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048

After that I use securtiry_admin.sh to create the internal_user.yaml and config.yaml, which works perfectly


Here is my current theory:

stephen-crawford commented 6 months ago

[Triage] Hi @malayh thank you for following up. It sounds like you were able to find a workaround (yay!), but that the issue still occurs. The demo_installer would be handled under the Kubernetes/Helm repo for since there are some processes they handled around how the cluster comes up. I would recommend filing an issue over there and linking back to this one.

In the meantime, someone will try to review these configurations and see if anything stands out.

stephen-crawford commented 6 months ago

[Triage] Marking as triaged so that someone takes a look at the configuration and closes this issue assuming nothing stands out.