TLS Connection not possible with node-sdk

[Eyalm321]

Describe the bug: Configuring Milvus in 1 or 2 way TLS doesnt work, it returns:

[Nest] 4549 - 07/03/2024, 6:20:19 PM ERROR [MilvusService] Error initializing Milvus Client: error:1E08010C:DECODER routines::unsupported Error: error:1E08010C:DECODER routines::unsupported at setKey (node:internal/tls/secure-context:93:11) at configSecureContext (node:internal/tls/secure-context:204:7) at createSecureContext (node:_tls_common:116:3) at Object.createSsl (/app/node_modules/@grpc/grpc-js/src/channel-credentials.ts:131:46) at new BaseClient (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/grpc/BaseClient.ts:206:34) at new Collection (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Collection.js:21:9) at new Data (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Data.js:17:9) at new Partition (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Partition.js:17:9) at new User (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/User.js:17:9) at new GRPCClient (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/grpc/GrpcClient.ts:60:5) [Nest] 4549 - 07/03/2024, 6:20:19 PM ERROR [ExceptionHandler] error:1E08010C:DECODER routines::unsupported Error: error:1E08010C:DECODER routines::unsupported at setKey (node:internal/tls/secure-context:93:11) at configSecureContext (node:internal/tls/secure-context:204:7) at createSecureContext (node:_tls_common:116:3) at Object.createSsl (/app/node_modules/@grpc/grpc-js/src/channel-credentials.ts:131:46) at new BaseClient (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/grpc/BaseClient.ts:206:34) at new Collection (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Collection.js:21:9) at new Data (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Data.js:17:9) at new Partition (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/Partition.js:17:9) at new User (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/grpc/User.js:17:9) at new GRPCClient (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/grpc/GrpcClient.ts:60:5)

Steps to reproduce:

1. Generate keys via OpenSSL1.1.1f/1.1.1k/3.0.2 with ./gen.sh code given in docs
2. Copy paste keys to milvus/configs/cert inside Milvus container and /tls/milvus/ inside node.js(Nest) project
3. Define Milvus.yaml tlsmode to 1 or 2 and configure path in client based on that

4. Run this:

   private initializeMilvusClient(): void { const address = this.configService.get('MILVUS_URL', 'localhost:19530'); const rootCertPath = this.configService.get('MILVUS_ROOT_CERT_PATH', './tls/milvus/ca.pem'); const privateKeyPath = this.configService.get('MILVUS_PRIVATE_KEY_PATH', './tls/milvus/server.key'); const certChainPath = this.configService.get('MILVUS_CERT_CHAIN_PATH', './tls/milvus/server.pem'); const serverName = this.configService.get('MILVUS_TLS_SERVER_NAME);

   this.milvusClient = new MilvusClient({
       address,
       tls: {
           rootCertPath,
           privateKeyPath,
           certChainPath,
           serverName,
       }
   });
   this.logger.log('Milvus Client initialized successfully.');

   }

I also tried public CA downloaded certificates from smallstep

Milvus-node-sdk version: @zilliz/milvus2-sdk-node@ 2.4.3

Milvus version: 2.4.5

[Eyalm321]

Update:

As I suspected after doing research on the issue , it was a formatting issue, but im back to getting the call cancelled after handshake:

/app/node_modules/@grpc/grpc-js/src/call.ts:82 const error = new Error(message); ^ Error: 1 CANCELLED: Call cancelled at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/src/call.ts:82:17) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client.ts:360:55) at /app/node_modules/@grpc/grpc-js/src/call-interface.ts:149:27 at Object.onReceiveStatus (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Grpc.ts:207:15) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48) at /app/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24 at processTicksAndRejections (node:internal/process/task_queues:77:11) for call at at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/src/client.ts:325:42) at ServiceClientImpl. (/app/node_modules/@grpc/grpc-js/src/make-client.ts:189:15) at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:37:21 at new Promise () at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:34:10 at Generator.next () at fulfilled (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:5:58) at processTicksAndRejections (node:internal/process/task_queues:95:5)

What I did this time was I create new const variables with the keys pasted directly and converted these to a buffer, I then edited the node sdk client parameters to accept a buffer so I dont have to rely on file convertion from Milvus. Ran it, this time it recognizes the key yet call cancelled.

[Eyalm321]

Update 2:

I fixed ./gen.sh from documentation and also added a script to pack the files together for easy copying:

!/bin/bash

Define necessary variables

PRIVATE_DIR="/etc/ssl/private" CSR_DIR="/etc/ssl/csr" CERTS_DIR="/etc/ssl/certs" OPENSSL_CONF="/etc/ssl/openssl.cnf" Country="US" State="Texas" Location="" Organization="" Organizational="" CommonName=""

Ensure necessary directories exist

mkdir -p $PRIVATE_DIR $CSR_DIR $CERTS_DIR

Generate CA private key if it doesn't exist

if [ ! -f $PRIVATE_DIR/cakey.pem ]; then echo "Generating CA private key" openssl genpkey -algorithm RSA -out $PRIVATE_DIR/cakey.pem fi

Generate CA certificate if it doesn't exist

if [ ! -f $PRIVATE_DIR/cacert.pem ]; then echo "Generating CA certificate" openssl req -new -x509 -key $PRIVATE_DIR/cakey.pem -out $PRIVATE_DIR/cacert.pem -days 3650 -subj "/C=$Country/ST=$State/L=$Location/O=$Organization/OU=$Organizational/CN=$CommonName" fi

Generate server key

echo "Generating server key" openssl genpkey -algorithm RSA -out $PRIVATE_DIR/server.key

Generate server CSR

echo "Generating server CSR" openssl req -new -key $PRIVATE_DIR/server.key -out $CSR_DIR/server.csr -subj "/C=$Country/ST=$State/L=$Location/O=$Organization/OU=$Organizational/CN=$CommonName" -config $OPENSSL_CONF

Sign server certificate

echo "Generating server certificate" openssl x509 -req -days 3650 -in $CSR_DIR/server.csr -out $CERTS_DIR/server.pem -CA $PRIVATE_DIR/cacert.pem -CAkey $PRIVATE_DIR/cakey.pem -CAcreateserial -extfile $OPENSSL_CONF -extensions v3_req

Generate client key

echo "Generating client key" openssl genpkey -algorithm RSA -out $PRIVATE_DIR/client.key

Generate client CSR

echo "Generating client CSR" openssl req -new -key $PRIVATE_DIR/client.key -out $CSR_DIR/client.csr -subj "/C=$Country/ST=$State/L=$Location/O=$Organization/OU=$Organizational/CN=$CommonName" -config $OPENSSL_CONF

Sign client certificate

echo "Generating client certificate" openssl x509 -req -days 3650 -in $CSR_DIR/client.csr -out $CERTS_DIR/client.pem -CA $PRIVATE_DIR/cacert.pem -CAkey $PRIVATE_DIR/cakey.pem -CAcreateserial -extfile $OPENSSL_CONF -extensions v3_req

echo "Certificates generated successfully"

and ./package-certs.sh

!/bin/bash

Define necessary variables

PRIVATE_DIR="/etc/ssl/private" CSR_DIR="/etc/ssl/csr" CERTSDIR="/etc/ssl/certs" timestamp=$(date +%Y%m%d%H%M%S) tarball="certs_${timestamp}.tar.gz"

Ensure necessary directories exist

mkdir -p $PRIVATE_DIR $CSR_DIR $CERTS_DIR

Find and package all .pem, .key, .csr, and .srl files from all directories

find $PRIVATE_DIR $CSR_DIR $CERTS_DIR -type f ( -name ".pem" -o -name ".key" -o -name ".csr" -o -name ".srl" ) | tar -czvf $tarball -T -

Output the name of the generated tarball

echo "Generated package: $tarball"

Move the tarball to the CERTS_DIR

mv $tarball $CERTS_DIR

Delete the packaged files

find $PRIVATE_DIR $CSR_DIR $CERTS_DIR -type f ( -name ".pem" -o -name ".key" -o -name ".csr" -o -name ".srl" ) -delete

echo "Packaged files deleted from all directories."

and now I'm getting

/app/node_modules/@grpc/grpc-js/src/call.ts:82 const error = new Error(message); ^ Error: 14 UNAVAILABLE: No connection established. Last error: self-signed certificate (2024-07-03T22:26:34.884Z) at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/src/call.ts:82:17) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client.ts:360:55) at /app/node_modules/@grpc/grpc-js/src/call-interface.ts:149:27 at Object.onReceiveStatus (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Grpc.ts:207:15) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48) at /app/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24 at processTicksAndRejections (node:internal/process/task_queues:77:11) at runNextTicks (node:internal/process/task_queues:64:3) at processImmediate (node:internal/timers:449:9) for call at at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/src/client.ts:325:42) at ServiceClientImpl. (/app/node_modules/@grpc/grpc-js/src/make-client.ts:189:15) at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:37:21 at new Promise () at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:34:10 at Generator.next () at fulfilled (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:5:58) at processTicksAndRejections (node:internal/process/task_queues:95:5)

how to make grpc trust self-signed certs?

[shanghaikid]

I will test it tomorrow.

[Eyalm321]

Ok so as it stands of now , I managed to connect to TLS but in an insecure way, what I did is I open acme.sh docker container and generated the required certificates, I then put them in the perspective folders and added certificates to trust store in each container, after all that it gave me this:

/app/node_modules/@grpc/grpc-js/src/call.ts:82 const error = new Error(message); ^ Error: 14 UNAVAILABLE: No connection established. Last error: unable to get issuer certificate (2024-07-05T01:55:26.330Z) at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/src/call.ts:82:17) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client.ts:360:55) at /app/node_modules/@grpc/grpc-js/src/call-interface.ts:149:27 at Object.onReceiveStatus (/app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Grpc.ts:207:15) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:458:34) at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48) at /app/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24 at processTicksAndRejections (node:internal/process/task_queues:77:11) for call at at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/src/client.ts:325:42) at ServiceClientImpl. (/app/node_modules/@grpc/grpc-js/src/make-client.ts:189:15) at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:37:21 at new Promise () at /app/node_modules/@zilliz/milvus2-sdk-node/milvus/utils/Function.ts:34:10 at Generator.next () at fulfilled (/app/node_modules/@zilliz/milvus2-sdk-node/dist/milvus/utils/Function.js:5:58) at processTicksAndRejections (node:internal/process/task_queues:95:5)

So what I did is added this variable to env:

NODE_TLS_REJECT_UNAUTHORIZED='0'

and it connects but I dont like it, I tried also NODE_EXTRA_CA_CERTS=/app/tls/milvus/ca.pem

but no success. as it is it seems like a node.js isolated issue

[shanghaikid]

First round test: Only change tls mode

common:
  security:
    tlsMode: 2

using this files https://github.com/milvus-io/milvus-sdk-node/tree/main/test/cert

it(`should create a grpc client with cert file successfully`, async () => {
    const mc = new MilvusClient({
      address: 'https://localhost:19530',
      tls: {
        rootCertPath: `test/cert/ca.pem`,
        privateKeyPath: `test/cert/client.key`,
        certChainPath: `test/cert/client.pem`,
        serverName: 'localhost',
      },
      ssl: true,
      logLevel: 'debug',
    });

    const healthy = await mc.checkHealth();
    expect(healthy.isHealthy).toEqual(true);
  });

everything works.

Next round, I will test the gen.sh

[Eyalm321]

Could you try a public domain for production use ? I'm sure it wouldve worked in localhost

[Eyalm321]

Ok, weirdest thing. I got it to work, but I had to set rootCA path to /etc/ssl/certs/ca-certificates.crt and it worked! BUT its not how it was intended, upon checking the handshake, the keys seems completely the same as the ones I tried putting it for ca, I'm confused why it wouldn't take it.

[Eyalm321]

after running a quick script to extract all certs from main cert file, I found the one and pasted into CA and it works. Been working on it since Sunday good riddance.

Script:

!/bin/ash

TARGET="target" CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt" CERTS_DIR="./certs"

mkdir -p $CERTS_DIR cd $CERTS_DIR

Extract individual certificates from the CA bundle

awk 'BEGIN {c=0} /-----BEGIN CERTIFICATE-----/ {c++} { print > ("cert" c ".pem") }' < $CA_BUNDLE

Test each certificate

for cert in cert*.pem; do echo "Testing with $cert..." RESULT=$(openssl s_client -connect $TARGET -CAfile $cert < /dev/null 2>&1)

if echo "$RESULT" | grep -q "Verify return code: 0 (ok)"; then
    echo "Success with $cert"
    echo "This certificate successfully verified the server's certificate:"
    echo "$cert"
    break
else
    echo "Failed with $cert"
fi

done

[shanghaikid]

Ok, weirdest thing. I got it to work, but I had to set rootCA path to /etc/ssl/certs/ca-certificates.crt and it worked! BUT its not how it was intended, upon checking the handshake, the keys seems completely the same as the ones I tried putting it for ca, I'm confused why it wouldn't take it.

About permissions?

[Eyalm321]

No, doesnt have anything to do with permissions, the ca cert they gave me didnt authenticate, but it was already in default ca folder

[shanghaikid]

I have written a document about how to enable TLS for milvus, although I feel its content is unrelated to your issue. https://github.com/milvus-io/milvus-sdk-node/tree/main/test/cert

Your issue is more related to generating CA scripts, which I have less knowledge about.

[Eyalm321]

Thank you, The only thing I could think of you can help me is refactor milvusClient class to take buffers OR path just to prevent formatting issues.

And then you could also add a comment to errors related to root CA file to check /etc/ssl/certs/ca-certificates.crt for the right cert.

[shanghaikid]

Hi there, the new version has been released, thanks. https://github.com/milvus-io/milvus-sdk-node/releases/tag/v2.4.4