search

minio / minio-js

MinIO Client SDK for Javascript
https://docs.min.io/docs/javascript-client-quickstart-guide.html
Apache License 2.0
955 stars 277 forks source link

Incompatibility between formio (minio.js) and ECS S3 #1206

Closed danileau closed 1 year ago

danileau commented 1 year ago

Dear Minio Community

I'm trying to connect FormIO via minio.js with our selfhosted Dell ECS S3 Service - which is not working. I've observed the following constellation: When I enter the value "https://ecs-s302.xy.test.com/" as our internal S3 endpoint I get the error from minio.js: 

12:03:42 0|pdf-server | You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
12:03:42 0|pdf-server | InvalidEndpointError: Invalid endPoint :"https://ecs-s302.xy.test.com/"
12:03:42 0|pdf-server | at new Client (/src/node_modules/minio/dist/main/minio.js:116:13)"

If I now correct this to "ecs-s302.xy.test.com", I get the following error: 

11:19:25 0|pdf-server | Error: Parse Error: Expected HTTP/
11:19:25 0|pdf-server | at Socket.socketOnData (node:_http_client:542:22)
11:19:25 0|pdf-server | at Socket.emit (node:events:513:28)
11:19:25 0|pdf-server | at Socket.emit (node:domain:489:12)

After tracing the issue in the minio.js file, I find the following section in helpers.js. Theoretically, it would solve our problem if an explicit use of "http://" or "https://" was allowed / returned true – if found and placed before search for the alphanumeric-characters.

function isValidEndpoint(endpoint) {
  return isValidDomain(endpoint) || isValidIP(endpoint);
} // isValidDomain - true if input host is a valid domain.

function isValidDomain(host) {
  if (!isString(host)) return false; // See RFC 1035, RFC 3696.

  if (host.length === 0 || host.length > 255) {
    return false;
  } // Host cannot start or end with a '-'

  if (host[0] === '-' || host.slice(-1) === '-') {
    return false;
  } // Host cannot start or end with a '_'

  if (host[0] === '_' || host.slice(-1) === '_') {
    return false;
  } // Host cannot start with a '.'

  if (host[0] === '.') {
    return false;
  }

  var alphaNumerics = '`~!@#$%^&*()+={}[]|\\"\';:><?/'.split(''); // All non alphanumeric characters are invalid.

  for (var i in alphaNumerics) {
    if (host.indexOf(alphaNumerics[i]) > -1) {
      return false;
    }
  } // No need to regexp match, since the list is non-exhaustive.
  // We let it be valid and fail later.

  return true;
} // Probes contentType using file extensions.

What do you think about this? Could this be an improvement for also other of your customers? Would this be possible to adapt on the minio-side, or do I have to implement this myself, “break” your code and thus adapt it again and again with every future update or implement some kind of a proxy?

Thank you very much for your opinion, response and kind regards Danileau

kannappanr commented 1 year ago

Can you please paste a self contained example here?

danileau commented 1 year ago

Hey @kannappanr - of course You can validate the described Issue with the following steps:

## Create the Docker image
vi Dockerfile

## Base the image on the official MinIO Docker image
FROM minio/minio

# Set environment variables for the MinIO access key and secret
ENV MINIO_ACCESS_KEY=minio
ENV MINIO_SECRET_KEY=minio123

# Start MinIO as a server
CMD ["server", "/data"]

# -------------------------------------------

## Build the Docker-Image
docker build -t custom-minio .

# -------------------------------------------

## Start the Image and setup the example Bucket
vi start-minio.sh

#!/bin/bash
# Start the MinIO container
docker run -d --name minio_instance -p 9000:9000 custom-minio
# Wait a few seconds to ensure that MinIO has started
sleep 10
# Install the MinIO client 'mc'
wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
# Configure 'mc' for the local MinIO server
./mc alias set myminio http://127.0.0.1:9000 minio minio123
# Create a bucket
./mc mb myminio/mybucket
# Print the access token and secret
echo "Access Key: minio"
echo "Secret Key: minio123"

# -------------------------------------------

## Execute the start-minio.sh script
./start-minio.sh

# -------------------------------------------

## Connect to the bucket with curl
vi manual-connect.sh

#!/bin/bash

SERVICE="s3"
REGION="us-east-1"
SECRET_KEY="minio123"
ACCESS_KEY="minio"
ENDPOINT="127.0.0.1:9000" ## This is the crucial line

# Generate the date and string in the required format
LONG_DATE=$(date -u +"%Y%m%dT%H%M%SZ")
SHORT_DATE=$(date -u +"%Y%m%d")
REQUEST_TYPE="aws4_request"

# Generate the signature
SIGNING_KEY=$(printf "${SHORT_DATE}${REGION}${SERVICE}${REQUEST_TYPE}" | openssl dgst -sha256 -hmac "AWS4${SECRET_KEY}" -hex | sed 's/^.* //')
SIGNATURE=$(printf "${LONG_DATE}${SHORT_DATE}${REGION}${SERVICE}${REQUEST_TYPE}" | openssl dgst -sha256 -hmac "${SIGNING_KEY}" -hex | sed 's/^.* //')

# Execute the curl command
curl -vX GET "${ENDPOINT}/" \
     -H "Host: ${ENDPOINT}" \
     -H "x-amz-date: ${LONG_DATE}" \
     -H "Authorization: AWS4-HMAC-SHA256 Credential=${ACCESS_KEY}/${SHORT_DATE}/${REGION}/${SERVICE}/${REQUEST_TYPE}, SignedHeaders=host;x-amz-date, Signature=${SIGNATURE}"

# -------------------------------------------

## Execute the manual-connect.sh Script without http:// as part of the Endpoint
./manual-connection.sh

# We will get the following output, which states that the accessKey and Secret are invalid and the calculated signature doesn't match. Thats ok for this testcase and proves that we successfully connected to our minio-Container.

./manual-connection.sh 
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:9000...
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:9000
> User-Agent: curl/7.81.0
> Accept: */*
> x-amz-date: 20230912T104933Z
> Authorization: AWS4-HMAC-SHA256 Credential=minio/20230912/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=52f8a43b7d2b9830fd46c01234dec0fc567734a4c6b5643f8f66dd34df2944cf
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< Accept-Ranges: bytes
< Content-Length: 362
< Content-Type: application/xml
< Server: MinIO
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Vary: Origin
< Vary: Accept-Encoding
< X-Amz-Id-2: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
< X-Amz-Request-Id: 1784211E23F7A35C
< X-Content-Type-Options: nosniff
< X-Xss-Protection: 1; mode=block
< Date: Tue, 12 Sep 2023 10:49:33 GMT
< 
<?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host 127.0.0.1 left intact
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource>/</Resource><RequestId>1784211E23F7A35C</RequestId><HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId></Error>

# -------------------------------------------

## Now change Line 7 of the manual-connection.sh script from "127.0.0.1:9000" to "http://127.0.0.1:9000"
vi manual-connection.sh
ENDPOINT="127.0.0.1:9000"
## to
ENDPOINT="http://127.0.0.1:9000"

## Execute it again
./manual-connection.sh 

Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:9000...
* Connected to 127.0.0.1 (127.0.0.1) port 9000 (#0)
> GET / HTTP/1.1
> Host: http://127.0.0.1:9000
> User-Agent: curl/7.81.0
> Accept: */*
> x-amz-date: 20230912T104641Z
> Authorization: AWS4-HMAC-SHA256 Credential=YOUR_MINIO_ACCESS_KEY/20230912/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=bab43a5001f56d64299275c53070edb9034037a634ebf167be82534585e599bd
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request: malformed Host header
< Content-Type: text/plain; charset=utf-8
< Connection: close
< 
* Closing connection 0
400 Bad Request: malformed Host header

## And here we get the '400 Bad Request' Error as response, which proves that minio does not work with http:// in the endpoint/host

I hope I was able to clearly demonstrate the use case. Have a nice Day

prakashsvmx commented 1 year ago

SDK expects useSSL flag to be true/false. on this basis, it would decide http vs https

var Minio = require('minio')

var minioClient = new Minio.Client({
  endPoint: 'play.min.io',
  port: 9000,
  useSSL: true, //  based on this protocol will be decided.
  accessKey: 'Q3AM3UQ867SPQQA43P2F',
  secretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
})
prakashsvmx commented 1 year ago

Could you check @danileau ?

danileau commented 1 year ago

@prakashsvmx Thank you for your idea - since i'm using https://github.com/formio/formio which uses https://github.com/minio/minio-js as a dependency, i don't see a possibility to set "useSSL: true," in this case.

As described in the initial issue Description, i suspect that the error lies in the internal/helper.js function "isValidDomain()" https://github.com/minio/minio-js/blob/master/src/internal/helper.ts#L128 which rejects domains with "http://" as part of the string. With a small change of this line, we could enable connections to non minio s3 stores, which need a http:// as part of the endpoint-string

@kannappanr fyi

prakashsvmx commented 1 year ago

So how is minio-js MinIO Client initialised ? Inside of the code may be in formio? @danileau 

var minioClient = new Minio.Client({
  endPoint: 'play.min.io',
  port: 9000,
  useSSL: true, // this needs to be set while initializing
  accessKey: 'Q3AM3UQ867SPQQA43P2F',
  secretKey: 'zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
})
danileau commented 1 year ago

Hey @prakashsvmx excuse me for my late response Besides the dependency definition of https://github.com/minio/minio-js I can't find any initialization of minio-clients in https://github.com/formio/formio. To work around this incompatibility, we have now built a simple proxy based on NodeJS, which receives the request for minio, rewrites it and forwards it to ECS S3. Currently we are testing this implementation

prakashsvmx commented 1 year ago

There is no reference to minio js npm package in https://github.com/formio/formio/blob/master/package.json so, not sure what exactly is being used or referred. We are closing this issue. Feel free to re-open with examples to help us investigate.