The IDL upload not work for alibaba cloud of mindsphere

[nrh387]

Describe the bug A clear and concise description of what the bug is. For alibaba cloud mindsphere, upload the file to datelake, the header need add the "Content-Type": "application/octet-stream", this is not need for eu1 and eu2 platform, but that's need for ali platform.

https://github.com/mindsphere/mindconnect-nodejs/blob/master/src/api/sdk/data-lake/data-lake.ts

Suggest add the logic check for alibaba signedURL as below:

public async PutFile(file: string | Buffer, signedUrl: string): Promise { const myBuffer = typeof file === "string" ? fs.readFileSync(file) : (file as Buffer);

    const proxy = process.env.http_proxy || process.env.HTTP_PROXY;
    const proxyHttpAgent: any = proxy ? new HttpsProxyAgent(proxy) : null;

    let headers;

    if (signedUrl.match(".*oss-.*.aliyuncs.com.*") != null ) {
        /**
         * In case of MindSphere in Alibaba cloud,
         * it needs to set content-type to be 'application/octet-stream' when to put file to data lake.
         */
        headers = { "Content-Type": "application/octet-stream" };
    } else {
        headers = { "x-ms-blob-type": "BlockBlob" }
    }

    // x-ms-blob is necessary on eu2 and is ignored on eu1
    const request: any = { method: "PUT", headers: headers, agent: proxyHttpAgent };
    request.body = myBuffer;
    const response = await fetch(signedUrl, request);
    return response.headers;
}

[nrh387]

@sn0wcat Could you please kindly help to check that? We are try to use the MindConnect Node-RED Agent v3.12.0 for IDL upload, and it blocked by this issue.

[sn0wcat]

@nrh387 I will add the support in the next version of the library. In the mean time as a workaround, you can use the node just to generate the signed URL (but don't do the upload) and do the actual file upload with standard node-red http nodes , while adding the necessary header. Sorry that i missed that, I don't have access to mindsphere on ALI...

// Uncomment the next code line if you just want to generate an upload url (in msg._signedUrl) 
// without actually uploading the file
msg._ignorePayload = true;

[nrh387]

Yes, i try to do that, but it looks it hard for me to pass the signed URL to standard node-red http node, the http node not accept the msg.payload from previous node.

[sn0wcat]

@nrh387 I released an alpha version of the libary. Can you manually install the alpha version of the library over the currently installed one with

npm install @mindconnect/mindconnect-nodejs@alpha

in the folder where your nodes are installed. (e.g. in docker it would be /data folder) . The correct folder has flows.json, flows_cred_json etc and check if it is working?

After the installation the package.json in the node_modules/@mindconnect/mindconnect-nodejs folder should have the version 3.19.0-1

[sn0wcat]

@nrh387 any luck?

[nrh387]

I have install the alpha version, but it looks still not work for me

[nrh387]

from code perspective, it looks ok to check the alibaba gateway address. Is there any debug information could get from node red?

[sn0wcat]

@nrh387 I've just gotten access to a tenant in CN, I have the same behavior, looking into it.

[sn0wcat]

@nrh387 I got it running ;) It was my fault , I told you to install the alpha in the wrong directory, the correct one for hacking the dependency is

It is working now, the example is here:

https://playground.mindconnect.rocks/#flow/5eda11af.df734

I will release a new version some time later today, which should work as expected ;)

[sn0wcat]

@nrh387 the release is out, uninstalling and reinstalling the node so that it picks up the 3.19.0 version of the library should fix the problem.

[nrh387]

Yes, i verified and it works. Thanks so much @sn0wcat

[nrh387]

@sn0wcat May i know if the latest docker image have been updated for this change or not? Right now, i fixed it with npm install manually. https://hub.docker.com/r/mindconnect/node-red-contrib-mindconnect

[sn0wcat]

Not yet, sorry I forgot to start the docker image build I will take care of it today ...

[nrh387]

@sn0wcat that's great, please let me know when the docker image have been pushed to docker hub. Thanks a lot