microsoft / CDM

The Common Data Model (CDM) is a standard and extensible collection of schemas (entities, attributes, relationships) that represents business concepts and activities with well-defined semantics, to facilitate data interoperability. Examples of entities include: Account, Contact, Lead, Opportunity, Product, etc.
https://powerplatform.microsoft.com/en-us/common-data-model/
Creative Commons Attribution 4.0 International
1.68k stars 531 forks source link

Typescript SDK using ADLAdapter SharedKey throws "Invalid character in header content ["Authorization"]" #159

Closed Akkilz closed 4 years ago

Akkilz commented 4 years ago

tried using ADLSAdapter typescript and its failing in Authorization while reading the manifest.cdm.json file.

Err Info: code:"ERR_INVALID_CHAR" message:"Invalid character in header content ["Authorization"]" stack:"TypeError [ERR_INVALID_CHAR]: Invalid character in header content ["Authorization"] at ClientRequest.setHeader (_http_outgoing.js:529:3) at new ClientRequest (_http_client.js:241:14) at Object.request (https.js:314:10) at C:\Users\Arun\source\repos\BMCL_Adf\CDM-Typescript\bmcl_cdm\libs\CDM-master\objectModel\TypeScript\lib\Storage\request.js:54:29 at new Promise () at C:\Users\Arun\source\repos\BMCL_Adf\CDM-Typescript\bmcl_cdm\libs\CDM-master\objectModel\TypeScript\lib\Storage\request.js:35:16 at Generator.next () at C:\Users\Arun\source\repos\BMCL_Adf\CDM-Typescript\bmcl_cdm\libs\CDM-master\objectModel\TypeScript\lib\Storage\request.js:10:71 at new Promise () at __awaiter (C:\Users\Arun\source\repos\BMCL_Adf\CDM-Typescript\bmcl_cdm\libs\CDM-master\objectModel\TypeScript\lib\Storage\request.js:6:12)"

ADLSAdapter.ts -> private applySharedKey(sharedKey: string, url: string, method: string, content?: string, contentType?: string): Map<string, string>

3rd header item of Authorization has malformed or someting...

need some info to proceed further and it would be really helpful.

Akkilz commented 4 years ago

After comparing to c# code have corrected typescript code in ADLSAdapter.ts below function Line no starting 324

    private applySharedKey(sharedKey: string, url: string, method: string, content?: string, contentType?: string): Map<string, string> {
        const headers: Map<string, string> = new Map<string, string>();

        // Add UTC now time and new version.
        headers.set(this.httpXmsDate, new Date().toISOString());
        headers.set(this.httpXmsVersion, '2018-06-17');

        let contentLength: number = 0;

        const uri: URL = new URL(url);

        if (content) {
            contentLength = Buffer.from(content).length;
        }

        let builder: string = '';
        builder += `${method}\n`; // verb;
        builder += '\n'; // Content-Encoding
        builder += ('\n'); // Content-Language.
        builder += (contentLength !== 0) ? `${contentLength}\n` : '\n'; // Content length.
        builder += '\n'; // Content-md5.
        builder += contentType ? `${contentType}; charset=utf-8\n` : '\n'; // Content-type.
        builder += '\n'; // Date.
        builder += '\n'; // If-modified-since.
        builder += '\n'; // If-match.
        builder += '\n'; // If-none-match.
        builder += '\n'; // If-unmodified-since.
        builder += '\n'; // Range.

        for (const header of headers) {
            builder += `${header[0]}:${header[1]}\n`;
        }

        // Append canonicalized resource.
        const accountName: string = uri.host.split('.')[0];
        builder += '/';
        builder += accountName;
        builder += uri.pathname;

        // string accountName = uri.Host.Split('.')[0];
        // builder.Append("/");
        // builder.Append(accountName);
        // builder.Append(uri.AbsolutePath);

        // Append canonicalized queries.
        if (uri.search) {
            const queryParameters: string[] = uri.search.split('&');

            for (const parameter of queryParameters) {
                const keyValuePair: string[] = parameter.split('=');
                builder += `\n${keyValuePair[0]}:${keyValuePair[1]}`;
            }
        }

        // hash the payload
        const dataToHash: Buffer = Buffer.from(builder.trimRight());
        const bytes: Buffer = new Buffer(sharedKey);

        const hmac: crypto.Hmac = crypto.createHmac('sha256', bytes);
        const signedString: string = `SharedKey ${accountName}:${hmac.digest('base64')}`;
        headers.set(this.httpAuthorization, signedString);

        // hmac.on('readable', () => {
        //     const data: string | Buffer = hmac.read();
        //     if (data) {
        //         const signedString: string = `SharedKey ${accountName}:${data.toString('base64')}`;
        //         headers.set(this.httpAuthorization, signedString);
        //     }
        // });

        // hmac.write(dataToHash);
        // hmac.end();

        return headers;
    }

Getting 'timeout' exception in below location

protected async executeRequest(httpRequest: CdmHttpRequest): Promise in NetworkAdapter.ts line no 87 its throwing TimeOut exception

Akkilz commented 4 years ago

ADLSAdapter.txt

Fixed the above issue, please find the code attached in the .txt file..

Now I am facing other issue, PersistenceLayer | Failed to write to the file 'digitalTwin.manifest.cdm.json' for reason Error: Could not create folder for document /manifest/digitalTwin/digitalTwin.manifest.cdm.json | saveDocumentAsAsync

sukanyamsft commented 4 years ago

hi, does /manifest/digitalTwin/ folder exist?

Akkilz commented 4 years ago

This issue i had fixed in the typescript code,

  1. the date for the date header was in ISO instead of UTC (line 328)
  2. in line 342 where, the string build with some extra ()
  3. missing the account name in the string (line 358-361)
  4. the sharedKey was not converted from base64 before using in hmac
  5. the whole lines of hash creation was messed (382-385)

ADLSAdapter.txt

Plz find the code changes from attached file.

After fixing this I am getting other issues. image

Not able to save the manifest.cdm.json file

Akkilz commented 4 years ago

TypeScript SDK: Fixes sharedKey signature creation for ADLS access #160

This issue fixed by above changes.

Akkilz commented 4 years ago

hi, does /manifest/digitalTwin/ folder exist?

Yes, it exist.

sukanyamsft commented 4 years ago

Ok. Thank you for your note. We will look into and get back to you.

dingbx commented 4 years ago

Hi Akkilz,

Sorry for the late response. And sorry for not saying in advance but we currently do not support accepting PRs from GitHub. The fix for Typescript sharedkey header exception is now published to this repo. Most of your changes are included except the line builder += ('\n').

Are you still getting the error when saving the manifest file now? Can you please open a new issue to differentiate with this Typescript auth issue if it is still happening so that we can better track it?

Thanks, Yijing