As the authentication in multi-cloud is changed to be compatible with Auth v2/v4, dashboard need to change accordingly.

[sfzeng]

In Daito release, the authentification algorithm used in multi-cloud is opensds specific, not be compatible with Auth v2/v4, now multi-cloud is change to be compatible with Auth v2/v4, so dashboard need to change accordingly.

[kumarashit]

@anvithks Please put the details, analysis and effort estimation here

[anvithks]

The changes implemented in the backend for this are as follows:

• Change in API path and port for S3 APIs
• Change in signature generation from Auth V2 to Auth V4.

Change in API path The path of the multi-cloud APIs earlier was /v1 and were all directed to port 8089. In the new implementation the API calls are split between port 8089 and 8090. All S3 related API calls are directed to port 8090 and path /s3. All other multi-cloud calls are directed to port 8089 and path /v1, as before.

Change in Signature Generation from Auth V2 to V4 In the earlier implementation the signature generation was calculated using custom code and the Auth V2 algorithms. The design can be found here

In the backend, the signature generation is now changed to Auth V4 and is using the open source implementation by Minio. As all S3 API requests to the backend must now be V4 compliant the code in the dashboard has to be modified or removed. The SODA Dashboard is an Angular / Typescript application and usually any library that is found in the npm package manager can be used inside. Pure Javascript implementations may or may not work out of the box. Sometimes the dependencies may not be available or the typings may not be available. After reading up on the requirements for the client side implementation I am considering the following:

1. Use the existing code and change only the signature generation. (Most probably this method will not work so did not spend any further time on this.)
2. Use the aws4 library located here (Need to check whether it works within Angular)
3. Use the minio-js library located here (Known issues working inside an Angular App. Need to check if it can be fixed or is already fixed)
4. Use the EvaporateJS library located here (Need to check whether it works within Angular)
5. Use the official aws-sdk for Javascript located here(Need to check how to use. There is a known issue with Angular CLI generated applications. Workaround is provided at the above link in the Angular section.)

Currently I am trying out each of the libraries above one by one and checking if they can be easily installed and used within our Angular Application. Once that is done the next step will be to use the API or methods provided by the library to generate the signature and validate the API calls.

[anvithks]

@sfzeng @kumarashit I am still looking into the different libraries and which one works as expected. I am trying to use the Minio library as it would be the same as the backend.

[anvithks]

@skdwriting @kumarashit @sfzeng @himanshuvar @nguptaopensds For the AWS V4 auth support from browser I have used the ngx-aws4 library. With this library I am able to successfully generate signatures for all the S3 compatible APIs as per our requirements. I could not use the MinioJS library as it was not developed for the browser. It is developed in Javascript for use as a NodeJS library in server mode. A PR #303 has been raised to fix this issue. But there are problems with some of the APIs which have a request payload. In the Object upload (PUT) call we have to generate the signature as for all APIs. As per the AWS documentation signature calculations vary depending on the method you choose to transfer the request payload. S3 supports the following options: (Excerpt from this documentation)

Transfer payload in a single chunk – In this case, you have the following signature calculation options:

• Signed payload option – You can optionally compute the entire payload checksum and include it in signature calculation. This provides added security but you need to read your payload twice or buffer it in memory.

  For example, in order to upload a file, you need to read the file first to compute a payload hash for signature calculation and again for transmission when you create the request. For smaller payloads, this approach might be preferable. However, for large files, reading the file twice can be inefficient, so you might want to upload data in chunks instead.

  We recommend you include payload checksum for added security.

• Unsigned payload option – Do not include payload checksum in signature calculation.

Transfer payload in multiple chunks (chunked upload) – In this case you transfer payload in chunks. You can transfer a payload in chunks regardless of the payload size.

You can break up your payload into chunks. These can be fixed or variable-size chunks. By uploading data in chunks, you avoid reading the entire payload to calculate the signature. Instead, for the first chunk, you calculate a seed signature that uses only the request headers. The second chunk contains the signature for the first chunk, and each subsequent chunk contains the signature for the chunk that precedes it. At the end of the upload, you send a final chunk with 0 bytes of data that contains the signature of the last chunk of the payload. For more information, see Signature Calculations for the Authorization Header: Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version 4).

When you send a request, you must tell Amazon S3 which of the preceding options you have chosen in your signature calculation, by adding the x-amz-content-sha256 header with one of the following values:

• If you choose chunked upload options, set the header value to STREAMING-AWS4-HMAC-SHA256-PAYLOAD.

• If you choose to upload payload in a single chunk, set the header value to the payload checksum (signed payload option), or set the value to the literal string UNSIGNED-PAYLOAD (unsigned payload option).

Upon receiving the request, Amazon S3 re-creates the string to sign using information in the Authorization header and the date header. It then verifies with authentication service the signatures match. The request date can be specified by using either the HTTP Date or the x-amz-date header. If both headers are present, x-amz-date takes precedence. If the signatures match, Amazon S3 processes your request; otherwise, your request will fail.

Our backend currently supports transfer in single chunk and mulitple chunks. It does not support UNSIGNED-PAYLOAD. After understanding the process for signature generation and referring to the backend code I realised that the signature generation required the file to be read in memory as a binary string, used the SHA256 hashing and use this hash as explained above. This process of reading a file in memory and generating the signature is quite heavy, especially to do in the browser. JavaScript is not a low-level language and is primarily a client side scripting language. As such file handling and file manipulation support is just basic and not as comprehensive as some low level languages like Go. For file manipulation on the browser I used the FileReader API, in particular the readAsArrayBuffer() method. This reads the file in as an ArrayBuffer object.

The ArrayBuffer object is used to represent a generic, fixed-length raw binary data buffer. It is an array of bytes, often referred to in other languages as a "byte array".You cannot directly manipulate the contents of an ArrayBuffer; instead, you create one of the typed array objects or a DataView object which represents the buffer in a specific format, and use that to read and write the contents of the buffer.

With this I was able to successfully upload files with text content (tested with these formats: .txt, .yml ) but failed for other file types (Tested with these formats: .gif, .png, .jpg, .docx, .xslx, .pdf, .deb, mp4, .mp3)

This is where the major issue is with reading files in memory. To put together the binary string I have to use one of the TypedArrays or DataView which would involve putting together the binary string at byte level.

For large file uploads we generally use Blob or multipart FormData from the browser. Trying to upload using both these methods results in a signature mismatch on the multi-cloud backend (since backend is not expecting a Blob or FormData object).

Possible solutions:

1. We have to enable UNSIGNED-PAYLOAD support in the multi-cloud backend. The browser will upload the file to the backend with the signature generated without payload. The backend can then verify the file and upload it to appropriate cloud backend.
2. We create a signature generation API on the backend that any client can use to generate the signature and use the same for uploading the object.
3. We use pre-signed URLs that our backend will have already authorised with the cloud backend.
4. We enable support for FormData object or Blob to be accepted while uploading an object alongwith the signature generation enabled with Blob or FormData (Needs to be tested.)

I am still working on finding a viable solution that can be implemented. If anyone is interested to contribute and has any questions please tag me in the comments below.