Support asynchronous read-file function (aka read-from-stream)

[drzraf]

Reroll of #304 (see initial discussion there) based on v3 branch. See #295, #296 and #300

[bertrandg]

Hi, 👋

Thanks for your great work, I've just discovered this v3 branch/issue #304 and i'm following it with lot of interest (will need to do file E2EE).

What's the current state of this v3 branch, is it currently "unfinished/break" or "working/need more tests" ? What could I do to help you on it? Do you plan to publish an npm "v3.0.0-beta" version soon?

[drzraf]

It's definitely usable (and developing a E2EE project already relying on it, also not -production ready). As you know it's not only about async but about streams. What's already in, works, but other details must still be polished.

ES6 + class-split gave us the base. But subsequent code cleaning and improvements are needed:

• The (sensitive) part about concurrency could (and should) be made more readable by now.
• async initFileFn must be fixed (https://github.com/flowjs/flow.js/pull/296/)
• hooks should be discussed (https://github.com/flowjs/flow.js/issues/319)
• Handling stream needs more polishing regarding file.size, predictable chunks number and progressbar (https://github.com/flowjs/flow.js/issues/42#issuecomment-122209654)

[bertrandg]

Awesome, I will experiment with it and will tell you if I have some feedbacks 👍

[bertrandg]

Just to let you know, I've coded an e2ee POC with v3 and it works well 👍

Here is how i did with latest v3 commit:

function readFileChunk(file, startByte, endByte) {
    return new Promise(resolve => {
        const reader = new FileReader();
        reader.onload = () => {
            const bytes = new Uint8Array(reader.result);
            resolve(bytes);
        };
        const blob = file.slice(startByte, endByte);
        reader.readAsArrayBuffer(blob);
    });
}

async function encryptFileChunk(plaintextbytes, iv, key) {
    let cypherchunkbytes = await window.crypto.subtle.encrypt({name: 'AES-GCM', iv}, key, plaintextbytes);

    if(cypherchunkbytes) {
        cypherchunkbytes = new Uint8Array(cypherchunkbytes);
        return cypherchunkbytes;
    }
}

function init() {
    const flow = new Flow({
        testChunks: false,
        target: '/upload',
        chunkSize: window.chunkSize,
        allowDuplicateUploads: true,
        forceChunkSize: false,
        simultaneousUploads: 4,
        uploadMethod: 'POST',
        fileParameterName: 'file',
        asyncReadFileFn: async function(flowObj, startByte, endByte, fileType, chunk) {
            const plaintextbytes = await readFileChunk(flowObj.file, startByte, endByte);
            const cypherbytes = await encryptFileChunk(plaintextbytes, window.ivbytes, window.key);

            // Add 16 bytes from initialization vector
            chunk.chunkSize = chunk.chunkSize + 16; 

            const blob = new Blob([cypherbytes], {type: 'application/octet-stream'});
            return blob;
        }
    });

    flow.on('fileAdded', function(file) {
        // Add 16 bytes from each chunk initialization vector to total encrypted file size
        file.size += file.chunks.length * 16;
    });
}

[drzraf]

Very good. In my case I had the need for an async initFileFn because of openpgp encryption key initialization and full stream based encryption. For that purpose: https://github.com/flowjs/flow.js/pull/329 you may want to test it too in case you want to avoid keeping the whole regular file in memory ;)

[AidasK]

@bertrandg Can you add this to our samples folder? It would be great to have it for others to see. It's enough just to make one samples/encryption.md file with some explanations

[bertrandg]

@drzraf I don't think I got the whole regular file in memory there.. I read/load only file chunk (const blob = file.slice(startByte, endByte); reader.readAsArrayBuffer(blob);) in asyncReadFileFn on the fly called before each chunk POST request. Is the lib loading full file in memory?

@AidasK Yes sure, happy to help! :)