Trying to generate a Document with 100 High Resolution images (react-native), App close itself after a while

[Kohys58]

Hello,

I've been working on this for days now and I can't figure out a way of achieving my goal here.

react-native : 0.72.6 docx : 8.5.0 react-native-blob-util : 0.19.10

Packer.toBase64String(docRapport).then(async (b64stringRapport) => {
                const rapporPath = FolderAGRapports + nomRapport + " - " + typeRapport + ".docx";

                await ReactNativeBlobUtil.fs.writeFile(rapporPath, b64stringRapport, "base64");

                const filteredNotesParcoursRapport = notesParcoursRapport.filter(note => note.chemin_fichier_note_parcours ?? false);

                const BATCH_SIZE = 30;

                const chunked_arr_notes = chunks(filteredNotesParcoursRapport, BATCH_SIZE);

                for (const chunked_arr_note of chunked_arr_notes) {
                    let patches: any = {};

                    for (const chunked_note of chunked_arr_note) {
                        switch (chunked_note.type_note_parcours) {
                            case "image":
                                try {
                                    const imagePath = Platform.OS === "android" ? "file://" + chunked_note.chemin_fichier_note_parcours : chunked_note.chemin_fichier_note_parcours;
                                    const imageDimensions = await getImageDimensions(imagePath);
                                    const imageData = await ReactNativeBlobUtil.fs.readFile(chunked_note.chemin_fichier_note_parcours, "base64");

                                    patches[chunked_note.chemin_fichier_note_parcours?.split("/").pop()] = {
                                        type: PatchType.DOCUMENT,
                                        children: [
                                            new Paragraph({
                                                alignment: AlignmentType.CENTER,
                                                children: [
                                                    new ImageRun({
                                                        data: base64ToArrayBuffer(imageData),
                                                        transformation: {
                                                            width: imageDimensions.width / 3,
                                                            height: imageDimensions.height / 3,
                                                        },
                                                        altText: {
                                                            title: "Parcours " + chunked_note.parcours.num_parcours + " - Note " + chunked_note.position_note_parcours,
                                                            name: "Parcours " + chunked_note.parcours.num_parcours + " - Note " + chunked_note.position_note_parcours,
                                                            description: "Parcours " + chunked_note.parcours.num_parcours + " - Note " + chunked_note.position_note_parcours,
                                                        }
                                                    })
                                                ]
                                            })
                                        ],
                                    };
                                } catch (error) {
                                    console.error(`Erreur lors de la récupération des dimensions de l'image du parcours : ${error}`);
                                }
                                break;
                            case "relevé":
                                const contenu_releve_csv = await ReactNativeBlobUtil.fs.readFile(chunked_note.chemin_fichier_note_parcours, "utf8");
                                const json_brut_releve = csvJSON(contenu_releve_csv);
                                const json_headers = JSON.parse(json_brut_releve.headers);
                                const json_releve = JSON.parse(json_brut_releve.json_from_csv);
                                let tmp_arr_releve_data = [];
                                for (let i = 0; i < json_releve.length; i++) {
                                    Object.keys(json_releve[i]).forEach((k) => { json_releve[i][k] = json_releve[i][k].trim() });
                                    tmp_arr_releve_data.push(Object.values(json_releve[i]));
                                }
                                // On replace la lettre en début de l'array
                                for (let i = 0; i < tmp_arr_releve_data.length; i++) {
                                    tmp_arr_releve_data[i].unshift(tmp_arr_releve_data[i].pop());
                                }

                                const chunkSize = 20;
                                // Si l'array contient plus de chunkSize éléments alors on split en chunkSize Cells à chaque fois
                                if (json_headers.length > chunkSize) {
                                    json_headers.shift();
                                    const chunked_arr_headers = chunks(json_headers, chunkSize);
                                    let chunked_arr_releve_data = [];

                                    let letters_array = [];

                                    for (let i = 0; i < tmp_arr_releve_data.length; i++) {
                                        let shifted_letter = tmp_arr_releve_data[i].shift();
                                        letters_array.push(shifted_letter as string);
                                        chunked_arr_releve_data.push(chunks(tmp_arr_releve_data[i], chunkSize));
                                    }

                                    chunked_arr_headers.forEach(headers => headers.unshift(""));
                                    chunked_arr_releve_data.forEach((dataChunks, index) => dataChunks.forEach(chunk => chunk.unshift(letters_array[index])));

                                    const arr_global_data = chunked_arr_headers.flatMap((headers, i) => [headers, ...chunked_arr_releve_data.map(dataChunks => dataChunks[i])]);

                                    patches[chunked_note.chemin_fichier_note_parcours?.split("/").pop()] = {
                                        type: PatchType.DOCUMENT,
                                        children: [
                                            new DTable({
                                                width: {
                                                    size: `${100}%`,
                                                    type: "pct",
                                                },
                                                alignment: AlignmentType.CENTER,
                                                rows: arr_global_data.map((row_releve, index_row_releve) =>
                                                    new TableRow({
                                                        height: {
                                                            value: `${0.6}cm`,
                                                            rule: "atLeast",
                                                        },
                                                        children: row_releve.map((cell_row_releve: string, index_cell_row_releve: number) =>
                                                            new TableCell({
                                                                width: {
                                                                    size: `${2}%`,
                                                                    type: WidthType.PERCENTAGE
                                                                },
                                                                verticalAlign: VerticalAlign.CENTER,
                                                                shading: {
                                                                    fill: index_row_releve === 0 ? "F2F2F2" : (index_row_releve % (chunked_arr_releve_data.length + 1) === 0 ? "F2F2F2" : (index_cell_row_releve === 0 ? "F2F2F2" : undefined)),
                                                                },
                                                                children: [
                                                                    new Paragraph({
                                                                        alignment: AlignmentType.CENTER,
                                                                        children: [
                                                                            new TextRun({
                                                                                bold: index_row_releve === 0 ? true : (index_row_releve % (chunked_arr_releve_data.length + 1) === 0 ? true : (index_cell_row_releve === 0 ? true : false)),
                                                                                text: cell_row_releve,
                                                                                font: "Arial",
                                                                                size: `${10}pt`,
                                                                            })
                                                                        ]
                                                                    }),
                                                                ],
                                                            }),
                                                        ),
                                                    })
                                                ),
                                            })
                                        ],
                                    };
                                } else {
                                    tmp_arr_releve_data.unshift(json_headers);
                                    const arr_releve_data = tmp_arr_releve_data;

                                    patches[chunked_note.chemin_fichier_note_parcours?.split("/").pop()] = {
                                        type: PatchType.DOCUMENT,
                                        children: [
                                            new DTable({
                                                width: {
                                                    size: `${100}%`,
                                                    type: "pct",
                                                },
                                                alignment: AlignmentType.CENTER,
                                                rows: arr_releve_data.map((row_releve, index_row_releve) =>
                                                    new TableRow({
                                                        height: {
                                                            value: `${0.6}cm`,
                                                            rule: "atLeast",
                                                        },
                                                        children: row_releve.map((cell_row_releve: string, index_cell_row_releve: number) =>
                                                            new TableCell({
                                                                width: {
                                                                    size: `${100 / json_headers.length}%`,
                                                                    type: WidthType.PERCENTAGE
                                                                },
                                                                verticalAlign: VerticalAlign.CENTER,
                                                                shading: {
                                                                    fill: index_row_releve === 0 ? "F2F2F2" : (index_cell_row_releve === 0 ? "F2F2F2" : undefined),
                                                                },
                                                                children: [
                                                                    new Paragraph({
                                                                        alignment: AlignmentType.CENTER,
                                                                        children: [
                                                                            new TextRun({
                                                                                bold: index_row_releve === 0 ? true : (index_cell_row_releve === 0 ? true : false),
                                                                                text: cell_row_releve,
                                                                                font: "Arial",
                                                                                size: `${10}pt`,
                                                                            })
                                                                        ]
                                                                    }),
                                                                ],
                                                            }),
                                                        ),
                                                    })
                                                ),
                                            })
                                        ],
                                    };
                                }
                                break;
                        }
                    }
                    console.log("RAPPORT LOADING...");

                    const rapportContent = await ReactNativeBlobUtil.fs.readFile(rapporPath, "base64");

                    console.log("RAPPORT LOADED");

                    console.log("RAPPORT PATCHING...");

                    const patchedDoc = await patchDocument(
                        base64ToArrayBuffer(rapportContent),
                        {
                            patches: patches,
                        }
                    );

                    console.log("RAPPORT PATCHED");

                    patches = {};

                    console.log("RAPPORT WRITTING...");

                    await ReactNativeBlobUtil.fs.writeFile(rapporPath, uint8ArrayToBase64(patchedDoc), "base64");

                    console.log("RAPPORT WRITTEN");
                }
            }).catch(err => {
                console.error(err);
            });

Initially I was generating the whole document (with the 100 images) directly in the new Document() constructor, but I had this issue where the app would close itself after a while, usually up to an hour.

So I tried doing it with the patchDocument method (with the 100 images at once), still had the same issue.

Then I tried to process it with "batches", so that It would free some memory after each operation, but even if I have let's say a batch size of 1 and that I write the document each time a new image is proccessed, it takes even more time (few hours), and the app still close itself at some point.

At each iteration the console.log() takes more time to print, which I believe is logical since the document is getting bigger each time.

The App close "bug" usually happens (no matter the batch size), after around 30 images have been processed, it prints my "RAPPORT LOADING..." log, then App close itself (no error message).

I thought of using some multi threading library to deal with it, but unfortunately none of them has been updated with the react native version that i'm using (0.72.6), plus, i'm not sure if this type of computation can be done in a parallel thread.

This is a soon to be production App and I have no idea on how to handle this issue. Help needed.

Here are my utilities functions :

const chunks = (a: any[], size: number) =>
    Array.from(
        new Array(Math.ceil(a.length / size)),
        (_, i) => a.slice(i * size, i * size + size)
    );

function uint8ArrayToBase64(uint8Array: Uint8Array): string {
    const binaryString = uint8Array.reduce((acc, byte) => acc + String.fromCharCode(byte), '');
    return btoa(binaryString);
};

function base64ToArrayBuffer(base64string: string) {
    var binaryString = atob(base64string);
    var bytes = new Uint8Array(binaryString.length);
    for (var i = 0; i < binaryString.length; i++) {
        bytes[i] = binaryString.charCodeAt(i);
    }
    return bytes.buffer;
};

[Kohys58]

If anyone sees this, it was not caused by something related to the docx lib in itself, I just changed my way of managing the readFile and writeFile from the react-native-blob-util library.

Instead i'm using readStream and writeStream, which fixes the issue.