Using streamSaver with excelJS

[Congyi-Shi]

Hello, The problem: In my project, I have a back-end (server) which is a rest webservice that sends json files in streaming, and on the other side a front-end (client) in Angular 8 which consumes the data json and the transforms it into an xlsx file using the exceljs library and uses the File-saver library to save it to disk. But if there is large of data, we have the OutOfMemery error in the browser. The question: By reading the documentation of your library (StreamSaver), I understood that it makes it possible to record the large files in disk with streaming, do you think that it is possible to record a workbook which is created by exceljs in streaming to use StreamSaver ? Thank you¬

[jimmywarting]

I don't know, but it's certainly possible if you are able to transform your json data to an excel and get a stream out of it. I don't know if that is possible but i have a few tricks in my sleeve if you want to save multiple files into a zip.

[kmessai]

Hello I am the colleague of @Congyi-Shi, Thank you @jimmywarting for your answer but we want more help please. In order to save a Workbook of Exceljs I tried several ways of StreamSaver but nothing worked (It's not StreamSaver's fault, but surely I did not find the right way :( ), my goal was to use Exceljs and StreamSaver together. Exceljs offers streaming saving but just for the Nodejs here is the URL of the documentation.

here is my code, do you think that there is a way to do what I want or rather I have taken a wrong direction ?

I remind you that our front application is in Angular 8

    let delta;
    let start = Date.now();
    let header = VocExportHeaders;
    let filetitle = 'text_exportINdiv';   
    let workbook = new Excel.Workbook();
    let indivWorksheet = workbook.addWorksheet('INDIV');

    indivWorksheet.columns = header;
    // Stremaing JSON from Back-End
    const oboe = require("oboe");
    const exportVOCFlux = async () => {
      try {
          const post$ = new Observable(client => {
            oboe({
              url:url,
              method:'POST',
              body:params,
              headers: {
                'Content-Type': 'application/stream+json',
                'Content-Length': params.length
              }
            })
            .done(indiv => {
              // The indiv is the Base JSON
              indivWorksheet.addRow(indiv['values']).commit(); // The OutMemoryError In the Browser when adding many rows
              // The if that verify the end of the streaming              
              if (indiv['count'] === indiv['total']) {
                indivWorksheet.commit();
                // The commit the Workbook object
                workbook.commit().then(()=> {
                  workbook.xlsx.writeBuffer().then((data) => {
                    let blob = new Blob([data], { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8' });                   
                    FileSaver.saveAs(blob, filetitle + '.xlsx');
                  });
                })             
              }
            })
            .fail(e => client.error(e))
          });
          await post$.pipe(
            timestamp(),
            map(({timestamp, value}) => {
              delta = timestamp - start
              return value
            }),
            toArray()
          ).toPromise();
      } catch (error) {
        console.error(error);
      }
    };

    exportVOCFlux().then(all => {console.log('all converd')}).catch(() => console.error('Failed!'));

[jimmywarting]

well, this part won't work

workbook.commit().then(()=> {
  workbook.xlsx.writeBuffer().then((data) => {
  let blob = new Blob([data], { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8' });                 
  FileSaver.saveAs(blob, filetitle + '.xlsx');
  });
})

shows that you build up the hole excel in memory. you need to get a whatwg-stream, node-stream, an iterator somehow from excelJS

it seems like they removed that functionality from the browser...? I think they follow some unconventional design in there api.

if they want to build a more cross platform application that can work mostly everywhere then they should follow cross-js style guide https://github.com/cross-js/cross-js#dont-create-node-or-web-readable-stream-yourself

[Congyi-Shi]

well, this part won't work

workbook.commit().then(()=> {
  workbook.xlsx.writeBuffer().then((data) => {
  let blob = new Blob([data], { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;charset=UTF-8' });                   
  FileSaver.saveAs(blob, filetitle + '.xlsx');
  });
})

shows that you build up the hole excel in memory. you need to get a whatwg-stream, node-stream, an iterator somehow from excelJS

it seems like they removed that functionality from the browser...? I think they follow some unconventional design in there api.

if they want to build a more cross platform application that can work mostly everywhere then they should follow cross-js style guide https://github.com/cross-js/cross-js#dont-create-node-or-web-readable-stream-yourself

Thank you for the suggestions and answers to out project. We will try others methodes and contact with group ExcelJS. Anyway, thank you a lot.