jsreport read data from stream in order to generate report

[tahirakhter]

we are using axios to read large data from an API server as JSON stream as below:

axios({
  url: params.url,
  method: 'GET',
  responseType: 'stream' // important
}).then(function (response) {
//?? what should we do here to stream response.data as input for jsreport. something like response.data.pipe(??)
})

below is our jsreport.render function

jsreport.render({
                            template: {
                                content: body,
                                engine: type === 'xlsx' ? 'handlebars' : 'jsrender',
                                recipe: recipe,
                                phantom: {
                                    format: 'A3',
                                    margin: {'top': '1cm', 'left': '0cm', 'right': '0cm', 'bottom': '1cm'},
                                    header: header,
                                    headerHeight: '3cm',
                                    orientation: resData.content.orientation,
                                    footer: footer
                                }
                            },
                            data: //?? what should we use here. how should we add stream pipeline
                        })

[bjrmatos]

what should we use here. how should we add stream pipeline

jsreport does not support receiving data as stream input, and it does not support it not because we don't want to.. it is more because the template evaluation limitations (limitation of template engines like handlebars, etc), it needs to have the whole data in object in order to make the data replacement in the template, there is no data stream flow support on those template engines so even if we accept that you pass an stream we'll still need to read and concat the data of the stream and put it in an object before the template evaluation starts.

you can use something like concat-stream for collecting the data in the stream before calling render.

i will let the issue open in case @pofider wants to add some comments about accepting data as stream (but as i said we will still need to read the whole stream before starting to render, adding support for the stream will be just for convenience)

[pofider]

Yes, it is the limitation of the templating engines we use. We had already a discussion about this before releasing v2 and it ended up that we can't support streams so far.

Maybe you just need to do a better data projection to limit data size.

[pofider]

Closing. Evaluating streams isn't technically possible. Even if jsreport uses just streams in its internal, in the end, there are other dependencies that need to load everything to the memory.