search

GoogleCloudPlatform / node-red-contrib-google-cloud

Node-RED nodes for Google Cloud Platform
Apache License 2.0
90 stars 57 forks source link

Improve reading speed of gcs-read #88

Closed lockercho closed 3 years ago

lockercho commented 3 years ago

I used gcs-read node to read a ~50MB file and it took several minutes to read the file stream. After carefully checking, I found the bottleneck was this line msg.payload = Buffer.concat([msg.payload, data]);. Every time this line is invoked, it creates a new buffer and does memory copy.

In this PR, with the help of pre-allocating buffer and only copying small data chunk every time, the final total reading time became < 10 seconds.

kolban-google commented 3 years ago

I love it!!! Thank you SO much for the contribution. I'm now going to re-package the npm release and push it out. Again THANK YOU!!!