Using grib2json without any output file or even input file

[drmrbrewer]

First, thanks for this excellent library!

I am using it as a node module, along the following lines:

const jsondata = await grib2json('gfs.grib', {
  data: true,
  output: 'output.json'
})

At the end of this, I have the data I need in the jsondata object and I do not actually need the json written to any file. Is it possible to use grib2json without any output, to save on unnecessary disk operations?

Similarly, my gfs.grib file is being fetched from a remote url and ideally I would be piping the read stream directly into grib2json rather than having to write a grib file first and then run grib2json on that... i.e. NOT the following:

const response = await request(requestOptions);
const readStream = response.data;
const writeStream = fs.createWriteStream('gfs.grib');
readStream.pipe(bz2()).pipe(writeStream);

[claustres]

Thanks for your feedback.

If you use it as a node module you can already get the data in memory without writing a file, for this simply remove the output option in your call as in our tests.

Piping would be great but I am afraid it would be very complicated to implement because under the hood the grib file is actually read by a Java process not a native node process. Unfortunately, I did not find any great grib reader in Node :-(

[drmrbrewer]

simply remove the output option in your call

I tried that, but then it seems that the output just goes to stdout... and since the output is rather large (to say the least), I then get RangeError [ERR_CHILD_PROCESS_STDIO_MAXBUFFER]: stdout maxBuffer length exceeded.

I have also tried:

output: '/dev/null'

but then I just get SyntaxError: /dev/null: Unexpected end of JSON input

Is there any way to completely suppress the output rather than sending it to anywhere?

[claustres]

About stdout, that's actually how it works internally. As processing is performed by an external Java command, the most simple way to get data back into memory without requiring to write a temporary file is to read it from stdout. You can play with the bufferSize option to increase the max buffer size (defaults to 8 MB) and avoid the range error.

[drmrbrewer]

aaah OK I gotcha. Thanks for the explanation, which makes sense. And indeed, increasing bufferSize to 16 MB avoids the range error, and it does indeed work fine without ever having to write the json to a file... great!

Again, thanks so much for this library. I was surprised that there aren't a whole bunch of different libraries available for processing and parsing GRIB files within a node environment... of those that are available, this was the only one that actually seems to work (for me)! Just converting the grib to json is enough for my purposes... I don't need any fancy grib-related functions, just plain old json is fine. (At least, until it comes to working with grib files which are encoded onto a grid based on something other than plain "lat/long" coordinates and which require some complicated mapping to get to lat/long... but I will cross that bridge when I come to it!)

[claustres]

Improved the doc for this https://github.com/weacast/weacast-grib2json#as-module.