Errors: "ERR_FS_FILE_TOO_LARGE" and "Cannot create a string longer than 0x3fffffe7 characters"

[drmrbrewer]

So far this excellent library has been great (thanks!). But I've recently been trying to convert grib files from this page to json, within a node.js environment. It seems to be choking on the size of the grib file (or rather the resulting json)... I guess that they are big (e.g. 88 MB) but not completely extreme in the scheme of things (they are supplied by MeteoFrance in that format/size so they must think this is generally usable).

Specifically, I am getting the following error on one of the Europe/SP2 grib files (downloadable using the drop-down selections at the bottom of the above page):

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2259325381) is greater than possible Buffer: 2147483647 bytes
    at FSReqCallback.readFileAfterStat [as oncomplete] (fs.js:282:11)
    at FSReqCallback.callbackTrampoline (internal/async_hooks.js:126:14) {
  code: 'ERR_FS_FILE_TOO_LARGE'
}

And on a Europe/SP1 grib file I seem to get a slightly different error:

Error: Cannot create a string longer than 0x3fffffe7 characters
    at Object.slice (buffer.js:604:37)
    at Buffer.toString (buffer.js:801:14)
    at stripBom (/root/workspace/meteofrancegetter/node_modules/jsonfile/index.js:122:51)
    at /root/workspace/meteofrancegetter/node_modules/jsonfile/index.js:29:12
    at /root/workspace/meteofrancegetter/node_modules/graceful-fs/graceful-fs.js:123:16
    at FSReqCallback.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:63:3)
    at FSReqCallback.callbackTrampoline (internal/async_hooks.js:126:14) {
  code: 'ERR_STRING_TOO_LONG'
}

These seem to be different to the error I reported here... different error and the solution there (increasing bufferSize) doesn't help here.

Maybe there is a simple solution to overcoming the limits I'm hitting? Fingers crossed...

[claustres]

What is your command line exactly ? Are you using a filter on a specific parameter (you should for large files) ? You can also try to limit output precision to 2 digits with -p 2. I will try to run it on my side as well.

[drmrbrewer]

I'm not applying any parameter filter because I need to extract all parameters. But maybe it would still be better (or at least it may avoid the problem) to apply a different filter in turn, on the same file, rather than trying to process all parameters in one go? I'll try with -p 2.

EDIT I get the exact same errors even with precision and fc and fp options, as per the following:

{
    data: true,
    names: true,
    precision: 2,
    fc: 0,
    fp: 0,
    output: 'output.json',
    bufferSize: 32 * 1024 * 1024
}

[claustres]

Yes you need to process each parameter separately, probably using fs, fp and fv. This will also allow for paralell processing.

[drmrbrewer]

I'll try filtering on fs and fv too, but surely filtering on fc and fp (which I already tried) should have made enough of a difference to prevent the error? EDIT OK I think that my choice of fc: 0 and fp: 0 was maybe still letting too much through... when I try fc: 3 and fp: 0 it works without error. So it looks like I just need to get my filters sorted... thanks for the tip!

[drmrbrewer]

I think I discovered a slight quirk in the way that the filter values are interpreted. In my post a couple of entries up, I use numerical values of 0 for the filter items fc and fp, and was surprised that the errors were still caused... I thought that this should limit to only the temperature parameter.

But it seems that, to be properly interpreted, we need to pass these as string values '0' instead:

{
    data: true,
    names: true,
    precision: 2,
    fc: '0',
    fp: '0',
    output: 'output.json',
    bufferSize: 32 * 1024 * 1024
}

With string values for these filters, it works fine. If you agree, perhaps it would be an idea for the library to use .toString() internally to avoid this gotcha?

[claustres]

Good catch, as we are using options as args of execFile it should be strings. I will fix that with a conversion to string. Thanks for feedback.

[drmrbrewer]

when will the above fix (conversion to string) be available as an actual release?

[claustres]

Done https://github.com/weacast/weacast-grib2json/releases/tag/v1.0.3