search

NOAA-EMC / NCEPLIBS-g2

Utilities for coding/decoding GRIB2 messages.
Other
6 stars 15 forks source link

NCEPLIBS-g2 consumes too much memory with large files #654

Closed edwardhartnett closed 5 months ago

edwardhartnett commented 6 months ago

It's not a leak, it's just that g2 works by copying the whole grib message into a memory buffer, before doing anything to it.

So this means a opening a multi-gig GRIB2 message starts by completely copying the whole message into memory. Well that's not helpful.

This happens even if we are only getting some metadata about the message.

The g2c library fixes this by only reading from the file what needs to be read.

edwardhartnett commented 5 months ago

We are going to resolve this by moving more of the functionality to g2c, which makes more efficient use of memory.

I will close this issue.