'message.latlons()' raises MemoryError on rotated lat-lon grids

[weatherfrog]

latlons() raises a MemoryError on grib files/messages that contain a rotated lat-lon grid, which is supported according to the docstring.

Code that reproduces the problem:

msg = pygrib.open("example.grb2")[1]
msg.latlons()

example.grb2 with rotated lat-lon grid: example.zip

Output:

  File "test.py", line 10, in main
    msg.latlons()
  File "pygrib.pyx", line 1808, in pygrib.gribmessage.latlons
  File "[...]/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4698, in meshgrid
    output = [x.copy() for x in output]
  File "[...]/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4698, in <listcomp>
    output = [x.copy() for x in output]
MemoryError

Tested with:

• Pygrib 2.0.3
• Python 3.6
• Cython 0.29.1
• Ubuntu 18.04

The error does not occur with files (from same origin) that contain a regular lat-lon grid.

[jswhit]

The code assumes that distinctLatitudes and distinctLongitudes are 1D arrays, and then uses numpy.meshgrid to create 2D arrays. This is apparently a bug - those arrays are flattened 2d arrays. numpy.meshgrid is trying to create a huge 2d array and runs out of memory.

From the file I see:

Ni = 651 Nj = 716

However

distinctLatitudes.shape = (260003,) distinctLongitudes.shape = (460257,)

Ni*Nj = 466116

so I really don't know how to use that information. Can you ask the data provider?

[jswhit]

OK, here's my hypothesis. The pygrib code assumes that distinctLatitudes and distinctLongitudes are the 1D arrays of lats and lons in the rotated coordinate system. ecCodes (formerly grib_api) used to provide this info, but it no longer does. I don't know what those arrays are populated with any more. We do have a test_rotated_ll.py test script - it used to work but now produces the same MemoryError.

In summary, somewhere along the line the behavior of the C lib changed for the rotated lat/lon grid.

[jswhit]

Potential fix in pull request #100 (branch issue99). Please give it a try and let me know if it solves it for you.

[weatherfrog]

Thanks for the quick fix. And sorry for the delay. I'll test it...

[weatherfrog]

OK, it solves the issue. Thank you.

I'm not familiar with the C lib, so I can't tell whether this is the "right" fix.