search

SciTools-incubator / iris-esmf-regrid

A collection of structured and unstructured ESMF regridding schemes for Iris.
BSD 3-Clause "New" or "Revised" License
19 stars 17 forks source link

GridToMeshESMFRegridder: TypeError: buffer is too small for requested array #246

Open arjclark opened 1 year ago

arjclark commented 1 year ago

🐛 Bug Report

@stephenworsley - When working with large source data and trying to generate a regridder am hitting TypeError: buffer is too small for requested array

How To Reproduce

Steps to reproduce the behaviour:

  1. Start a SPICE interactive session with 180GB memory (not actually necessary but gives sufficient overhead to prove this isn't a memory availability problem)
  2. module load scitools
  3. set: ulimit -u unlimited
  4. start python
  5. load in large source file (please contact separately for path)
  6. load in a C48 target cube
  7. generate regridder
  8. wait for traceback to be generated

Expected behaviour

regridder to be generated

Environment

Additional context

We need to be able to regrid this particular large source file to a range of UGrid resolutions (target C896 for now) in order to generate ancillary files for starting LFRic. An example script and source files can be provided on request but am not able to advertise on a public repo.
stephenworsley commented 1 year ago

As mentioned offline, this seems to be due to the way esmpy handles arrays and I'm not sure how possible it would be to solve this directly so I think this could be solved with a work around. If it is only the source data which is too large, it ought to be possible to create a series of regridders from slices of the large source which are each small enough to fit in memory. once regridding is done, the result can be concatenated back together. If this can be demonstrated as a viable solution, it should be possible to refine this process to be something that iris-esmf-regrid can handle internally.

github-actions[bot] commented 1 year ago

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 90 days. Remove stale label or comment, otherwise this issue will close automatically in 7 days time.

arjclark commented 1 year ago

Still relevant and proof of concept provided by @stephenworsley seems to be working in getting round this issue.

github-actions[bot] commented 7 months ago

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

arjclark commented 7 months ago

@stephenworsley - any chance this is going to get worked on any time soon?

stephenworsley commented 7 months ago

@arjclark We're hoping to work on this as part of a focus on performance as planned in the roadmap here #278. The next release (0.10) ought to finish the work on regridder unification so that this should be included in work on 0.11 which ought to be starting up around March. If things go well there should be a release by summer, though that depends on how successful attempts at performance improvements are so I can't guarantee when these improvements will be released

github-actions[bot] commented 1 month ago

@SciTools-incubator/esmf-regrid-devs This issue is stale due to a lack of activity in the last 180 days. Remove stale label or comment, otherwise this issue will close automatically in 14 days time.

arjclark commented 1 month ago

@stephenworsley - where are we at with this?

stephenworsley commented 1 month ago

This is lined up to be the next bit of big work we do on iris-esmf-regrid. The timeline on this depends on how big this turns out to be, but I'd hope to start looking into this next month.