Kernel crashes on grids generation for large countries at `zoom_level = 18`

[tm-abby-moreno]

📓 Context

Below I wrote down a series of assumptions and tests I did to see if it would fix the problem leading me to concluding that morecantile seems to be the culprit in the grids generation issues we've been facing.

Reiterating that the initial goal was to generate bingtile zoom level 18 grids for Indonesia to match the resolution of the other data marts.

🧪 Assumptions and tests

1. Indonesia is very large for the usual grid generation workflow.

• Kernel keeps shutting down for all admin levels.
• Tried different administrative levels. Tested from adm0 to adm3 and all failed.
• Memory reaches 99% and then shuts down. Experiencing memory errors.

2. Since even at the lowest level the kernel dies, perhaps the geometries of the regions are too complicated adding to the complexity of generating grids.

• Simplify geometries at 20m.
• Tried this for adm2 and adm3 --- both still failed.
• Simplify even further to reduce geometry complexities.
  • Generate bounding boxes all over the adm2 regions.
  • Generate intermediate bounding boxes (bingtile = 16) all over the adm2 bounding boxes for the whole indonesia. (THIS WORKED 🟢 )
  • Generate the high res bingtiles over the intermediate grids for each adm2 area. (THIS FAILED 🔴)

3. The jump in sizes from a small adm2 area might be causing out of memory issues since there are areas that could be too large to process.

• Chunk the intermediate grids for a constant size of geometries to be inputted to the grid generation function.
• Tried 150k, 100k, 60k, 40k, 15k, and 5k chunk sizes, but all failed.
• “Silver lining” is that this was the longest run. It always ran for 40-50 mins. no matter the chunk size threshold.

4. Overhead / some memory retention is happening within geowrangler code because simplification and chunking at varied sizes does not solve the problem.

• Added psutil functions to track memory at each loop during the chunking. Tested this for chunk_size of 15,000 intermediate grids at each iteration.
• Chunk 4
  • Memory usage after grid generation chunk 4: 9473.66 MB
  • Memory usage after saving file chunk 4: 9576.93 MB
  • Memory usage after cleanup chunk 4: 8964.80 MB
• Chunk 6
  • Memory usage after grid generation chunk 6: 10835.73 MB
  • Memory usage after saving file chunk 6: 10966.70 MB
  • Memory usage after cleanup chunk 6: 10473.70 MB

5. Kyle’s fix for the PH unfiltered grids for generate_grid function works for the whole PH at adm0.

• Failed. Tried this for the whole intermediate grids and for adm0.
• Reason for using the whole dataframe as in input to grid generation is to see where the function will stop in the pipeline
• FINDINGS: Kernel crashes at Get all tiles for polygons section (related to next)

7. Tile retrieval is causing an additional processing overhead due to list comprehensions

• First test: Convert list comprehension to a generator object that yields each value instead of storing to a list and see if it goes past the function. Reason is the list might be causing the memory retention issues.
  • FALSE. Already a generator object.
• No more optimizations to do on this function. At this point it seems to be related to morecantile as Butch has mentioned.

8. Inspect Morecantile

• Open item, have not done a deep-dived yet on the package itself. Will do other optimizations at a lower level instead (currently ongoing).

[joshuacortez]

@tm-abby-moreno FastBingTileGenerator is now merged to master! 🔥 See sample usage here.

Thanks so much for your preliminary investigation that uncovered the bottlenecks. It cascaded into the development of the faster grinding approach 😄