search

mapbox / robosat

Semantic segmentation on aerial and satellite imagery. Extracts features such as: buildings, parking lots, roads, water, clouds
MIT License
2.02k stars 382 forks source link

rs rasterize crashes after a while - memory consumption? #143

Closed MichaelTUD closed 5 years ago

MichaelTUD commented 5 years ago

Hi,

I'm following the guide for running robosat pipeline at https://www.openstreetmap.org/user/daniel-j-h/diary/44321.

When running rs rasterize it keeps 'crashing' after a while in the 2nd step. pic01 I managed to set up robosat under google colab because I thought it was due to the performance of my laptop and I am facing the same problem: pic02

I think it's due to memory consumption as mentioned in #25 . The issue closed with #134 but i'm not sure where to use this batch rasterization.

Otherwise what do I have to consider if I have to split the Geojson file by myself?

Is it possible to resize the map.osm.pbf with osmium, e.g. to quarter it so that I get buildings[1-4].geojson and buildings[1-4].tiles, rasterize them all individually and in the next step I put together the individual slippy map directories to a big directory (for tilesas well as for masks)?

daniel-j-h commented 5 years ago

rs rasterize can also do batch rasterization: https://github.com/mapbox/robosat/pull/134 It works by iteratively rasterizing batches of features into (potentially the same) images.

For batch extraction rs extract I just opened https://github.com/mapbox/robosat/pull/148 which will allow you to easily split the features into batches of a certain size. Then you can call rasterize on all of the generated extracted files (not in parallel, since rasterize will both read as well as write).

The end result is the same compared to non-batched extract and rasterize.