questions related to offline geotiff image and export option

[jaigsingla]

Hi, It looks very nice development. I have few queries:

1.Can we run this module on Geotiff image already available in my machine if yes, how? how to convert it to slippy map format ? do one requires full scene or multiple pieces of one scene?

2. Pls give guidelines for offline (without internet ) processing.

3. If I have segmented version (classified building image in png format) of the images available and just want to use your post processing tool for saving in to geojson or shape file, how to do that ?

thanks

[daniel-j-h]

1. Robosat only works with slippy map tiles; you can tile e.g. into 256x256 or 512x512 pixel tiles via
   • https://gdal.org/programs/gdal2tiles.html
   • https://github.com/cogeotiff/rio-tiler

Here's a quick and unoptimized tiler script; if your use case gets more involved you should definitely look into better ways of doing this:

https://gist.github.com/daniel-j-h/69a967d63d833f74a123e2540bd146b9

2. I don't understand what you mean by offline processing. All tools except for downloading satellite tiles from a remote server work "offline".

3. Only use the post-processing tools then.

[ocourtin]

@jaigsingla, to complete @daniel-j-h answer, on imagery tiling: You could also give a look on rsp tile, from https://github.com/datapink/robosat.pink Usually faster than the others, able to deal with coverage imagery, and to skip nodata tiles...

HTH,

[jaigsingla]

1. Robosat only works with slippy map tiles; you can tile e.g. into 256x256 or 512x512 pixel tiles via

• https://gdal.org/programs/gdal2tiles.html
• https://github.com/cogeotiff/rio-tiler

Here's a quick and unoptimized tiler script; if your use case gets more involved you should definitely look into better ways of doing this:

https://gist.github.com/daniel-j-h/69a967d63d833f74a123e2540bd146b9

1. I don't understand what you mean by offline processing. All tools except for downloading satellite tiles from a remote server work "offline".
2. Only use the post-processing tools then.

Hello daniel, All tools are offline , I agree. But how to install and use this library on offline machine is a difficult task.

Request you provide all dependencies as a zip or virtual environment image >>

thanks

[daniel-j-h]

Look into

docker image save
docker image load

to create a self-contained docker image tarball, get it onto your machine, and load it again.

[jaigsingla]

Hi daniel, I could build everything offline without docker. Now i want to use it along with bavaria dataset : Following is available in datasets: Model as check point geojson file of the heat map area two comparison images

After going through your post, we need followings: 1.Any geo-referenced image

2. From building shape file, OSM extract in geoson format of that area.
3. these two files should be converted to slippy format using ./rs cover
4. rs rasterize on osm slippy format files
5. rs train like that

I want to simulate entire scenario using bavaria data-sets first but i am unable to fetch the same. pls guide.

[daniel-j-h]

Please read the documentation for our tools and read through the diary posts here

https://github.com/mapbox/robosat#overview

they explain in detail how to run the pipeline, where to get the training data from, and so on.

[jaigsingla]

ok thanks. I am directly starting with my own data. i executed ./rs cover --zoom 20 .geojson building.tiles but it gives error of xy() takes from 2 to 3 positional arguments but 5 were given.

[daniel-j-h]

Do you have a stack trace? Which version are you running, latest? How does your geojson file look like?

[jaigsingla]

Stack trace. I used latest version only. geojson file is clipped over Ahmedabad, India .

0%| | 0/482 [00:00<?, ?feature/s] Traceback (most recent call last): File "/home/deep/anaconda3/envs/fastai/lib/python3.6/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/deep/anaconda3/envs/fastai/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/deep/building_extraction/robosat-master/robosat/tools/main.py", line 58, in args.func(args) File "/home/deep/building_extraction/robosat-master/robosat/tools/cover.py", line 30, in main tiles.extend(map(tuple, burntiles.burn([feature], args.zoom).tolist())) File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 62, in burn all_touched=True) File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/rasterio/env.py", line 397, in wrapper return f(*args, *kwds) File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/rasterio/features.py", line 280, in rasterize for index, item in enumerate(shapes): File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 59, in ((project_geom(geom['geometry']), 255) for geom in polys), File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 12, in project_geom for part in geom['coordinates'] File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 12, in for part in geom['coordinates'] File "/home/deep/anaconda3/envs/fastai/lib/python3.6/site-packages/supermercado/burntiles.py", line 11, in [mercantile.xy(coords) for coords in part] TypeError: xy() takes from 2 to 3 positional arguments but 5 were given

[daniel-j-h]

How did you generate the GeoJSON? What type of features do you have in there? From what I can recall we were supporting "Polygon" features only due to the supermercado dependency limitations.

For our extract tool see

https://github.com/mapbox/robosat/blob/3ae9c7e1d58fc446b7c8df3835c628c40ca85ba2/robosat/osm/building.py

And supermercado upstream tickets

• https://github.com/mapbox/supermercado/issues/13
• https://github.com/mapbox/supermercado/issues/30

[jaigsingla]

There is osm building shape file available over ahmedabad. Cropped subset over Ahmedabad was taken as an input and using QGIS I just saved this in geojson format and use for further processing.
recent_shapefile.zip

[daniel-j-h]

Yeap you have features in there which are not Polygon. The upstream supermercado library does not support that, see tickets linked above. I recommend you generate a GeoJSON file with Polygon features only.

[ocourtin]

@jaigsingla You could also give a try to rsp rasterize resilient with all GeoJSON geometries types.

Cf:

• https://github.com/mapbox/robosat/pull/138
• https://github.com/datapink/robosat.pink/blob/v0.5.5/docs/tools.md#rsp-rasterize

[jaigsingla]

@ocourtin can you share practical example of this ? and i guess we need to perform tilling on this also??

[jaigsingla]

Yeap you have features in there which are not Polygon. The upstream supermercado library does not support that, see tickets linked above. I recommend you generate a GeoJSON file with Polygon features only.

@daniel-j-h but these are polygons only ??

[ocourtin]

@ocourtin can you share practical example of this ?

https://github.com/datapink/robosat.pink/blob/v0.5.5/docs/101.md#retrieve-and-tile-labels

and i guess we need to perform tilling on this also??

If your labels are vector ones (GeoJSON or PostGIS), rsp rasterize performs the tiling for you. If your labels are PNG ones (it could happens), rsp tile --label is the way to deal with...

HTH,

[jaigsingla]

Hi daniel, I could build everything offline without docker. Now i want to use it along with bavaria dataset : Following is available in datasets: Model as check point geojson file of the heat map area two comparison images

After going through your post, we need followings: 1.Any geo-referenced image

2. From building shape file, OSM extract in geoson format of that area.
3. these two files should be converted to slippy format using ./rs cover
4. rs rasterize on osm slippy format files
5. rs train like that

I want to simulate entire scenario using bavaria data-sets first but i am unable to fetch the same. pls guide.

Hi daniel, I spoiled earlier virtual env and now facing problem of could not find libspatialindex_c library file. I tried to install it offline using conda install --offline option . but , still error persists.

[daniel-j-h]

You'll need libspatialindex - but you really should just use the pre-built docker images so that we can guarantee you have all packages. If you want to install on your own, check out the Dockerfile.

[jaigsingla]

solved again.

I could convert images and osm data into slippy format using ocourtin routeins. DIvided data into training, validation and images and labels also. Can i go ahead to run train routine of yours on that data now??

[daniel-j-h]

Sure why not.

[jaigsingla]

ok.

what I did is as following:

1) Created ds/training/images and ds/training/labels . similar to these ds/validation/images and ds/training/labels are created . 2) calculated weights using ./rs weights dataset_building.toml 3) run ./rs train using all parameters. 4) after 50 epochs validation IOU shows 0.45 ??

[daniel-j-h]

Okay - so what? How big is your dataset? How did you create it? Did you manually check it if it looks good? What do you want to get out of it? You have to investigate a bit on your own here..

[jaigsingla]

My dataset is not that big. it is small image file and osm data file as shared earlier too. I created data in slippy format using commands suggested by ocourtin. Manually , I check slippy files. it is properly placed in folders and looks ok. I want to generated vector shapes from image files. I need directions to investigate

[daniel-j-h]

My dataset is not that big.

Then what do you expect? We need a large dataset to learn from :)

There is only so much magic we can do here.

[jaigsingla]

shall i proceed with predictions or what you say?

[daniel-j-h]

How big is your dataset? How many samples do you have in train and val?

[jaigsingla]

Input image size is 777x674 with 4 bands. i have converted to slippy format using zoom level 19. i have used 90% for training and 10% validation.

[daniel-j-h]

If you have a single image with 777x674 pixel as your dataset, I don't think you will have much luck training machine learning models on it. It's way too small to learn anything interesting.

[jaigsingla]

So should I try on a bigger data set. Can you share sample from your side so that I am sure to get validated output

[daniel-j-h]

Check out the diary posts in https://github.com/mapbox/robosat#overview

The Tanzania example is probably as low as you should go with a few thousand of 512x512 tile images. If you follow the Bavaria diary post you can get a much bigger dataset.

[hamzahkhan]

Hi, It looks very nice development. I have few queries:

1.Can we run this module on Geotiff image already available in my machine if yes, how? how to convert it to slippy map format ? do one requires full scene or multiple pieces of one scene?

2. Pls give guidelines for offline (without internet ) processing.

3. If I have segmented version (classified building image in png format) of the images available and just want to use your post processing tool for saving in to geojson or shape file, how to do that ?

thanks @jaigsingla Any luck resolving your issue on converting segmented png to geojson? Stuck on same.

[jaigsingla]

@hamzahkhan not tried yet.

[jaigsingla]

@daniel-j-h pls make this Tanzania dataset available as a sample here. it is very difficult to fetch exact dataset without internet.

[daniel-j-h]

I no longer have access to the dataset and I'd have to fetch it from scratch myself, too.