Define and implement a no-bulk-download policy

[alexamici]

Data providers have download policies and often require bulk download to be authenticated or limited. We don't intend to make too easy for users to make bulk downloads, instead we want to make the need for bulk download be minimised to case when it is really needed (off-the-internet machines, real time applications, etc).

Right now we refuse to clip or seed more that 9 tiles (even if they are already cached, which is dumb!).

Let's define and implement a better strategy.

[alexamici]

Now that SRTM1 is sourced from Amazon this might not be a problem anymore. This needs to be re-evaluated.

[alexamici]

I received no complain about the current 9 tiles limitation so I'll move the issue to the next milestone.

[nickwg03]

@alexamici what would define a bulk-download? Is a bulk-download a download of the entire dataset, or would a large area also be defined as that? For downloading the entirety of India, the number of tiles is greater than 9, but I wouldn't think downloading for a single country would be a bulk-download.

So having some kind of method that would ensure we don't over-tax the server, but still allow downloads larger than 9 tiles at a time would be useful. Even just waiting a period of time before starting the next batch of 9, or something simple like that.

[tommylees112]

How can I download 12 tiles ( i need them for ethiopia ).

import elevation
import geopandas as gpd

world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
ethiopia = world.loc[world["name"] == "Ethiopia"]

bounds = ethiopia.bounds
west, south, east, north = bounds = bounds.values[0]

print(bounds)

[ 32.95418 3.42206 47.78942 14.95943]

output = data_dir + 'ethiopia.tif'
elevation.clip(bounds=bounds, output=output, product="SRTM3")

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-153-7f813bad82f0> in <module>()
      6 
      7 output = data_dir + 'ethiopia.tif'
----> 8 elevation.clip(bounds=bounds, output=output, product="SRTM3")

~/anaconda3/envs/crop_analysis/lib/python3.6/site-packages/elevation/datasource.py in clip(bounds, output, margin, **kwargs)
    171     """
    172     bounds = build_bounds(bounds, margin=margin)
--> 173     datasource_root = seed(bounds=bounds, **kwargs)
    174     do_clip(datasource_root, bounds, output, **kwargs)
    175 

~/anaconda3/envs/crop_analysis/lib/python3.6/site-packages/elevation/datasource.py in seed(cache_dir, product, bounds, max_download_tiles, **kwargs)
    144     if len(ensure_tiles_names) > max_download_tiles:
    145         raise RuntimeError("Too many tiles: %d. Please consult the providers' websites "
--> 146                            "for how to bulk download tiles." % len(ensure_tiles_names))
    147     ensure_tiles(datasource_root, ensure_tiles_names, **kwargs)
    148     util.check_call_make(datasource_root, targets=['all'])

RuntimeError: Too many tiles: 12. Please consult the providers' websites for how to bulk download tiles.

[lapp0]

@tommylees112 I know you asked in 2018, but in case anyone else is coming across this and needs help:

    gen_bounds = elevation.datasource.build_bounds(bounds, margin='0')
    datasource_root = elevation.datasource.seed(bounds=gen_bounds, product='SRTM3', max_download_tiles=300)
    elevation.datasource.do_clip(datasource_root, gen_bounds, outfile_name)