Add ZOOM_LEVEL_STRATEGY for picking overview zoom levels

[dmahr1]

Work done

• Added new ZOOM_LEVEL_STRATEGY environment variable that is intended to mimics the ZOOM_LEVEL_STRATEGY creation option in GDAL's COG driver
  • AUTO (default) upsamples or downsamples the zoom level whose resolution is closest to the desired resolution.
  • LOWER always upsamples the zoom level immediately below the desired resolution (requesting less data).
  • UPPER always downsamples the zoom level immediately above the desired resoluion (requesting more data).
  • Another integer from 0 through 100: if the desired resolution is more than this percentage of the way from the zoom level immediately below to the zoom level immediately above, then upsample the zoom level immediately below, else downsample the zoom level immediately above.
  • Internally, AUTO and LOWER and UPPER are just shorthand for 50 and 100 and 0, respectively.
• The intuition behind the integer values is best illustrated by a motivating example.
  • Consider a COG intentionally generated at zoom level 18 with 0.597 meter resolution and a 2x overview at 1.194 meter resolution.
  • Due to floating point imprecision, these may be saved as 0.597000001 and 1.1940000001, respectively.
  • A client requests zoom level 17 tiles at 1.194 resolution.
  • Under the UPPER/0 strategy, the request would be fulfilled by heaving downsampling the zoom level 18 tiles because 1.194 < 1.1940000001. For all practical purposes, this quadrupling of the requested data is wasteful and unnecessary.
  • Under the 1 strategy, the threshold is computed as 1% of the way from the zoom level 17 res to the zoom level 18 res or 1.1940000001 - (1.1940000001 - 0.597000001) * (1 / 100) = 1.188030000109. Since 1.194 >= 1.188030000109 then the request would be fulfilled by slightly upsampling zoom level 17 tiles.
  • Having this be an arbitrary integer allows users to customize the amount of wiggle room before going to the next higher zoom level.

Links

• Slack discussion that led to discovering this bug
• GDAL's COG driver

[kylebarron]

Man that's so much easier in Numpy :smile:

[dmahr1]

@geospatial-jeff I have very little experience contributing back to open source so please let me know if I'm not following decorum or best practices.

As discussed in the cogeotiff Slack, this is a work in progress PR for picking the closest overview level in a more reliable way. I would love to add some tests, but I am getting a large number of failures when I run pytest tests/ in a dedicated virtual environment. I think it has to do with accessing data via s3:// style links. I'm kind of a noob with boto (I use Google Cloud day-to-day) so perhaps there's some kind of initialization I'm neglecting?

Also, is there any way to run CI on PRs or is it only for builds of master?

[geospatial-jeff]

Also, is there any way to run CI on PRs or is it only for builds of master?

I think now that we've switched to github actions this might run CI on your next commit. But I'm not 100% sure. If it doesn't i'll have to take a deeper look.

I would love to add some tests, but I am getting a large number of failures when I run pytest tests/ in a dedicated virtual environment.

It should be easier now that we are using tox (thanks @vincentsarago ). I think you do need aws credentials configured in your environment or via aws configure (otherwise boto is unhappy), but it doesn't matter what your creds are as its accessing data in a public bucket. With #98 we can remove this dependency.

[kylebarron]

this might run CI on your next commit

I think you need to merge master into your branch

[dmahr1]

@geospatial-jeff @kylebarron Thanks for the suggestions! I pulled upstream, got my aws CLI configured, and on the master branch all tests are now passing. A few tests are failing on my PR branch but they are related to my change, so I'll fix them up and add some new tests as needed.

The CI did run but it ran into the same S3 credentials issue. It looks like @vincentsarago is looking into that in #100.

Really appreciate your help and patience here. 👍

[dmahr1]

@geospatial-jeff I've refactored this PR such that the ZOOM_LEVEL_STRATEGY environment variable controls the behavior. The business logic is not quite as elegantly simple as it was with np.argmin but it's a lot more flexible now. I put an explanation in the README (below) but I am open to other suggestions.

• ZOOM_LEVEL_STRATEGY - mimics GDAL's ZOOM_LEVEL_STRATEGY creation option:
  • AUTO or 50 (default) upsamples or downsamples the zoom level whose resolution is closest to the desired resolution.
  • LOWER or 100 always upsamples the zoom level immediately below the desired resolution (requesting less data).
  • UPPER or 0 always downsamples the zoom level immediately above the desired resoluion (requesting more data).
  • Another integer from 0 through 100: if the desired resolution more this percentage of the way from the zoom level immediately below to the zoom level immediately above, then upsample the zoom level immediately below, else downsample the zoom level immediately above. For example, 1 is the same as UPPER unless the COG's resolution is very close to the zoom level below e.g. due to floating point imprecision.

Also all tests are succeeding for me locally, but they are always failing in CI. Do you or @vincentsarago have any suggestions to fix the CI stuff?

P.S. Happy new year!

[vincentsarago]

@dmahr1 I think the CI is falling because of how actions works (and because we are not yet using moto to mock boto3).

[geospatial-jeff]

Yes @vincentsarago is correct