search

MAAP-Project / Community

Issue for MAAP (Zenhub)
2 stars 1 forks source link

Deploy Dynamic Tiler API to GCC UAT (WMTS) #298

Closed abarciauskas-bgse closed 2 years ago

abarciauskas-bgse commented 3 years ago

Acceptance Criteria:

Background:

Tasks:

Follow up tasks:

abarciauskas-bgse commented 3 years ago

Deploy Dynamic Tiler API to GCC UAT (WMTS)

TonHai1111 commented 3 years ago

Done:

abarciauskas-bgse commented 3 years ago

Working on generating a more complete / better set of test urls, but here is one where 8e9mu91qr6 should be replaced

@TonHai1111 what is the URL of your API gateway in UAH AWS account? What steps did you follow to get the maap-api-nasa working locally that weren't in that repos README? I think you mentioned something about having to run a postgres database?

TonHai1111 commented 3 years ago

This is the endpoint in my account: https://baxpil3vd6.execute-api.us-east-1.amazonaws.com/

TonHai1111 commented 3 years ago

Yes, If you follow the instructions: https://github.com/MAAP-Project/maap-api-nasa and reach to the following step:

FLASK_APP=api/maapapp.py flask run --host=0.0.0.0

Then you should need to create a postgres DB if you haven't had one. Here is the steps to create a database (named: maap_dev):

>>> 3. create an empty postgres db (maap_dev) (a fix for 'database maap_dev does not exist'):

>>> sudo -u postgres psql

>>> (in postgres shell): create database maap_dev;

>>> (in postgres shell): \q

If you have any other issues with postgres. Let's me know, I experienced a couple of them, but fixed them all, so I can send you a quick fix for that if you get any issues!

abarciauskas-bgse commented 3 years ago

@TonHai1111 thanks - can you create a branch and open a PR to maap-api-nasa to add these instructions to the README - perhaps somewhere in https://github.com/MAAP-Project/maap-api-nasa/blob/master/README.md#local-development-using-python-virtualenv ?

TonHai1111 commented 3 years ago

Sure: Here is the PR for the maap-api-nasa: https://github.com/MAAP-Project/maap-api-nasa/pull/16

I don't have permission for the titiler, can you give me write access to the titiler? Thanks!

abarciauskas-bgse commented 3 years ago

@TonHai1111 what additions or changes are you thinking about making for titiler?

I was thinking we need to add some documentation to maap-data-system-services about how to deploy titiler. Either we could add a few lines to this document: https://github.com/MAAP-Project/maap-data-system-services/blob/main/DEPLOYMENT.md about how to find the source code and deployment instructions for titiler and how to test the deployment.

I will generate another test URL for the mosaic endpoint, but I think the next step will be to include a script for testing /tiles and /mosaic endpoints with MAAP data similar to what @slesaad has done in https://github.com/MAAP-Project/maap-api-query-service/pull/4. Right now I'm thinking that test script can live in maap-data-system-services as well, what do you think?

TonHai1111 commented 3 years ago
  1. Yeah, I would also like to add some comments on a couple of issues I got during the deployment of titiler in the hope that it would help someone in the future.
  2. Yeah, agree! It would benefit for anyone who will work with maap-data-system-services. Do you want me to take a look into that and get it done?
  3. Yes, we can do somethings similar!
abarciauskas-bgse commented 3 years ago

Awesome thanks Hai, let me know if you have any questions and I will also come up with the mosaic url to include in the automated test.

abarciauskas-bgse commented 3 years ago

@TonHai1111 following up on :

I would also like to add some comments on a couple of issues I got during the deployment of titiler in the hope that it would help someone in the future.

Can you open an issue in developmentseed/titiler with your suggested changes?

abarciauskas-bgse commented 3 years ago

@wildintellect and @TonHai1111 and I have been investigating this, and at the moment, it looks like using titiler and maap-api-nasa's /wmts endpoint will not work as-is. maap-api-nasa's /wmts endpoint uses /mosaic/z/x/y.png and /mosaic/tilejson - neither of these endpoints exist in the newer titiler and from what we can tell the way you are able to pass multiple comma-delimited URLs to those /mosaic/ endpoints is not a feature of any titiler endpoint.

Right now, the code in maap-api-nasa for /GetTile collects a string of URLs which should be COGs on S3 it then calls gen_mosaic_url() to generate a complete URL to the TILER_ENDPOINT /mosaic/z/x/y, for example:

https://8e9mu91qr6.execute-api.us-east-1.amazonaws.com/production/mosaic/11/1077/1020.png?urls=s3://cumulus-map-internal/file-staging/nasa-map/AFLVIS2___001/LVIS2_Gabon2016_0308_R1808_048492_cog.tif,s3://cumulus-map-internal/file-staging/nasa-map/AFLVIS2___001/LVIS2_Gabon2016_0308_R1808_049095_cog.tif&rescale=11,57&indexes=1,2

The code in maap-api-nasa for GetCapabilities (which is used by clients such as the common mapping client which is used in maap via the ipycmc library, see documentation https://docs.maap-project.org/en/latest/visualization/using_pycmc.html) uses /mosaic/tilejson.json, see https://github.com/MAAP-Project/maap-api-nasa/blob/57236a44311f5095ed08a6e67047cbd7b9ff7048/api/endpoints/wmts.py#L177

In practice I don't think either of these /mosaic endpoints is necessary. There should really just be 2 use cases we need to fulfill:

In short, I think we can update the existing code to just use the Tile endpoint for GetTile and the bounds endpoint for GetCapabilities. Users will no longer be able to pass a list of granules or S3 URLs but visualizing all of the AFLVIS2 collection should still be possible.

Next steps:

wildintellect commented 3 years ago

There is a 3rd use case which is dynamic visualization of a collection search result (could be made to also use a list of user specified granules). There are a few options to support.

  1. Translate the search results from CMR to STAC and use the stac endpoint
  2. Return a tilejson or mosiacjson, and use the mosaicjson endpoint (which is basically what was happening before)

I could see a python module in the ADE to create mosaicjson for ADE users who want to make custom visuals with specific granules.

wildintellect commented 3 years ago

This is what we were missing, pass an arbitrary list of COG urls and get a mosaicjson back, that can then be forwarded to titiler. I was incorrect above the /stac only does single items.

from cogeo_mosaic.mosaic import MosaicJSON

# list of COG
dataset = ["1.tif", "2.tif"]
mosaic_definition = MosaicJSON.from_urls(dataset)

print(mosaic_definition.tiles)
> {"tile": {"00001": ["cog1.tif", "2.tif"]}}

https://developmentseed.org/cogeo-mosaic/intro/

How to do it from a STAC api https://developmentseed.org/cogeo-mosaic/examples/Create_a_Dynamic_StacBackend/

wildintellect commented 3 years ago

Another option is to create a custom backend for Titiler, example

abarciauskas-bgse commented 3 years ago

This is deployed to UAT, however it uses a different deployment configuration to test, since we are waiting on CDK bootstrapping of the UAT environment - which there is a GCC ticket for https://gcc-jira.gsfc.nasa.gov/servicedesk/customer/portal/4/GSD-518

The deployment is https://titiler.uat.maap-project.org/. Setting

TILER_ENDPOINT = 'https://titiler.uat.maap-project.org'

in settings.py of maap-api-nasa and running the tests runs successfully:

(maap-api-nasa) aimeebarciauskas@Aimees-MacBook-Pro-2 maap-api-nasa % python3 -m unittest -v test/api/endpoints/test_wmts_get_capabilities_new_titiler.py

test_get_capabilities_default (test.api.endpoints.test_wmts_get_capabilities_new_titiler.GetCapabilitiesCase) ... ok

----------------------------------------------------------------------
Ran 1 test in 5.814s

OK
(maap-api-nasa) aimeebarciauskas@Aimees-MacBook-Pro-2 maap-api-nasa % python3 -m unittest -v test/api/endpoints/test_wmts_get_tile_new_titiler.py
test_get_tile_collection_returns_image (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... ok
test_get_tile_no_browse (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... [<FrameSummary file /Users/aimeebarciauskas/github/maap-project/maap-api-nasa/api/endpoints/wmts.py, line 131 in get>]
ok
test_get_tile_no_identifier (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... ok
test_get_tile_returns_image (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... ok
test_get_tile_returns_image3 (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... ok
test_get_tile_returns_image4 (test.api.endpoints.test_wmts_get_tile_new_titiler.GetTileCase) ... ok

----------------------------------------------------------------------
Ran 6 tests in 5.601s

OK

So outstanding work is to update the deployment to use API Gateway + Lambda package, plus the CDK updates in https://github.com/MAAP-Project/maap-data-system-services/pull/33/files

abarciauskas-bgse commented 3 years ago

Update GCC UAT titiler deployment to match test environment