sat-utils / sat-api

One API to search public satellites metadata on AWS
https://sat-api.developmentseed.org/search/stac
MIT License
177 stars 20 forks source link

How to deploy locally (not AWS) #224

Open jlaura opened 4 years ago

jlaura commented 4 years ago

Apologies for the likely naive question. I have a local, test STAC-API compliant instance running on in a docker cluster (minio for the holding STAC compliant JSON files, Apache, etc.) and then have STAC-Browser serving these files out in the nice web instance.

I am interested in testing sat-api without pushing to AWS. So I have also set up an ES cluster and pushed the stac metadata into that. How do I go about deploying the sat-api locally?

My current attempts to use a simple docker-compose file with the API are failing and I am just guessing at the environment variables and the ports that I need to populate/expose.

version: '3'

services:
  satapi:
    image: 'satutils/sat-api:develop'
    ports:
      - '9380:80'
    environment:
      STAC_ID: THEMIS
      ES_HOST: smalls:9300
    networks:
      - default
drewbo commented 4 years ago

@jlaura not a naive question at all. A while back this was an express API (easier to run locally) but that's deprecated now that we've moved to API Gateway. Because we aren't using SAM or serverless, there aren't great options to run this locally out of the box. Likely the "easiest" (but not that easy) way to run the API locally is by following the deploy instructions here until the deploy step and then running kes compile instead of kes deploy. This will give you a cloudformation template which you can use with localstack. There's a chance @matthewhanson is working on updated deployment steps which simplify this a bit as well

matthewhanson commented 4 years ago

Hi @jlaura , you might want to check out the new STAC-API: https://github.com/stac-utils/stac-api It's a completely refactored version of sat-api, and is compliant with the latest version of STAC, 0.9.0. I have not deployed it locally, but it uses serverless so should be easier to set up.

jlaura commented 4 years ago

@matthewhanson Thanks for the link. I have been able to make some more progress getting this running locally.

I wanted to post some info in case it is of use for anyone else trying to set this up locally.

Minio(S3) / Apache are running to store / serve the STAC compliant metadata, COGs, and thumbnail jpegs using the following docker-compose. Note that I am also using traefik locally, but this can be removed / omitted.

Minio/Apache docker-compose.yml

``` version: '3' services: apache: image: 'bitnami/apache:latest' ports: - '9080:8080' - '9443:8443' volumes: - ../single_image/:/app labels: - "traefik.enable=true" - "traefik.frontend.priority=600" - "traefik.frontend.rule=Method:GET" - "traefik.frontend.rule=PathPrefixStrip:/stac" - "traefik.backend=apache" - "traefik.docker.network=traefik-public" networks: - default - traefik-public minio: image: 'minio/minio:latest' ports: - 9000:9000 volumes: - ./minio_data/:/data labels: - "traefik.enable=true" - "traefik.frontend.priority=600" - "traefik.frontend.rule=Method:GET" - "traefik.frontend.rule=PathPrefixStrip:/minio" - "traefik.backend=minio" - "traefik.docker.network=traefik-public" networks: - default - traefik-public command: server /data environment: MINIO_ACCESS_KEY: minio MINIO_SECRET_KEY: minio123 networks: traefik-public: external : true ```

We then need an elasticsearch backend to get the metadata pushed into. I am running this in yet another docker-compose:

ElasticSearch docker-compose.yml

``` 1,1 All version: '3' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.8.7 environment: - cluster.name=docker-cluster - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 ports: - "9300:9200" kibana: image: docker.elastic.co/kibana/kibana:6.3.2 ports: - "9601:5601" ```

I am currently working on getting the ingest and api lambdas exposed as well in this setup. Luckily, lambci offers containers that I think will work.

Lambda docker-compose.yml

``` version: '3' services: lambda_ingest: image: lambci/lambda:nodejs12.x environment: DOCKER_LAMBDA_STAY_OPEN : 1 ES_HOST : http://smalls:9300/ ports: - "9001:9001" networks: - default volumes: - ./lambdas/ingest/dist:/var/task:ro,delegated - ./libs:/opt command: index.handler lambda_api: image: lambci/lambda:nodejs12.x environment: DOCKER_LAMBDA_STAY_OPEN : 1 ES_HOST : http://smalls:9300/ SATAPI_URL: http://smalls:9002/ ports: - "9002:9001" networks: - default volumes: - ./lambdas/api/dist:/var/task:ro,delegated command: index.handler ```

Using curl, I can do an ingest without a problem: curl -d '{"url":"http://127.0.0.1/stac:9000/themis/test/collection.json"}' http://127.0.0.1:9001/2015-03-31/functions/myfunction/invocations and can see in the ES logs that things are being written. It is also possible to see the return on the call to the api function: curl http://127.0.0.1:9002/2015-03-31/functions/myfunction/invocations.

Next step is to test sat-search. I am running into a few issues here that I wonder if others might have insight:

Let me know if this info is helpful or if you all prefer users to focus on doing an AWS deploy.

danielklim commented 4 years ago

@drewbo @jlaura A bit late to the party but I am putting the finishing touches on a SAT-API stack running on a local openfaas /kubernetes cluster with a local ES/KB deployment: sat-api-openfaas. My notes aren't finished yet but it does appear to work per spec with the Sentinel-5P dataset hosted on AWS. Please let me know if you have any questions -- thanks! sao-cli sao-web-ui