Update features API with Alexey's 2019 and 2020 data

[abarciauskas-bgse]

AC:

• [ ] Deploy Features API and database
• [ ] Ingest eis_fire_perimeter, eis_fire_fireline, eis_fire_new_fire_pixels data including datetime column

Details

• Alexey provided new data in s3://veda-data-store-staging s3://veda-data-store-staging/EIS/other/feds-WesternUS/2019/Largefire/00-latest s3://veda-data-store-staging/EIS/other/feds-WesternUS/2020/Largefire/00-latest
• Each folder has 3 FlatGeobuf files, corresponding to fire perimeters, active fire lines, and active fire pixels. Those should be distributed using separate features endpoints (separate tables in the feature database, though all linked by fid ). The files are separated by year, but years should be combined in the feature service.
• Tim previously added fires data using this code base: https://github.com/NASA-IMPACT/veda-features-api
• The current stack is deployed to the API Gateway in the covid-dashboard 853 AWS account with the API endpoint https://sbbj812m1j.execute-api.us-west-2.amazonaws.com/

Current state:

• Tim's previously deployed cloudformation stack is tifeatures-timvt-staging
• https://sbbj812m1j.execute-api.us-west-2.amazonaws.com/ is returning an internal error "cannot import name 'as_shape' from 'pygeoif.geometry'" - it's not clear when or why this started happening
• I have attempted to write new features to the DB by downloading them through a cognito client session (see https://github.com/NASA-IMPACT/veda-documentation/blob/ab/add-cognito-auth-example/cognito_login_example.ipynb)

year = 2019 # 2020
collection_prefix = f'EIS/other/feds-WesternUS/{year}/Largefire/00-latest'
files = client.list_objects_v2(Bucket=bucket, Prefix=collection_prefix)
first_object = files['Contents'][0]['Key']
filename = client.download_file(Bucket=bucket, Key=first_object, Filename=first_object.split('/')[-1])
# Fetch DB credentials output from Tim's stack and use them to populate this script:
!ogr2ogr -f "PostgreSQL" \
  PG:"host=HOST \
  dbname=DBNAME user=USER password=PASSWORD" "fireline.fgb" \
    -nln fire_boundearies_2019 -append

• This completed after a few minutes (5? But on a slow wifi). I also had to add an IP to the DB security group.

[abarciauskas-bgse]

I started to try and run tifeatures locally and I'm getting this error: asyncpg.exceptions.UndefinedTableError: relation "geometry_columns" does not exist

Also note I had to install postgis extension on my database. The tifeatures documentation may be incomplete as the setup locally does not include much instructions about how to setup the local postgres postgis database.

[abarciauskas-bgse]

I was able to get the tifeatures app to run locally after running these commands:

➜ psql postgis                 
psql (14.1, server 14.5 (Homebrew))
Type "help" for help.

postgis=# CREATE EXTENSION postgis;
CREATE EXTENSION

with DATABASE_URL=postgresql://postgres:@0.0.0.0:5432/postgis

and then I was able to insert into the database using:

ogr2ogr -f "PostgreSQL" \
  PG:"host=0.0.0.0 dbname=postgis user=postgres" \
  -t_srs EPSG:4326 \
  "../veda-documentation/fireline.fgb" \
  -nln fire_boundearies_2019

[vincentsarago]

sorry about that @abarciauskas-bgse, I'll try to update the documentation.

FYI: We are moving away from tifeatures/timvt to https://github.com/developmentseed/tipg (empty for now) which will have both mvt and features endpoints.

[j08lue]

see https://github.com/NASA-IMPACT/veda-documentation/blob/ab/add-cognito-auth-example/cognito_login_example.ipynb

Nice example, @abarciauskas-bgse!

Let me know when it would make sense to replace the ogr2ogr upload by a Python solution, for example with GeoDatatFrame.to_postgis. Should be pretty straightforward.

[abarciauskas-bgse]

@vincentsarago oh I wasn't expecting the documentation to be complete btw :) I realize that some of the "setup postgis" is left as an exercise to the reader (and I am a novice postgis user)

[abarciauskas-bgse]

I started to lookinto using GeoDataFrame.to_postgis but an error arose, see: https://github.com/NASA-IMPACT/veda-features-api/blob/ab/add-features-script/scripts/add_fire_features.ipynb

Also recapping some discussion from slack with Alexey, Bitnerd and Alex M:

• for now, we're going to create 3 tables eis_fire_perimeter, eis_fire_fireline, eis_fire_new_fire_pixels. Data from different years should be appended to the appropriate table.
• this may be replaced by a more robust solution, with different databases for different collections or a "source" column when we have more features API use cases