search

NYCPlanning / ae-data-flow

Data pipelines to populate Application Engineering databases
1 stars 0 forks source link

test API DB changes using the Zoning API and Zoning PoC #29

Closed damonmcc closed 2 months ago

damonmcc commented 3 months ago

Since this repo's purpose is to change data in remote databases, we must use data in those databases to confirm that this repo's code/design works.

One way to do that is to use remote Data Flow and API databases (dev for example) with local instances of the Zoning API and the Zoning Map PoC

  1. set .env files for remote DB use (see 1password for dev database credentials)
  2. start local API and App servers
  3. see data in App

Another way is to use local everything: DBs, API, PoC

  1. set .env files for local use
  2. start two dockerized DBs (Data Flow and API)
  3. run Data Flow bash scripts
  4. start local API and App servers
  5. see data in App
damonmcc commented 3 months ago

using the changes in this data flow PR and this Zoning API PR, I was able to:

  1. run a local Zoning Map PoC
  2. which was using a local Zoning API
  3. which was using from a remote API database (dev environment)
  4. whose data was populated by the data flow github action (run here)

screenshots

API testing using hoppscotch

Screenshot 2024-03-29 at 6 32 45 PM

Zoning Map POC

Screenshot 2024-03-29 at 6 35 16 PM
dhochbaum-dcp commented 2 months ago

I was able to get it all up and running locally, using Docker for both the Zoning API and the Data Flow, to create the completed tables in the Zoning API database. Did this with the help of @damonmcc. I had 2 pieces of feedback:

pratishta commented 2 months ago

I was able to test Data Flow DB update to Zoning API DB locally by setting up with changes from this PR. The set up downloaded, transformed and created the tables in Data Flow DB.

I tested changes by changing the address of the first row in tax_lot table in Data Flow DB (1000010010) to 'HELLO WORLD'. I ran ./bash/export.sh and ./bash/update_api_db.sh to create the db.dump and update the local Zoning API DB. I was able to see the 'HELLO WORLD' address for the Zoning API DB afterward.

bmarchena commented 2 months ago

I was able to get everything running locally! I ran through the README and was able to successfully run all scripts