search

datakind / data-recipes-ai

MIT License
2 stars 0 forks source link

Ingestion pipeline got an error #33

Open NicholasStC opened 1 month ago

NicholasStC commented 1 month ago

Error while ingesting data. I ran these commands:

  1. docker exec -it haa-ingestion /bin/bash
  2. python3 ingest.py

Version observed: commit: 5b91e0072ba5ffc0f4a3c7f6f502a2ea1533db3f

`URL https://stage.hapi-humdata-org.ahconu.org//api/v1/metadata/currency/?limit=1000&offset=1000&app_identifier=ZGF0YS1yZWNpcGVzLWFpLXN0YzpuaWNob2xhcy5kcmFib3dza2lAc2F2ZXRoZWNoaWxkcmVuLm9yZw%3D%3D [] 127 0 127 Before DF 127 After DF Saved ./api/hapi//api_v1_metadata_currency.csv /api/v1/util/version https://stage.hapi-humdata-org.ahconu.org//api/v1/util/version URL https://stage.hapi-humdata-org.ahconu.org//api/v1/util/version/?limit=1000&offset=0&app_identifier=ZGF0YS1yZWNpcGVzLWFpLXN0YzpuaWNob2xhcy5kcmFib3dza2lAc2F2ZXRoZWNoaWxkcmVuLm9yZw%3D%3D Traceback (most recent call last): File "/app/ingest.py", line 432, in main() File "/app/ingest.py", line 408, in main download_openapi_data( File "/app/ingest.py", line 128, in download_openapi_data output = get_api_data(url, query, data_node) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/ingest.py", line 70, in get_api_data data = data[data_node]


KeyError: 'data'
root@48ee003f3007:/app# `
dividor commented 3 weeks ago

Thanks! This should be resolved now. It was a new metadata endpoint introduced in the API which didn't have a 'data' node.