search

worldbank / DECAT_Space2Stats

https://worldbank.github.io/DECAT_Space2Stats/
Other
1 stars 2 forks source link

Error "Getting Started Locally" #62

Closed bpstewar closed 1 month ago

bpstewar commented 1 month ago

Describe the bug I have encountered the following errors when running the "Getting Started Locally" instructions:

  1. The .sh files are difficult to run in Windows
  2. download_parquet.sh is looking for wb_aws.env, which is absent
  3. chunk_parquet.py is an empty file
  4. load_parquet_chunks.sh is missing

Screenshots image

Desktop (please complete the following information):

zacharyDez commented 1 month ago

@bpstewar the wb_aws.env has these environment variables defined:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=

If you follow docs/acceptance/db.md, I jumped to defining how to download the parquet file to be uploaded. The download.sh file was a helper for me during development. I assumed you could run aws s3 cp --quiet s3://$S3_BUCKET/$PARQUET_FILE $LOCAL_PARQUET_FILE and replace the s3 path needed.

We can reword anything in the doc and perhaps find some way to support the download process better. It'd be great if you define your ideal.

zacharyDez commented 1 month ago

@bpstewar, would running the workflow within docker be feasible on your Windows machine? It would make the bash commands available, I believe.

@bitner, @alukach, @vincentsarago; how have we previously supported running bash scripts on Windows? Is setting up a docker image a reasonable way to go about it?

bpstewar commented 1 month ago

Yes, Zac, I see that now; the download.sh is unneccessary if I have the file to ingest locally.

I was also thinking about running these commands inside the docker, that would eliminate the worry about OS issues. I will play around with that this afternoon and test some of the other commands.

zacharyDez commented 1 month ago

Ok great! Let me know if something comes up. The docker-compose is only to run a local postgis database. You may want to use the ubuntu image. We can set up a custom image to mount a volume for the files and maybe some other configurations.

bpstewar commented 1 month ago

I am able to run the postgres container with a local volume mounted, but you are correct, the image isn't complete. Could you please create a custom image for testing this ingestion?

zacharyDez commented 1 month ago

@bpstewar; were you able to get up and running with the ingestion cli?

bpstewar commented 1 month ago

Yes, I was able to stand up the docker and run the ingestion scripts on multiple PCs. I am still waiting to test a different dataset (nighttime lights), but that is taking time to prep the data. For now, we can close this