Currently, the build script for Postgres does not load all the data.
Need to fix the docker build so that it correctly loads all the data into the database
Step 8/12 : RUN for file in /tmp/states/*/*/*.csv; do dos2unix $file; sed -i -e '$a\' $file; tail -n +2 $file >> /tmp/allbudgets.csv; done
---> Running in ecd0e87f965e
dos2unix: converting file /tmp/states/DC/DC/budget.csv to Unix format...
dos2unix: converting file /tmp/states/Maryland/Allegany/budget.csv to Unix format...
dos2unix: /tmp/states/New: No such file or directory
dos2unix: Skipping /tmp/states/New, not a regular file.
dos2unix: York/NYC/budget.csv: No such file or directory
dos2unix: Skipping York/NYC/budget.csv, not a regular file.
sed: can't read /tmp/states/New: No such file or directory
sed: can't read York/NYC/budget.csv: No such file or directory
tail: cannot open '/tmp/states/New' for reading: No such file or directory
tail: cannot open 'York/NYC/budget.csv' for reading: No such file or directory
dos2unix: /tmp/states/South: No such file or directory
dos2unix: Skipping /tmp/states/South, not a regular file.
dos2unix: Carolina/Greenville/budget.csv: No such file or directory
dos2unix: Skipping Carolina/Greenville/budget.csv, not a regular file.
sed: can't read /tmp/states/South: No such file or directory
sed: can't read Carolina/Greenville/budget.csv: No such file or directory
tail: cannot open '/tmp/states/South' for reading: No such file or directory
tail: cannot open 'Carolina/Greenville/budget.csv' for reading: No such file or directory
dos2unix: converting file /tmp/states/Virginia/Accomack/budget.csv to Unix format...
dos2unix: converting file /tmp/states/Virginia/Albemarle/budget.csv to Unix format...
dos2unix: converting file /tmp/states/Virginia/Arlington/budget.csv to Unix format...
dos2unix: converting file /tmp/states/Virginia/Richmond/budget.csv to Unix format...
Removing intermediate container ecd0e87f965e
Why do we want to do this
To ensure we can get all data we have collected
How can I get started?
I would create a script to preprocess all the data into a single CSV that can be added to the DB, but you are free to tackle the problem any way you like.
What is the Task
Currently, the build script for Postgres does not load all the data. Need to fix the docker build so that it correctly loads all the data into the database
Line that causes bug https://github.com/banjtheman/defundthepolice/blob/main/docker/postgres/Dockerfile#L13
Here is the error
Why do we want to do this
To ensure we can get all data we have collected
How can I get started?
I would create a script to preprocess all the data into a single CSV that can be added to the DB, but you are free to tackle the problem any way you like.
To test run
Definition of Done
All data is correctly loaded into the Database