This is the issue to create the PNW Census pipeline.
The pipeline will need to:
[ ] validate a single csv file
[ ] incorporate the new "year_month" element from the filename in exactly the same way "year" is currently handled; for the purpose of differentiating between different returns e.g. PNW Census will allow 2024_Jan, 2024_Feb.... where 903 only has 2024; adding the column "year_month" instead of "year" to all files
[ ] produce a cleanfile per input following a pipeline.json to be defined with client
[ ] hash the values for unique identifiers in the census; and apply the same hash to 903 identifiers such that a single data model can be created at the end
[ ] concatenate cleanfiles at the la level
[ ] create a reports output for each list for the region
[ ] make the usual logs and outputs available in the standard places in the infrastructure
MichaelHanksSF
commented
1 week ago
This is the issue to create the PNW Census pipeline. The pipeline will need to: