As an open source contributor, I want to have a user guide for adding data sources, so that I know what steps to take when I want to submit a PR for a new data source.

widal001 commented 2 years ago

Description There are many steps involved in adding a new data source to the ETL pipeline that populates the tool. It can be especially challenging to know which steps to take and to confirm that all of the necessary steps have been taken when submitting a PR for a new data source as an open source contributor.

Solution To simplify the process of proposing code that adds a new data source to the tool, it would be useful to have a user guide that walks contributors through all of the steps required to develop an ETL class for that data source and to cover that code with the appropriate set of tests. It would be helpful to have this guide live directly in the data_pipeline/README.md or in an new data_pipeline/etl/README.md

Describe alternatives you've considered

Code Comments: One alternative is to simply outline the steps directly in the code comments or docstrings. This approach is challenging because code comments aren't really set up to provide detailed step-by-step guidance.
GitHub Wiki: Another alternative is to leverage GitHub's wiki feature to develop this user guide. This approach is sub-optimal because the repo isn't currently using the wiki feature, and wiki docs don't travel with the repository when cloned or forked.

Tasks

[ ] Confirm the steps required to add a new data source with the Justice40 team
[ ] Add a section to the existing data_pipeline/README.md or create a new data_pipeline/etl/README.md for the user guide

Definition of "Done"

[ ] Someone unfamiliar with the tool can review the user guide and understand the steps they need to take to submit a PR that adds a new data source to the tool and have it be approved by the CEJST team for inclusion in the repository.

esfoobar-usds commented 2 years ago

Since there's a fundamental refactor for the score and dataset models using YAML that's upcoming, we want to wait until #1417 is done.

widal001 commented 2 years ago

@esfoobar-usds Sounds good! Thanks for flagging that for me. I'll subscribe to updates on that refactor to make sure I'm clued into those changes, and wait until you all are done with that.

emma-nechamkin commented 2 years ago

ahh! sorry i didn't communicate this when we last spoke, @widal001 :(

usds / justice40-tool

As an open source contributor, I want to have a user guide for adding data sources, so that I know what steps to take when I want to submit a PR for a new data source. #1468