I added a function into the fetch step that gets some info about each site (fetch_site_info). This information includes the start and end dates of all parameters available at that site. I process this into a shorter table per site with each parameter code and the start and end dates for its observations (process_site_info_to_csv). The earliest of these start dates is used for the API call to pull all the data for that site. Today's date (on whichever day you run the script) is used as the end date. I have removed start and end dates from the config file since these are created in the code now.
A few other changes to note in this pull request:
Between the pulling of the raw parameter info file (with parameter codes, names, etc.) and processing it into a csv, no information was dropped. Therefore, I decided to process the parameter info into a csv in the fetch step, instead of the munge step. I am only saving the csv file, and not the original text file (They were basically totally redundant). This way, we can keep out munge outputs to just data, not any metadata files.
I created a new folder within the 01_fetch/out folder called metadata. This folder holds the parameter info table and the tables with start/end dates for each site. I think we should store all metadata in this folder, and keep the root of the out folder to store just the data files.
Happy to talk about this in our meeting next week, but feel free to reach out with questions before then too!
@galengorski and @ted80810 - would it be possible to review this? I was to create another pull request for the gap analysis, but it would be easiest if this was merged first!
I added a function into the fetch step that gets some info about each site (fetch_site_info). This information includes the start and end dates of all parameters available at that site. I process this into a shorter table per site with each parameter code and the start and end dates for its observations (process_site_info_to_csv). The earliest of these start dates is used for the API call to pull all the data for that site. Today's date (on whichever day you run the script) is used as the end date. I have removed start and end dates from the config file since these are created in the code now.
A few other changes to note in this pull request:
01_fetch/out
folder calledmetadata
. This folder holds the parameter info table and the tables with start/end dates for each site. I think we should store all metadata in this folder, and keep the root of theout
folder to store just the data files.Happy to talk about this in our meeting next week, but feel free to reach out with questions before then too!