Closed galengorski closed 3 years ago
Hi Scott, thanks for the pointers and edits. I have edited the pipeline to address the requested changes. As an aside, I'm not sure the best ways to highlight how the changes that I've made address your edits. Should I do it here? Should I do it as a response to each of your individual comments? Or should I just trust that you can track them down by looking at the changes in the code? Maybe it varies by preference, just curious. Thanks!
I tried to improve this pipeline by creating a target for each individual site. I also broke the
site_data
target into 4 separate targets: 1)create_directories
creates directories where each site's data will be downloaded, 2)site_data_n
downloads each site's data and writes it to the directory (one target for each site), 3)site_concat_data
reads the downloaded data and concatenates the data into a single data frame, finally 4)site_info_csv
queries the site info for the site and writes it to a csv.