[feat] Connect NYSERDA API and Download Data from NYISO

[itsliterallymonique]

This week, you will be collecting the necessary data from the NYSERDA database and NYISO database! You will need to do these things:

1. Create a webscraper folder in our repo — this is where you will write the scripts for our web scraper a. You can write the webscraper scripts in Python
2. Connect the web scraper to the NYSERDA API and create a function that will collect the following data from both the Small solar project data and Large-scale project data: a. Project Name b. Status c. Renewable Energy Technology d. Size (watts) e. Developer f. Proposed COD (commercial operations date) g. County h. Region i. Zip Code j. Latitude (georeference) k. Longitude (georeference)
   • if the small solar project data doesn’t have the data, it’s okay – you just need to take note of this!
3. Create a function that will download the excel spreadsheet from the NYISO page a. Page: https://www.nyiso.com/connecting-to-the-grid – the sheet is downloaded by clicking: “View the Interconnection Queue”
4. Write documentation for these functions in our MTP Documentation Notion Page: https://www.notion.so/calblueprint/MTP-Documentation-4e9963e4e70d42209a1f26963b3da10b

PR Reviewers: Neha (@nehaahussain) and Monique (@itsliterallymonique)

[deenasun]

Questions:

• what field from the nyserda data should we use to measure size? I think they're in mwh
• how do we handle empty fields? should I make strings empty strings or None?

diffs for the NYSERDA small-scale solar projects dataset:

• the NYSERDA small-scale data only has project_idbut no project name
• missing fields: project_status, region, latitude, longitude
• is interconnection_date a good estimate for the proposed cod?
• what field should we use for size?

[itsliterallymonique]

@deenasun

• If there are any missing fields, the NYISO or ORES could have them
• if a field is empty, make them NULL I will email our POC tomorrow about:
• NYSERDA small-scale data being missing to see how we can find those -- but for now you can just make them NULL
• interconnection_date for the proposed_cod
• The fields we must use for size