[DRAFT] [DO NOT MERGE] Create Course Data Scraper

unc-csxl / csxl.unc.edu

CS Experience Labs' web application.

https://csxl.unc.edu

MIT License

9 stars 7 forks source link

[DRAFT] [DO NOT MERGE] Create Course Data Scraper #380

Closed ajaygandecha closed 3 months ago

ajaygandecha commented 4 months ago

This draft pull request shows the preliminary design of a data scraper to upload all 9000+ courses in the UNC catalog into the CSXL Courses database table.

Open Questions:

Adding this to reset_demo does somewhat slow down this script. Thoughts on this?
If this is script is to be run independently, insert_scraped_data()can be called. Is this construction good?

Next Steps

Once this script is finalized, I will begin working on the section data scraper.

ajaygandecha commented 3 months ago

@KrisJordan, closing this ticket for now - I think there might be a better way to go about this in the future, plus the academics feature likely needs some upgrades to support all of the courses.