Adding Data - Githubissues

LabConnect-RCOS / LabConnect-Backend

Centralized website for people to post and find lab and research opportunities on campus

Apache License 2.0

1 stars 1 forks source link

Adding Data #42

Open sjaybush18 opened 11 months ago

sjaybush18 commented 11 months ago

Database integration with the webscraping scripts for consistent automation
May require updating and optimizing the webscraping scripts
Ensuring up-to-date professor, department, and opportunity data
Ensure these changes and updates work with the existing backend and frontend

swadekar commented 11 months ago

Implement error handling in webscraping scripts.
Document webscraping process and dependencies.
Schedule periodic updates for data consistency.
Test integration with the database for compatibility.
Document schema changes for new data fields.
Optimize webscraping performance.

RafaelCenzano commented 7 months ago

As a note, this should be done as a job where we scrape, process data, and input it into the database (ETL)

RafaelCenzano commented 4 months ago

Webscraping is not required, it can be useful for auto updating but can also break easily. Preference is creating ETL scripts/methods and separately setting up either webscraping or just downloading of data manually for example courses should change regularly enough to waste time now automating that process, departments similarly aren't updated very often

Course data: #67 Department data: #164