Closed JiaqiWang18 closed 8 months ago
Some research into course evaluation:
parsing/schools/jhu/HopkinsEvaluations
parsing/schools/jhu/evals.py
was run to parse the html and inject evaluations into databaseWe should discuss further on how to proceed with implementing this feature.
First run the ingest command to convert HTML to json Then run the digets command to save json to database
python manage.py ingest jhu --types evals
Note --years
and --terms
flags don't work
It generates parsing/schools/jhu/data/evals.json
The digestir loads the json data and save them into the database. It has multiple digestion strategies to ensure data consistency
python manage.py ingest jhu --types evals --years 2015
Evals.json
formatA list of below object
{
"course": {
"code": "AS.310.305"
},
"instructors": [
{
"name": "Marvin Ott"
}
],
"kind": "eval",
"score": 4.32,
"summary": "Students praised the course...",
"term": "Fall",
"year": "2013"
}
Currently, evals.py
the ingestor uses Beautifulsoup to parse HTMLs. Since we now need to use Selenium and require authentication, it is probably better if we run this part locally for security reasons and generate a json file to be read and give this file directly to the digestor.
For the digestion step, we should aim to use the existing code in digestor.py
because it has robust logic to validate, reconcile difference, and avoid duplicated records already. So we should try to generate a json file that has the same format as the one above so we can directly give it to digestor.
Contact Details
No response
Is your feature request related to a problem? Please describe.
Many users have requested us to update the course evaluation on Semester.ly
Describe the solution you'd like.
Take evluations published from the official source, and add them to course modals in a format similar to: https://jhu.semester.ly/course/EN.660.332/Fall/2023
It does not seem like there is an API for this data. So a solution is to create a web crawler and ingest these data manually.
Describe alternatives you've considered
Have users submit course evals thru semester.ly. Not recommended if we are not offering different questions from these on the official forms.
Additional Information
No response
Code of Conduct