Closed qasim closed 8 years ago
Yeah, the scraper is basically complete, just have to adapt it to Cobalt design.
Also, I think each scrape should only include schedules for December to August of the provided year.
So for 2015:
And 2016:
I agree, that makes more sense. Then, year calculating logic would be:
etc.
Started working on this. Current schema looks like:
{
"date": String,
"start_time": Number,
"end_time": Number,
"sections": [
{
"section": String,
"location": String
},
...
]
}
Currently, date
is in ISO 8601 format with time as 00:00:00
, would it be better to merge date + time and just have start_time
and end_time
(both in ISO 8601)?
Here's some example output:
"MAT137Y1Y": {
"date": "2016-04-21T00:00:00",
"start_time": 9.0,
"end_time": 12.0,
"sections": [
{
"section": "A - H",
"location": "EX 100"
},
{
"section": "I - R",
"location": "EX 200"
},
{
"section": "S - TH",
"location": "EX 300"
},
{
"section": "TI - WO",
"location": "EX 310"
},
{
"section": "WU - YE",
"location": "EX 320"
},
{
"section": "YI - ZHAN",
"location": "SF 2202"
},
{
"section": "ZHAO - ZZ",
"location": "SF 3202"
}
]
}
Also, how should data be separated – should each course have its own output file (MAT137Y1Y.json
, MAT138H1S.json
, ...) or should it be split up by month (DEC15.json
, APR16.json
, ...) with each file containing data for all courses from that semester?
I think start_time
and end_time
is a better idea, which would be ISO 8601 datetimes. You could keep date
as well and have it as just an ISO 8601 date (2016-04-21
).
As for the data, I think it will be worth separating data by course . A course_id
parameter could be added, like CSC165H1S20161
, so if someone using this course API wanted to, they could reference the course API to get more info.
Another reason for the course being top level is that a person using the API is more likely to search for their course's exam times than a date's exams, I'd say.
I think most people would be looking for a particular exam period (e.g. apr16), and then some course(s) within it.
In that case, user would filter for exam_session
and then their course, returning the course_id for their session and it's times.
Based on the data we have access to, I think it'd be easier to have a course_id
that matches a course_code
from the course API (so CSC165H1S
instead of CSC165H1S20161
).
The way the scraper currently works is that you give it a year and it'll scrape data for each exam period of that year. So 2016
will scrape dec15
, apr16
, june16
, and aug16
and then create an individual output file for each of them.
If we were to create a separate file for each course_id
, how would we differentiate between the exam periods? Would adding a period
key work? I guess the schema would look something like:
{
"course_id": String
"date": String,
"start_time": String,
"end_time": String,
"period": String,
"sections": [
{
"section": String,
"location": String
}
...
]
}
Some full year courses have exams in both December and April, so it might be better to give each section
that key instead.
{
"course_id": String
"date": String,
"start_time": String,
"end_time": String,
"sections": [
{
"section": String,
"location": String,
"period": String
}
...
]
}
This is what I was picturing: The mapping to the end of the course IDs are:
dec15
F
-> 20159
apr16
S
-> 20161
Y
-> 20159
june16
F
-> 20165F
aug16
S
-> 20165S
S
-> 20165
So if we assume, say CSC165 is taught in every single semester of the school year, the IDs are:
CSC165H1F20159
CSC165H1S20161
CSC165Y1Y20159
CSC165H1F20165F
CSC165H1S20165S
CSC165Y1Y20165
which are all unique and belong to a single exam period.
{
"course_id": "MAT137Y1Y20159",
"course_code": "MAT137Y1Y",
"period": "apr16",
"date": "2016-04-21",
"start_time": "2016-04-21T09:00:00-04:00",
"end_time": "2016-04-21T12:00:00-04:00",
"sections": [
{
"section": "A - H",
"location": "EX 100"
},
{
"section": "I - R",
"location": "EX 200"
},
{
"section": "S - TH",
"location": "EX 300"
},
{
"section": "TI - WO",
"location": "EX 310"
},
{
"section": "WU - YE",
"location": "EX 320"
},
{
"section": "YI - ZHAN",
"location": "SF 2202"
},
{
"section": "ZHAO - ZZ",
"location": "SF 3202"
}
]
}
So now if a user wants to find this class, they'd filter course_code:"MAT137Y1" AND period:"apr16"
Nice, that's a lot cleaner. How do we deal with December exams for full year courses?
We could introduce an id
for exams? I don't know if this is the cleanest or not :P
{
"id": "MAT137Y1Y20159DEC15"
"course_id": "MAT137Y1Y20159",
"course_code": "MAT137Y1Y",
"period": "dec15",
"date": "2015-12-17",
"start_time": "2015-12-17T09:00:00-04:00",
"end_time": "2015-12-17T12:00:00-04:00",
"sections": [
...
]
}
(if MAT137 had december exams)
Made suggested changes – went with this id
format for the time being (I think it looks fine, maybe an underscore between the period and the course id?)
Here's some example output:
{
"id":"CSC165H1F20159DEC15",
"course_id":"CSC165H1F20159",
"course_code":"CSC165H1F",
"period":"dec15",
"date":"2015-12-17",
"start_time":"2015-12-17T19:00:00-04:56",
"end_time":"2015-12-17T22:00:00-04:56",
"sections":[
{
"section":"A - KAP",
"location":"HA 403"
},
{
"section":"KAY - SH",
"location":"UC 266"
},
{
"section":"SI - Z",
"location":"UC 273"
}
]
}
{
"id":"CSC165H1S20161APR16",
"course_id":"CSC165H1S20161",
"course_code":"CSC165H1S",
"period":"apr16",
"date":"2016-04-14",
"start_time":"2016-04-14T19:00:00-04:56",
"end_time":"2016-04-14T22:00:00-04:56",
"sections":[
{
"section":"A - H",
"location":"BN 2N"
},
{
"section":"I - O",
"location":"BN 2S"
},
{
"section":"P - ZHAN",
"location":"BN 3"
},
{
"section":"ZHAO - ZZ",
"location":"ST VLAD"
}
]
}
Also, there seems to be a special course, SOC102STU, in the December schedule.
Apparently its a course from the Steps to University program. It's not on CourseFinder, so it's not in the courses API. For the time being I've ignored it and just left it out.
+1 to leaving the special case out.
The id
looks good in the current form. Output looks great!
We can .upper()
the period
key's value so that it's the same as the end of id
.
UTSG scrapers are ready to go https://github.com/cobalt-uoft/uoft-scrapers/pull/40.
I've kind of started on the UTSC / UTM scrapers, but they only provide data for the current period (UTSC has PDFs for past schedules) and I'm not too sure what happens when it isn't exam season.
It shouldn't be hard to produce the same schema with this data though, if we decide to do so.
For the case of UTSC / UTM, we can provide current period data only then.
I tested the scraper out and the results look good! Thanks for the contribution :)
Scraper should check the current year to determine what pages to scrape. Example:
etc. if the page returns a
404
, then just skip it and try the rest.