incorporate alternate subject info?

npfoss commented 5 years ago

Someone got back to me about why the developer.mit.edu subjects API wasn't working. I don't know if it has everything we want but it is an MIT-maintained source of this information so I thought I would record its existence in this issue.

The API has two endpoints:

And we need to pass valid credentials to get info back. I got access to this a while ago so feel free to use mine:

{headers:{client_id:'01fce9ed7f9d4d26939a68a4126add9b', client_secret:'D4ce51aA6A32421DA9AddF4188b93255'}}

venkatesh-sivaraman commented 5 years ago

I agree that we should eventually switch over to using this API! For those who haven't tried it, here's an example of what some of the output looks like:

{
    "termCode": "2019FA",
    "subjectId": "6.141",
    "academicYear": "2019",
    "title": "Robotics: Science and Systems",
    "cluster": "(Same subject as 16.405J)",
    "prerequisites": "((1.00 or 6.0001) and (2.003, 6.006, 6.009, or 16.06)) or permission of instructor",
    "units": "2-6-4 Institute LAB",
    "optional": null,
    "description": "Presents concepts, principles, and algorithmic foundations for robots and autonomous vehicles operating in the physical world. Topics include sensing, kinematics and dynamics, state estimation, computer vision, perception, learning, control, motion planning, and embedded system development. Students design and implement advanced algorithms on complex robotic platforms capable of agile autonomous navigation and real-time interaction with the physical word. Students engage in extensive written and oral communication exercises. Enrollment limited.",
    "offered": false,
    "instructors": "L. &nbsp;Carlone, S. Karaman",
    "instructorDetails": null
}

We're well equipped to use this data, although we would still have to parse it and pass it through FireRoad since some of the information groupings are a bit bizarre (e.g. the units field). The main issue is that it doesn't provide everything, with the most important aspect being the class schedules and the offering pattern. If there's another way we can get that information, that would be great, but (at least right now) switching to this API won't decrease the amount of parsing FireRoad has to do :'(

In any case, I've written a basic client that augments the existing catalog parser with basic info from developer.mit.edu, and put it in a branch. If the subject listing site changes format or we decide to just make the switch, we'll at least have a starting point.

npfoss commented 5 years ago

That looks good to me, it just augments the big parse with whatever it can from the API, right? And the API info overrides the other parse? Is there any reason not to add this now? We could also get a lot more historical class data from running it a few times on old years. There are definitely classes people have taken that aren't in there. (6.005 for example, still has people at MIT who have taken it I believe)

Thanks for figuring out how to include this, Venkat!

sipb / courseroad2

incorporate alternate subject info? #256