Extract course harvester from libsys

[cbeer]

Libsys runs a thing to generate course lists for us to harvest it:

https://github.com/sul-dlss/course_reserves/blob/master/config/schedule.rb#L2

We think we could harvest the data directly from the registrar.

[thatbudakguy]

what permissions do we need to ask UIT for in order to harvest data from the registrar?

[jgreben]

Sent a message to MaIS:

For our service that consumes the CourseClass data, we would like to re-write the service to use the API (https://uit.stanford.edu/developers/apis/course) instead of the WebLogic client to consume all new CourseClass records. In order to do this, will we need any more than the same certificate file that we currently use?

On the Getting Started with the MaIS Web APIs page I see these couple of items:

- Workgroup service API specific: for existing workgroup access — add certificate as a workgroup admin
- Workgroup service API specific: to enable workgroup creation - add certificate as a member of the stem
- Develop the software to connect to the API

See: https://uit.stanford.edu/developers/apis/getting-started

[jgreben]

To get started with this, we will definitely need a way to make an HTTP connection using a cert. That is work that can happen in the meantime while we await whatever extra permission is needed. We already have the signed cert on Bodoni that we should be able to use. I think there might be some extra Workgroup access that we also need.

[jgreben]

According to MaIS we will be able to use the same cert that we currently use for the CourseClass harvester:

The sul-harvester cert already has access to the CourseClass API documented here - https://uit.stanford.edu/developers/apis/course

[jgreben]

This works in the Chrome and Safari browser if you import the sul-harvester.p12 cert and the RegisterCA.der files into the keychain Access app. You may need to use a private browsing window. I could not get it to work in Firefox. Ask Josh for the keystore password if needed:

For the Fall 23 term: https://registry.stanford.edu/doc/courseterm/1232

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE CourseTerm SYSTEM "https://registry.stanford.edu/xml/courseterm/1.0/CourseTerm.dtd">
<CourseTerm term="1232" year="2022-2023" quarter="Autumn"> 
   <course id="100676" title="Oral Communication for Graduate Students">
      <class offering="1" id="1232-ORALCOMM-219">
         <section id="01" classnum="31621">
            <component code="PRA" value="value"/>
         </section>
      </class>
   </course>
   ...
</CourseTerm>

For course with id 100676: https://registry.stanford.edu/doc/courseclass/1232-100676

<CourseClass courseid="100676" id="1232-100676" shorttitle="ORALCOMMUNICATION" source="academic" term="1232" title="Oral Communication for Graduate Students">
(Formerly CTL 219.) Graduate student speaking activities ...
  <class catalognum="219" id="1232-ORALCOMM-219" offering="1" subject="ORALCOMM" term="1232">
    <owner percent="100.0">
      <organization acadid="VPUE" adminid="FDLI" regid="...">Vice Provost for Undergraduate Education</organization>
    </owner>
    <section classnum="31621" enrollment="yes" id="01" maxenroll="35" maxunits="2" minunits="1" status="open">
      <component code="PRA" value="Practicum">Practicum</component>
      <attribute code="SUM" type="NQTR">Summer</attribute>
      <attribute code="SUMSES" type="NRPT">Summer Session Course</attribute>
      <meeting number="1">
      <instructor type="primary">
        <person regid="..." sunetid="..." univid="...">...</person>
      </instructor>
      <location building="LATHROP" id="LATHROP190" room="190">Lathrop 190</location>
      <schedule day="WED" startdate="09/26/2022" starttime="13:30" stopdate="12/09/2022" stoptime="15:20"/>
      </meeting>
    </section>
  </class>
  </CourseClass>

[jgreben]

Key files and file names added to secrets.yml: https://github.com/sul-dlss/shared_configs/pull/1835

[hudajkhan]

Some documentation about how the process works with the XML files that the registry harvester extracts and makes available for the course reserves app.

a) A cron job will run the "fetch_xml" rake task (https://github.com/sul-dlss/course_reserves/blob/master/lib/tasks/fetch_xml.rake) every morning at 4:30 am (based on https://github.com/sul-dlss/course_reserves/blob/master/config/schedule.rb). b) The fetch_xm rake task determines what the current term is, and the next one, and then looks for those files in the shared directory where registry harvester deposits the XML files representing all the courses and sections, along with instructor information, for that term. The task will copy these XML files into the app's local /lib/course_work_xml directory. c) When the user logs into the app and clicks on the "create a new reserve list", an AJAX request is made to get all the courses, which is eventually funneled to https://github.com/sul-dlss/course_reserves/blob/master/lib/course_work_courses.rb . This method traverses the XML (https://github.com/sul-dlss/course_reserves/blob/master/lib/course_work_courses.rb#L121 ) and generates a Course object. Only those sections which have instructors are included, and each section of a course (which has an instructor) gets its own Course object. This information is then made available to the UI which generates a table display that also allows for searching across different pieces of information.

[hudajkhan]

First related pull request: #474

[hudajkhan]

Second main part (reading in API results) related pull request merged: https://github.com/sul-dlss/course_reserves/pull/479

[hudajkhan]

Relates to #482

[hudajkhan]

Relates to #483

[hudajkhan]

Relates to #484

[hudajkhan]

The new approach does the following: a) The rake task is now called "fetch_courses"and uses a new class called CourseAPI class to parse the course term info returned from the MaIS Course Term API (https://uit.stanford.edu/developers/apis/course) . CourseAPI uses the MaIS individual CourseClass API to retrieve section and instructor information, and generates the JSON that the fetch_courses rake task writes to the course_work_content directory. The process of generating these JSON files takes a few hours, so this step is better left as a rake task that can run independently and then make the info available to the app. One JSON file per term is generated. The older rake task requested the current and next immediate term, so that pattern remains the same. b) The app now looks for JSON files in the course_work_content directory, and reads those into populate the list of courses from which the user can select.

[hudajkhan]

Awaiting deployment to production, with last step #485