Refactoring Textbook API

[azharichenko]

Objective

The goal of this PR is to increase the testability of the code, improve performance by taking advantage of features of the language and having consistent documentation throughout the code. Also, improving requests to the textbook website to cut down parsing through data.

Textbook API Changes

• Added get_textbook and get_textbooks the replacement for get_book_data
  • get_textbook is for fetching textbooks for a single course
  • get_textbooks is for fetching textbooks for multiple courses
  • Each having its own efficiencies
• Added _fetch_term_codes function to fetch semester term codes from the textbook website
  • Added TERMS constant that fetches from _fetch_term_codes
  • Added check in case of a connection error, which simply defaults to an empty list
• Added _validate_term function to check entered term is correct
  • Added warning in case connection error occurred
• Added _validate_course function check if the course is in the proper format
  • Check if it's four digits long and is a number
  • If less than four digits it will add zero in front
• Added constants within the module
  • The TERMS constant as mentioned above
  • The KEYS list constant to tell which keys to filter from the book dictionary
  • The QUERIES dictionary constant to give which query needed depending on whether we are trying to fetch course data or book data
  • The LOOKUP_ERRORS dictionary constant used for crafting a proper error message in extract_id
• Added _find_item condense the sections, instructor, and section searches into one customizable ane easy to use function
  • Created _find_sections, _find_course_id_by_instructor, and _find_course_id_by_section from _find_item
• Added _extract_id to simplify the process of extracting the specific course id from the department data
• Added _extract_books to easily get textbooks for each course
• Added DefaultDict class to allow dictionary to return None when the key isn't in the dictionary
  • This is needed in _fetch_course to prevent having to deal with the raising of KeyError
• Added _fetch_course generator to make fetching multiple courses simple by having data in a tuple
• Added new module tests allowing better testing of module then the tests for using the module
• Updated documentation and added docstrings to all functions in the module
• Now has better exception handling when unable to reach site or when offlice

  Laundry API Changes

• Removal of Unicode strings (since we stop supporting Python 2)
• Silencing of tests for the time being until refactored

[RitwikGupta]

Is this still an active WIP?

[azharichenko]

Yes, this is still a work in progress. I just haven't had time for this project recently because I'm learning other things like machine learning, python, flask, etc.

[azharichenko]

I believe at this point this PR is ready for to be reviewed.

[azharichenko]

@dzheng256 Just fixed the typo. And with the getting rid of _get_department_number I had this interesting alternative idea that I'm working on with having a dictionary with all the departments. Since both textbook and course API have common departments, why not have a dictionary in __init__.py that gives access to the departments.

class DepartmentCodes(dict):
     def __init__(self):
     with open(os.path.join(__file__, '..', 'department.txt')) as file:
         text = ''.join(file.readlines())
         dict.__init__(self, json.loads(text))
      
     @property
     def codes(self):
        return self.keys()

departments = DepartmentCodes()

This is sort of what I came up with, for now, essentially having a text file with JSON that shows various details about each department and just quickly parsing through it when the library is loaded. The simple DepartmentCodes object is a dictionary with other properties such as being able to get all the codes by saying department.codes and being able to lookup the correct query or department number value very easily.

{
    "ADMJ": {
        "subject": null,
        "value": "22399"
    },
    "ADMPS": {
        "subject": "core",
        "value": "22400"
    },
    "AFRCNA": {
        "subject": "core",
        "value": "22401"
    },
    "AFROTC": {
        "subject": null,
        "value": "22402"
    },
    "ANTH": {
        "subject": "core",
        "value": "22403"
    },
    ...,
}

[dzheng256]

@azharichenko Yeah that's a good idea.But why not just make a json instead of a txt to load?

[azharichenko]

It loads JSON from a text file.

[dzheng256]

But you are using json.loads() and forming a string. You could just make departments.json and do json.load(file) and skip the string formation is what I mean.

[azharichenko]

json.loads() takes in a JSON string and converts into a python dict hence why it's in the __init__ . There is no way to skip getting the string out of the text file.

[dzheng256]

Okay I don't think I'm explaining myself clearly. What I'm trying to say is you're trying to store JSON in a .txt when it should be a .json file which you can load without making a string. Your snippet does something like this:

 with open('departments.txt') as file:
         text = ''.join(file.readlines())
         dict.__init__(self, json.loads(text))

I'm suggesting you do this:

 with open('departments.json') as file:
         dict.__init__(self, json.load(file))

[RitwikGupta]

Dan's suggestion should work regardless of file extension. For semantics, naming it departments.json will be cleaner, but not necessary.

[azharichenko]

Didn't realize json.load actually can take in the file so I don't have to do the read line nonsense

[RitwikGupta]

@azharichenko Push a commit to fix that, and I'll review

[azharichenko]

I prefer the current format of the code. It is much more readable and we should be using the keyword is since it's a compare operator, as in we are comparing section and None.