feat: add scraper functionality and store with TinyDB

compsci-adl / courses-api

API for getting University of Adelaide course information

MIT License

11 stars 3 forks source link

feat: add scraper functionality and store with TinyDB #4

Closed Kingteena closed 1 month ago

Kingteena commented 1 month ago

Description

Implements the scraper functionality in scraper.py and stores the data as a TinyDB database.

Changes Made

Fixes bug in data_fetcher.py
Implements more functions in data_parser.py
Iterates through all API endpoints in scraper.py and stores it in a Tinydb database
Adds rudimentary rate limiter

Related Issues

Implements most functionality outlined in issue #2

rayokamoto commented 1 month ago

There are some issues regarding rate limiting but we can open a new issue regarding this and it can be fixed in another PR

Kingteena commented 1 month ago

I've added a commit adding a rudimentary rate limiter that can be fixed with another issue and pull request, and implemented most changes requested by @rayokamoto.

I did not implement any exception handling for when the API call fails, because I feel like the most effective thing to do is to retry the request after waiting a few seconds, which should be implemented when the rate limiter gets implemented.