Add a "last updated at" indicator somewhere on the website for course information

[gannonprudhomme]

We should add some an indicator somewhere on the site (maybe on the navbar?) that says when the course information was last updated.

An easy way of doing this would probably be to just have a table with a single row / column that says when scrape_courses was last ran. The route would then just retrieve this value and the frontend would display it accordingly

Add the model to the backend

So I'm not really sure how we want to handle this, but my thought process would be that the easiest way to do it would be to have a model that just stores a term and date, and it's ID / primary key is the term. So it'll basically just show when the data was last updated for each term.

To do this, you'll need to:

1) Create the model

• You'll want to create the file in autoscheduler/scraper/models/ since this is related to scraping.
• Will probably want to call this LastUpdated or something like that.
• You'll want to have two fields, a term and a date, where the term is a string that is also it's ID / primary key.
• You should be able to get a gist of how to make the model by looking at the other models in that folder, but lmk if you have any questions. 2) Run python manage.py makemigrations to create a migration file.
  • This is how we record changes to the database (new tables, changing columns, etc) so after this change is merged we can apply the same changes to our database.
  • Make sure you commit the file! It'll be located in autoscheduler/scraper/migrations/ 3) Run python manage.py migrate to apply the migration to your database

Save/update these models in scrape_courses

Now that the model is created, you'll need to actually save them whenever scrape_courses is ran.

• The file is located in autoscheduler/scraper/management/commands/scrape_courses.py
• You'll simply need to scrape over all of the terms that are being scraped, then create a LastUpdated model(s) that have the term as the respective term and the current datetime as the date model.
  • Note that there can be a singular term stored in term, or multiple terms in terms, so make sure you handle both situations
  • This is b/c you can scrape a singular term with scrape_courses -t 202031, but can also scrape a group of terms, such as scrape_courses -y 2020 to scrape 2020, scrape_courses -r to do the most recent semester, or even just scrape_courses without args, which scrapes all terms possible.
  • You'll want to do something called an "upsert" - that is, insert if it doesn't exist, or update the model if it is.
    • The easiest way to do this is do just call LastUpdated.objects.delete() then LastUpdated.objects.bulk_create(objects). Sounds counter-intuitive, but it's what we're doing elsewhere and it seems to work for us (Django's bulk_update is pretty slow from what we've seen)
  • You'll want to do this around line 331, after we call save_models

Add a route that returns the respective model on the backend

We'll probably want to set this on a per-term basis? Or maybe a per-semester basis (since all locations are scraped each semester)

You should create a route like api/get_last_updated which takes a term query parameter. A potential query would look something like api/get_lasted_updated?term=202031

1) Create the API view in views.py

• Since this is a scraper route, you'll do this in autoscheduler/scraper/views.py. Note that scraper, scheduler, and user_sessions all have a views.py file
• You'll create a RetrieveLastUpdated view at the bottom of the file. It will take be a get request, so as such you can base this off of the RetrieveCourseView at the top of the file.
• All you need to do is get the term 2) Add the URL to urls.py
  • Again since this is a scraper route, you'll do this in autoscheduler/scraper/urls.py
  • You'll add a line to the bottom of the urlpatterns list that'll look something like path('get_last_updated, RetrieveLastUpdated.as_view())`
  • Note that the api/ isn't in here - that's b/c all of the routes in scraper automatically have the api/ appended to it. You can see this if you go to autoscheduler/autoscheduler/urls.py.

Add the component on the frontend + make it fetch the data

You'll then want to add a component on the frontend (and make it display in the navbar) that fetches the last updated data and displays it

We'll also probably want to have some mechanism for updating this, since in peak times we hope to scrape the data every 15 minutes. Maybe we can try to update this value every minute? Whatever you think is best really

Add frontend tests

You'll need to create some basic tests on the frontend. Since these are UI tests, you'll want to put them in autoscheduler/frontend/src/tests/ui/LastUpdated.test.tsx or something like that. When you get around to this message in Discord and we'll find a good example test.

Add backend tests

The tests should be relatively straightforward. There are a lot of examples in autoscheduler/scraper/tests/api_tests.py (which is also in the folder where you'll be making these tests)

• Since api_tests.py is already huge, make a separate test file called like api_last_updated_tests.py or something like that.
• Note to run these tests, you can run pytest in the autoscheduler/ folder (where the pytest.ini is located)

[gannonprudhomme]

Updated this with information on how to do it - let me know if you have any questions!

If you think it's too much work for one issue, we can always split it up into a backend + frontend task (in case you want to do one side but not the other). Just looking at the text makes it seem like it's a lot of work, but it should actually be a lot more straightforward than it looks.