NCEA Standards Navigator nsn.nz

This project is aimed at being a simpler navigation for the NZQA assessment resources (past papers, assessment schedules, exemplars, etc). NZQA have been kind enough to provide a dataset for their assessment standards, which will be used as reference and updated in future years (hopefully). As well as this, they have an excel spreadsheet for their literacy/numeracy credits, disgusting. Oh no, not just that, they have an excelx spreadsheet for their UE reading/writing/literacy credits, absolutely horrifying.

Description of contents:

• docker-compose.yml is the file that describes how all the Docker containers work together (port mappings, shared storage etc.)
• Caddyfile(.dev) is the config file for Caddy
• Makefile contains some quick scripts for development (so I can type make rather than docker-compose up -d basically)
• set_flag.sh allows me to quickly and easily add flags to the database so that the next time the scraping script checks, it will refresh or rescrape.

Running/install

docker-compose up -d

Should work. Tested on Debian Buster/Bullseye (apt has the necessary packages docker.io and docker-compose). Make sure you're in the group docker post-install.

More info/Kanban Board

Notion

Intentions

• Minimizing the number of clicks to get to a given assessment.
• I will not store every PDF, they will be links to the NZQA website (except for caching)
• Papers are listed alongside assessment schedules, exemplars, rather than being shown just a list of documents.
• Browser-storage list of user-specific subjects/standards for quick access/

Structure

• Python3.9 with Beautiful Soup and Pandas: for scraping, parsing, and managing the database of resources, standards, and subjects.
• MeiliSearch: for serving fast, typo-resistant search results for both subjects and standards.
• PostgreSQL: for providing a scalable, enterprise-level database.
• Python with Flask and Waitress: for serving the data from the database to the frontend with a RESTful API.
• Caddy: as a reverse proxy and a service retrieving auto-SSL/TLS certificates from LetsEncrypt.
• GoAccess: for reading Caddy logs and hosting a websocket, where a dashboard can be connected.
• HTML/CSS/JS + Bootstrap: for creating a responsive, dynamic frontend.