search

UniversityOfHelsinkiCS / oodikone

An application for analyzing university data
https://oodikone.helsinki.fi
MIT License
14 stars 5 forks source link

readme

Oodikone

An application for analyzing university data, running at https://oodikone.helsinki.fi/.

✔️ Requirements

To run Oodikone locally, you will need the following:

  1. Applications
  2. Access to Toska Docker Hub
  3. For real data setup, access to Toska's production servers
  4. For some features to work, access to Toska's service tokens

🚀 Installation

Pre-installation steps:

Then launch the CLI and follow the instructions:

npm run cli

Please use a terminal at least 80 characters wide, the CLI is a bit rudimentary 😊

What different CLI options do:

  1. Set up Oodikone from scratch
    • Cleans up any previous installations
    • Installs all needed npm packages locally
    • Sets up pre-commit linting hooks
    • Pulls and builds all needed Docker images and sets up dockerized development environment
    • Setup happens with anonymized data, which developers should use by default
    • Note: Running this option cleans up all real data too, so please don' run option 2 before this.
  2. Reset all real data
    • Cleans up any previous real data databases
    • Downloads needed database dumps from production servers.
    • Creates real data databases and populates them with downloaded dumps.
  3. Reset a single database
    • Like option 2, but only for single database
    • Leaves other databases untouched
  4. Restore data from dumps
    • Like option 2, but skips downloading, and simply restores databases from the dumps downloaded earlier
  5. Docker system prune
    • Runs docker system prune -a && docker volume prune -a
    • This will remove all your Docker data, also related to any other applications than Oodikone!
    • Sometimes necessary after strange errors due to caching

⌨️ Development

Architecture

graph TB
    subgraph Oodikone & Updater
        subgraph Oodikone
            client[React app] --> server[NodeJS server]
            server --> ok_redis(Redis)
            konedb[(kone-db)]
            userdb[(user-db)]
        end
        server -->|Read| sisdb[(sis-db)]
        subgraph Updater
            updaterscheduler[Scheduler] --> |Push jobs| nats
            nats(NATS server)
            updaterworker --> |Fetch jobs| nats
            updaterworker[Worker] -->|Write| sisdb
            updaterscheduler --> redis(Redis)
            updaterworker --> redis
        end
    end

    subgraph Common Toska services
        jami[JAMI]
        pate[Pate]
        importer[Importer]
    end

    updaterworker -->|Read db| importer
    updaterscheduler -->|Read db| importer
    importer -->|Fetch data| sisu[Sisu export APIs]
    server -->|Get IAM access| jami
    server -->|Send mail| pate

    subgraph Analytics
        direction TB
        sentry[Sentry]
        graylog[Graylog]
        grafana[Grafana]
    end

In development, backend and updater use the same Redis, but in production they are separate.

See also the older graph for more information.

To read a bit more details about what updater-scheduler and updater-worker do, see their own READMEs:

Documentation

The documentation directory contains some notes on the sis-db schema and is meant as a place for storing useful information on Oodikone to make life easier for future developers. Feel free to add anything relevant!

For internal meeting notes and sensitive information, see Toska GitLab.

Basics

The development environment is entirely configured in the docker-compose.yml file located in this repository. The file defines the services for Oodikone's two main components: oodikone and updater.

Running Oodikone with real data requires separated databases and redis, which are defined in docker-compose.real.yml file. Otherwise real data development environment uses the configuration as anonymized data development environment.

Some useful commands are defined in package.json and can be run with npm run <command> as follows:

Once you have ran setup for Oodikone , you can just execute the first one (npm run oodikone). After starting and waiting for a while for containers to compile, Oodikone can be accessed at http://localhost:3000 and Adminer (database investigation tool) at http://localhost:5050. Adminer requires you to login with username postgres and with any password you choose (for example p).

Run.sh script

As said, the development environment runs entirely inside Docker containers. To keep package.json clean and not filled with predefined scripts, we have created a simple helper script called ./run.sh. The script allows you to use docker-compose commands without the need to write long list of parameters. Try to run ./run.sh in the root of the project and see what happens!

./run.sh is simply a wrapper script to run Oodikone, updater or both services in either anon or real mode. If you take a look at package.json, you can see that most of the predefined scripts above use ./run.sh under the hood.

It is recommended to spend some time to become familiar with docker and docker compose cli commands. You can then use them directly or with ./run.sh wrapper. Here is some examples for day-to-day development situations:

User types for development

By default, you're login as mluukkai dev user when running Oodikone in development mode. If you want to debug with certain type of user, you can use mocking: go to the "Users" page, click "edit" on the user you want to use and then click the icon on the right corner of user card. Another way is to mock user headers - see frontend's API configuration for how this is done.

Anonymous data contains some preset user types, corresponding to most usual user types in real data. These are used in testing too (see "Testing" below for more info).

Linting

Cli script sets up pre-commit hooks that are used to lint and fix files before committing. If your package.json file doesn't include "lint-staged" -key, please set hooks manually with npm run prepare.

For more information on how files are tested, take a look at "lint-staged" in package.json. Some files (e.g. github action files, dockerfiles, shell scripts) are checked with external tools and may require you to install those tools in case you're modifying files in question.

Pre-commit hooks will fix auto-fixable problems. To set up quick formatting: In VSCode you can go to command palette, select "ESLint: Fix all auto-fixable problems" and assign a hotkey for it, like SHIFT + CTRL + '.'.

🔨 Testing & CI

We use cypress for end-to-end testing. No unit tests are currently used.

Cypress

Continuous integration (CI) works with Github actions and is defined in workflow files in .github/workflows folder:

Rapodiff can be used to check differences between Rapo API and Oodikone data.

❓FAQ

Modules are missing after updating package.json

You should always install the dependencies inside the container to have the application inside the container access them. Module might be missing for example when someone else installs a new library and you only pull the changes in package.json. Use docker exec <service> npm ci to install modules inside the container or npm run install:docker to install modules inside all packages (backend, frontend, sis-updater-scheduler and sis-updater-worker).

Studyguidance groups don't work on my machine

Make sure you have your VPN on.

Everything is broken, can't get oodikone running, data is not there etc.

First: Try npm run cli option 1: Set up oodikone from scratch and then option 2 to download and restore databases (or option 4 to skip downloading, if you've done it recently).

If that does not help, try option 5: Docker system prune. Notice that this clears up all Docker data, including data related to other projects. If this is not ok, you'll need to identify the Oodikone-specific Docker volumes yourself and caches and clear those invididually.

✌🏼 Maintainers and contribution

Toska - University of Helsinki

🛡️ Tietosuoja / data protection

See tietosuoja.md.