E2E testing framework

[jahpola]

1. How to test?
2. How to integrate testing into CI/CD?

[jahpola]

An external service that sends and reads.

[haphut]

• mosquitto mqtt broker docker to replace cloudamqp

• python http server to serve static test files, one for each gtfs realtime city, one for vehicle registry

  • python3 -m http.server -d served 8000 serves directory served.

  • This switches between the numerically ordered files in directory:

    #!/bin/bash
    
    set -Eeuo pipefail
    
    for filename in source/*; do
    echo 1>&2 "Set target to ${filename}"
    ln -f -s "${filename}" 'served/target'
    sleep 3
    done

  • curl 'http://localhost:8000/target' should ask for file target but has some bug
• Also need to send and receive MQTT messages, e.g. with mosquitto_sub and mosquitto_pub or with npx mqtt sub and npx mqtt pub.
• Sending messages in order is important. Maybe use carefully selected sleeps for that. That would take a long time to finish. Maybe one script should orchestrate all message sending and http server offering changes.
• Need to create test data. Anonymize real data by randomizing counts, maybe with jq?

[haphut]

Alternative:

• one typescript service
• sending loop
  • http server and mqtt client (producer)
  • static json files in data directory
  • mqtt client spews the apc json files into topics
  • http server reads json and turns it into GTFS Realtime VP protobuf data
  • pulsar client pushes vehicle profiles to the right topic (or http server also serves vehicle registry output but the profile computation would take a long time)
• receiving loop
  • mqtt client (subscriber)
  • expected json results in data directory
  • compares received data with data directory contents
• test running and expectations with e.g. jest or another test framework

• coordination file with contents similar to this:

  - scenario: bus-leaves-depot-or-something
   inputMessages:
     - path: input/http/vehicle-registry/all-vehicles.json
     - path: input/http/jyvaskyla/no-service.json
     - path: input/http/kuopio/en-route.json
     - path: input/mqtt/acme-vendor/door-1-in-2.json
     - path: input/mqtt/counting-corporation/door-2-out-1.json
     - path: input/mqtt/first-stop.json
       waitAfterInSeconds: 0.5 # Default as env var, value e.g. 0.1
     - path: input/http/kuopio/left-stop.json
     - path: input/http/jyvaskyla/bus-on-first-stop.json
   outputMessages:
     - path: output/mqtt/apc/authority-221/door-1-in-2.json
     - alternative_paths:
       - output/mqtt/anonymized-apc/authority-221/trip-y-empty.json
       - output/mqtt/anonymized-apc/authority-221/trip-y-many-seats-available.json

• how to do several scenarios?
  • no clean-up in between, just run message lists in order with a wait in between
  • easy to write but tests affect each other
  • in between each scenario: stop all k8s services, empty all pulsar topics, start k8s services again
  • how hard is this?
  • tests independent
• should alternative k8s manifests be in the e2e repo? can we just partly override dev manifests, e.g. URLs?
• note: naming files like 00001-... has the problem that most changes require renaming many files
• for linting, we could add json schemas and schema checking in the e2e repo

[jahpola]

1. Stopping Kubernetes service is rather easy just scale down to 0.
2. Emptying pulsar topics is the harder part, would most likely need pulsarctl/pulsaradmin cli tools to wipe them clean.
3. Overriding dev -> another kustomize overlay, might be easier to create "testing" namespace and deploy them all into into (feature branch testing)

[jahpola]

I see two choices:

1. Keep everything as is then monitor the application(s) via GCP monitoring tools
2. Create our own "framework" for testing the thing. Could be also "faked" via testcontainer mqtt+pulsar containers. Problem.... the majority of testcontainer mqtt+pulsar is written in Java.