Benchmark performance for xAPI over Celery to Ralph

[bmtcril]

Tests for various Celery configurations.

Remaining tests:

• Multiple workers with batching
• 10k row single worker with 100 batch size

[bmtcril]

Test 1 (913dd5) - 1k rows, no batching

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: False

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags celery 1k local novector nobatch

Data captured for results:

• Length of run
  • Event generation: 0:00:38.091298
  • Monitoring / total run length: 0:07:02.613890
• Sleep time: 0
• Events: 1000
• Raw stats attached: 913dd5_stats.txt

Findings:

• Celery could not keep up, ClickHouse was 376 seconds behind on a 416 second run at the end
• Celery queue grew to 1,340 pending events before the generation stopped and then took 380 seconds to catch up
• Inserted rows per second: 2.2 - 2.6

[bmtcril]

Test 2 (972dc4) - 1k rows, batching 10

Test system configuration:

• Tutor version: tutor, version 17.0.2-nightly
• Aspects version: 0.91.0
• Environment specifications:
  • Tutor local / MacOS / OrbStack
  • 10GiB RAM
  • 5 CPU limit
  • Storage unlimited
  • Tutor plugins: aspects, mfe
• Relevant settings
  • RUN_CLICKHOUSE: true
  • RUN_KAFKA_SERVER: false
  • RUN_RALPH: true
  • RUN_SUPERSET: true
  • RUN_VECTOR: false
  • EVENT_ROUTING_BACKEND_BATCHING_ENABLED: True
  • EVENT_ROUTING_BACKEND_BATCH_SIZE: 10

Load generation specifications:

• Tool - platform-plugin-aspects load_test_tracking_events management command
• Exact scripts
  • tutor local run lms ./manage.py lms monitor_load_test_tracking --sleep_time 5 --backend celery
  • tutor local run cms ./manage.py cms load_test_tracking_events --num_events 1000 --sleep_time 0 --tags celery 1k local novector batch10

Data captured for results:

• Length of run
  • Event generation: 0:00:57.751662
  • Monitoring / total run length: 0:01:09.289254
• Sleep time: 0
• Events: 1000
• No stats captured

Findings:

• Celery could barely keep up, but was only 12 seconds behind by the end of the run
• Celery queue remained even and did not grow over this test
• Inserted rows per second: 23 - 25