New Architecture for Auto Grading Job services

[jgwerner]

Overview

IllumiDesk needs to re-design the auto-grading services to improve scalability and decouple the service from the rest of the LMS system.

Desires and Goals

• The auto-grading service can be called from any client using standard APIs (RESTful, etc)
• Like the rest of the system, requests should be authenticated
• Leverage native Kubernetes services to the extent possible to manage jobs
• Maintain compatibility (at least for the first iteration) with the nbgrader database schema and nbgrader ipynb schema
• Allow system administrators to adjust CPU / Mem settings for auto-grading containers
• Enable notifications based on job status (completed, failed, in-progress, etc)
• Log auto-grading jobs to a central location via stdout/stderr (thus allowing administrators to integrate the logging tool(s) of their choice.

Notes

• The nbgrader auto-grading feature is tightly coupled with the Jupyter Notebook. The async-nbgrader project helps but need to take this a step further and siphon off the auto-grading service into a more traditional queue/task-based job architecture.
• The queue/job architecture setup naturally has many options. We are more inclined to stick with a Kubernetes-native approach. This of course has pros/cons. If we use something with native Kubernetes (Argo, K8s jobs, etc) then it's more complex to setup and maintain, however it would allow us to provision this with any cloud vendor if we chose to do so and/or offer this with the IllumiDesk Enterprise version. Perhaps when managing this with our SaaS solution we could combine K8s with a managed service, such as Fargate via plugins/configs/middleware but that is wait-and-see.
• The queue/job architecture should be generic enough so that we can receive notifications based on the status of the desired state and alert the end-user and/or internal IllumiDesk collaborators based on certain rules.
• We could use the gofer_service and gofer_submit projects as sources of inspiration.

Nice to haves

• Once the auto-grading has been completed, allow users to configure whether or not they need to auto-submit grades to their LMS either with LTI 1.3 or third-party integration service (Zapier, generic Webhooks, etc).

Diagrams and Architecture

This diagram has a basic first draft of how the re-architecture could look.

Resources

• For the background workers which can be configured by the end-user, we can use Horizontal Pod Autoscaler dependent on external metrics: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects (this will need change in our helm-chart)
• Background worker with kubernetes - https://kubernetes.io/docs/tasks/job/coarse-parallel-processing-work-queue/

Complications identified

• Worker containers would need shared volume access of the jupyter, in order to read the notebook. This is on basis of the current solution where we are using file storage.
• Acquiring locks on basis of Notebook so that multiple workers don't write to the same notebook

[rupeshparab]

One of the solution for having Auto-Grading with queue/task-based job approach is to integrate otter-grader, there are 2 approaches to do this:

1. Replacing nbgrader with ottergrader

Pros

• Will make it easier to maintain and take updates from otter-grader upstream
• Better error outputs when compared to nbgrader

Cons

• Will need to write an adapter for jupyterhub, for example when ottergrader relases an assigment, it will just generate a zip. Will need to replicate certain nbgrader processes like copying the files at the exchange location from where students can access it
• Integrate usage of existing nbgrader tables from within the adapter (might need to check if it is possible to do it without making changes to core of ottergrader)

2. Update autograding of nbgrader to use ottergrader module

Pros:

• Will help in reusing most of the nbgrader apis and tables, and exchange process integration with Jupyterhub

Cons

• Low maintainability of otter-grader, if any upstream changes are pushed

[jgwerner]

@rupeshparab from what I understand you are implementing this solution with Argo. Do you have any requirements definitions yet? For example would it use primarily Argo Workflow, ArgoCD, etc.

With this information we can start preparing our helm chart to support these additional resources.

cc/ @Abhi94N

[rupeshparab]

We will be argo workflows for configuring event handlers, initially I was planning on using AMQP for event source

although, its configuration doesn't seem very straight forward, trying to check if configuration with Kafka is easier.

[jgwerner]

@rupeshparab to confirm, are you implementing this with Argo Events and then using something like AMQP as the event source? If so perhaps our first version can use something simpler such as using webhooks and then provide something more performant in future iterations.