QUANTUM JOBS EXECUTION

[GeorgDaniel]

FEATURES

• [x] #23
• [x] #22
• [x] SeQuenC-Consortium/qunicorn-core#8
• [x] SeQuenC-Consortium/qunicorn-core#13

Overall Requirements

• Developers must be able to execute different types of quantum jobs:

  • Quantum Circuits embedded as the JSON payload in HTTP Requests
  • Quantum Circuits sent as files
  • Quantum application with classical dependencies (containerized or packaged, deployed somewhere)
  • Qunicorn's API must support executing different jobs

• Qunicorn must have a Job Manager component to register and monitor jobs completion

• Qunicorn's Job Manager must register typed job requests and persist them in a store

  • Evaluate SQL/NoSQL options for storing jobs submitted to qunicorn, e.g., Redis vs. Mongo vs. MySQL etc.

• Qunicorn must have a Job Scheduler component that maintains a job queue and distributes jobs to execution components

  • Possible: Python Celery as a job queue, other options possible too - Evaluate
  • Job Scheduler could support different scheduling algorithms and enable switching between them
  • Stale jobs should be monitored and reported by Qunicorn
  • Unsuccessful jobs may be resubmitted or canceled - possibly decide via configuration in Job

• Qunicorn must support pluggable execution components (aka Qunicorn Pilot Managers)

  • see Master-Worker and Pilot-Job patterns for general idea

• Different Pilot Managers can be implemented for different Job Types, for example:

  • Pilot Manager A: Supports running IBMQ jobs, i.e., quantum circuits embedded in HTTP requests
  • Pilot Manager B: Supports running AWS Braket jobs, i.e., quantum circuits embedded in HTTP requests
  • Pilot Manager C: Supports execution of containerized quantum programs, e.g., Python file + dependencies
  • Can be implemented as a FaaS platform that "deploys" a referenced container and exposes it via endpoint

• Qunicorn must support registering local and remote Pilot Managers

  • For example, users might want to run an existing Piliot Manager locally and register it within Qunicorn such that jobs could be registered by Qunicorn and then submitted for execution on their local infrastructure

• Pilot Managers need to maintain their internal job queues to keep track of existing quantum jobs and their status, and convey this information periodically to the Job Manager component

  • see Section 4 in the funcx paper for an idea of hierarchical queues

[obstjn]

This issue need finer granularity to be workable. As a next step, smaller issues should be created that are referenced here.