fractal-analytics-platform / fractal-server

Fractal backend
https://fractal-analytics-platform.github.io/fractal-server/
BSD 3-Clause "New" or "Revised" License
11 stars 3 forks source link

Introduce `TaskGroup`s #1817

Closed ychiucco closed 1 month ago

ychiucco commented 1 month ago

ref #1803

Checklist before merging

github-actions[bot] commented 1 month ago

Benchmark comparison

GET /api/alive/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 0.8 0.9 0.89 0.0 0.0
power 0.8 0.8 1.00 0.0 0.0
dataset 0.8 0.8 1.00 0.0 0.0
project 0.8 0.8 1.00 0.0 0.0
job 0.8 0.8 1.00 0.0 0.0

GET /api/v2/dataset/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 10.2 10.3 0.99 0.6 0.6
power 395.6 390.9 1.01 389.0 389.0
dataset 149.1 146.0 1.02 145.6 145.6
project 46.5 45.6 1.02 50.9 50.9
job 27.4 27.1 1.01 35.3 35.3

GET /api/v2/job/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 6.6 6.8 0.97 2.2 2.2
power 139.1 135.2 1.03 652.0 652.0
dataset 141.3 135.5 1.04 904.8 904.8
project 48.9 44.4 1.10 283.7 283.7
job 37.4 38.7 0.97 224.8 224.8

GET /api/v2/project/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 7.9 7.9 1.00 0.1 0.1
power 8.0 7.8 1.03 0.1 0.1
dataset 7.8 7.8 1.00 0.1 0.1
project 9.9 9.6 1.03 2.3 2.3
job 8.0 7.7 1.04 0.1 0.1

GET /api/v2/task/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 6.4 5.7 1.12 0.7 0.7
power 6.9 5.6 1.23 0.7 0.7
dataset 7.0 5.9 1.19 0.7 0.7
project 6.7 5.8 1.16 0.7 0.7
job 6.9 5.8 1.19 0.7 0.7

GET /api/v2/workflow/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 12.7 11.6 1.09 0.8 0.8
power 26.5 24.3 1.09 27.3 27.2
dataset 23.8 19.0 1.25 15.9 15.8
project 23.6 22.1 1.07 19.9 19.9
job 12.3 11.5 1.07 0.8 0.8

POST /api/v2/project/3/dataset/403/images/query/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
dataset 152.4 148.1 1.03 275.1 275.1

GET /auth/current-user/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
vanilla 5.2 5.1 1.02 0.2 0.2
power 5.1 5.0 1.02 0.2 0.2
dataset 5.1 5.0 1.02 0.2 0.2
project 5.0 5.0 1.00 0.2 0.2
job 4.9 5.0 0.98 0.2 0.2

POST /auth/token/login/

User Time current (ms) Time main (ms) Ratio current/main Size current (Kb) Size main (Kb)
dataset 264.4 263.0 1.01 0.2 0.2
tcompa commented 1 month ago

The core changes to database models are now in.

The next step is to make sure that all TaskV2 objects in the CI conform to the new models:

  1. The task_factory_v2 should create a dummy TaskGroupV2 for each task that it creates. Note that we need to add an argument user_id to task_factory_v2, to be used when creating the task group.
  2. The single-task POST endpoint should create a TaskGroupV2 entry and establish the association with the new TaskV2
    • user_id is the one of the user making the API request
    • user_group_id is the one of the All group - for the moment
    • user_group_id is an request-body parameter - TBD
  3. The same applies to the two (local/ssh) task-collection endpoints
  4. The same applies to the custom task-collection endpoint
github-actions[bot] commented 1 month ago

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  fractal_server
  __init__.py
  fractal_server/app/models/v2
  __init__.py
  task.py
  fractal_server/app/routes/api/v1
  task.py
  fractal_server/app/routes/api/v2
  __init__.py
  _aux_functions.py
  submit.py
  task.py
  task_collection.py
  task_collection_custom.py
  workflow.py
  workflowtask.py
  fractal_server/app/routes/auth
  _aux_auth.py
  fractal_server/app/runner/executors/slurm/sudo
  executor.py
  fractal_server/app/schemas/v2
  __init__.py
  workflow.py
  workflowtask.py
  fractal_server/app/security
  __init__.py
  fractal_server/tasks/v2
  background_operations.py
  background_operations_ssh.py
Project Total  

This report was generated by python-coverage-comment-action