A refactor of the Scan Report upload / process in app/workers, that enables larger Scan Reports being uploaded. This is the first step as part of #612 , and unlocks work on #617
The UX change is:
User uploads their Scan Report.
User sets the Person ID / Date Event for each table, the Concepts are now created and reuse of existing concepts.
User clicks "Refresh Rules" to generate the mapping rules for the Scan Report.
This behaviour is a significant change, and so the now "legacy" behaviour has been preserved, and the two can be switched between by UPLOAD_ONLY environment variable. This can be removed, and the ProcessQueue function, when we are satisfied the new behaviour is suitably robust.
Includes:
A new UploadQueue Azure function, responsible for uploading a Scan Report.
A new CreateConcepts Azure function, responsible for creating Scan Report Concepts, AND reusing existing ones.
Whilst the existing logic is retained in these functions, they have been refactored from the original ProcessQueue, adding docstrings, types, move to some modularity, and general clean up.
Move any API requests to /shared_code/api.py to centralise it.
Add tests (Pytest) for app/workers (a test CI workflow will follow as part of #639 , including #424 )
Added AZ_FUNCTION_USER to the sample-env.txt
Updated ScanReportTableFilterViewSet to return a 403 if the user is not the Azure function user.
A corresponding PR will be attached for the documentation updates.
Environment changes
This does necessitate changes to any deployed instances of CaRROT Mapper, including:
Add Azure queues:
[x] Upload queue: upload
[x] dev
[x] test
[x] prod
[x] Create concepts queue: concepts
[x] dev
[x] test
[x] prod
Add environment variables to the Web app(s):
[x] UPLOAD_QUEUE_NAME: Name of the Azure upload queue: upload
[x] CREATE_CONCEPTS_QUEUE_NAME: Name of the Azure queue: concepts
[x] UPLOAD_ONLY: true (dev only initially)
Checks
Important: please complete these before merging.
[x] Run migrations, if any.
[x] Update changelog.md, including migration instructions if any.
Changes
A refactor of the Scan Report upload / process in
app/workers, that enables larger Scan Reports being uploaded. This is the first step as part of #612 , and unlocks work on #617The UX change is:
This behaviour is a significant change, and so the now "legacy" behaviour has been preserved, and the two can be switched between by
UPLOAD_ONLYenvironment variable. This can be removed, and theProcessQueuefunction, when we are satisfied the new behaviour is suitably robust.Includes:
UploadQueueAzure function, responsible for uploading a Scan Report.CreateConceptsAzure function, responsible for creating Scan Report Concepts, AND reusing existing ones.ProcessQueue, adding docstrings, types, move to some modularity, and general clean up./shared_code/api.pyto centralise it.app/workers(a test CI workflow will follow as part of #639 , including #424 )AZ_FUNCTION_USERto thesample-env.txtScanReportTableFilterViewSetto return a 403 if the user is not the Azure function user.A corresponding PR will be attached for the documentation updates.
Environment changes
This does necessitate changes to any deployed instances of CaRROT Mapper, including:
Add Azure queues:
uploadconceptsAdd environment variables to the Web app(s):
UPLOAD_QUEUE_NAME: Name of the Azure upload queue:uploadCREATE_CONCEPTS_QUEUE_NAME: Name of the Azure queue:conceptsUPLOAD_ONLY:true(dev only initially)Checks
Important: please complete these before merging.
changelog.md, including migration instructions if any.