As noted in #62, I have identified a bug in the pipeline where is_downloaded in a record is not set the True, yet all the subsequent stages (pre-processing, prepare, and upload) are invokved. This is because the pre-processing stage does not check if a record is_downloaded and so it is to to True as are all subsequent stages.
64 adds measures in place to raise an error if such an error occurs, whilst this PR addresses the known bug.
Testing
Modify __group_by_composite_key in /db/main.py by adding an IF statement for the one known user who where this bug was discovered:
if _record.patient_id in ["USER_ID"]:
key = f"{_record.patient_id}/{_record.device_id}"
results[key].append(_record)
Run the DRM pipeline and note that the error no long exists.
Ideally do both steps on master and then on this branch to see the change in action. Note that when on master the patient's records (1/2 dreem files) will be uploaded to DMP whereas when on this branch none will be uploaded and all subsequent stages will be false.
As noted in #62, I have identified a bug in the pipeline where
is_downloadedin a record is not set the True, yet all the subsequent stages (pre-processing, prepare, and upload) are invokved. This is because the pre-processing stage does not check if a recordis_downloadedand so it is to to True as are all subsequent stages.64 adds measures in place to raise an error if such an error occurs, whilst this PR addresses the known bug.
Testing
__group_by_composite_keyin/db/main.pyby adding an IF statement for the one known user who where this bug was discovered:Ideally do both steps on master and then on this branch to see the change in action. Note that when on master the patient's records (1/2 dreem files) will be uploaded to DMP whereas when on this branch none will be uploaded and all subsequent stages will be false.