[FHIR Road Map]: Add Bulk upload interface to web.

peterMuriuki commented 7 months ago

Issue Context?

Support an interface that users can use to initialize bulk upload of resources to the web. This can be a wizard that:

Allows users to download a template for upload of the necessary resources.
Upload the csv template
Track upload progress(what resource uploads are pending), success and error rates(if upload was succesful, pointers on what errored out, completion percentage)

Issue Implemetation details?

R&D on building upon existing python tool that enables the upload.

Issue Acceptance criteria?

User can via fhir-web:

[ ] Access the bulk upload module
[ ] Download sample templates that they will use to populate the data.
[ ] Upload the csv templates to upload the data.
[ ] See a progress report of the upload process,
[ ] See errors and warning messages with enough context on how to fix the same.
[ ] Or see a summary of resources uploaded after a successful operation

FHIR resources to be used?

No response

Relevant Information

No response

peterMuriuki commented 7 months ago

graph TD

subgraph Dependencies
Binary
Group -->|depends on| Binary
UserRoles
UserGroups -->|depends on| UserRoles
Users -->|depends on| UserGroups
Locations
Organizations -->|depends on| Users
CareTeams -->|depends on| Users
CareTeams -->|depends on| Organizations
InventoryGroup -->|depends on| Locations
InventoryGroup -->|depends on| Group
UserCareTeamAssignment -->|depends on| CareTeams
UserCareTeamAssignment -->|depends on| Organizations
OrganizationLocationAssignment -->|depends on| Organizations
OrganizationLocationAssignment -->|depends on| Locations
end

AnnieMungai commented 2 months ago

@peterMuriuki QA feedback on bulk upload:

[ ] Uploading all CSVs does not work. Could be the formatting. Updated on June 27th: This could be due to the max size of all the files (more than 8MB), or because individual files were failing.
[ ] There is no button to upload service points.
[ ] Uploads are taking awhile - there is no progress bar to show the user that there is a process going on. User may upload 2 similar files.
[ ] Error/ success message is the same for all uploads " Commodity successfully uploaded". This is misleading as it says its a success, then fails. We need to be more clear on the errors.
[ ] Organizations showed successful upload but I could not find any additional orgs on the orgs list.
[ ] Locations upload - failed even though I seemed to have done everything correctly on the template (except missing geo-coordinatesand these should not be required.). See error below. I also tried breaking the locations to less 1000 and they still failed. Do we have a max number of items one can upload at a time?
[x] Users - successfully uploaded. Can upload and view them on the users list.
[ ] Careteams - successully uploaded but cannot see the careteams on the UI. Same as the organizations.
[ ] Careteam - users/practitioners - successully uploaded but cannot see the careteams on the UI.
[ ] Products upload- failed.
[ ] Inventory - failed- probably coz individual products and SPs did not load successful?
[ ] Org-locations- Failed.

peterMuriuki commented 2 months ago

CareTeams csv

[template] status field has value as single letter t

Organization field

[template] status field has incorrect value 'TRUE' should be 'true'

Products

[template] id is not a uuid
[importer] Material number uses id field. either missing id field or material Number on the template.

Inventory

[importer] bug when adding adding the usualId, po_number identifiers.Looks like it requires po_number filed to not be None which is the case in the template

Locations

Q: Do we want to still migrate inactive service points?
adminLevel 0 should be root fhir location
madagascar should point to root fhir location id.
row 589 Anjeke Ankilikira has no status seems to be duplicate
service points have admin levels starting with 0
[importer] location physcial type is not deduped to type field
[importer] location parents are not added if parent name is not provided

service points
- some locations missing location type

Careteam to organization

importer expects this to be in the careteams csv file

participant to CareTeam

importer expects this to be in the careteams csv file

onaio / fhir-web