AFG-Polio-Data / APMIS-Project

APMIS (Afghanistan Polio Management Information System) is an supplemental Immunization Activities management system to manage the vaccination activities in the country.
https://test.afghanistan-apmis.com
GNU General Public License v3.0
4 stars 8 forks source link

Import should be full success/fail; No partial import allowed #454

Open bnygren opened 1 year ago

bnygren commented 1 year ago

Currently users can import records and the system will allow some correctly formatted records to be added, while preventing others from being imported where they don't meet required criteria. This places a burden on the user to figure out the partial import and re-import the remaining records to avoid duplication, since unique IDs are not available for all data (e.g., sub districts in PCA)

Behavior needed: If any errors occur during the import, the entire import is canceled, the user is notified of errors to fix and then the user can re-attempt the import.

EDIT: @bnygren suggested differentiating the Dry Run UI to be more clear to the user that they are performing the dry run check prior to actual import

sundayAdu commented 1 year ago

This is not logically doable. when we import data to the system we import it line by line and the system flags the problems with each line and record the problematic line in a seperate file; which is downloadable after the end of the import process. When we fix issue 472, this issue should be resolved. But there is no way we will rollback the imported data on the system.

bnygren commented 1 year ago

Thanks Sunday. I feel strongly that partial imports are very problematic. Is this limitation inherited from SORMAS? If we were designing this system, we would not favor partial imports for large data sets which some are without UIDs at the record level. Can the data be imported to an interim table, then if fully successful, that table's contents are imported wholesale, otherwise the interim table is purged? Does the existing import feature need to be rebuilt? I also don't see the how #472 solves this except in cases where users are interrupted by the system logout.

bnygren commented 1 week ago

@Eutit5 Thanks to the team for work on this. We would like to revisit the issue details in the 1st week of October following the APMIS meeting

RaziaMahmodi commented 1 week ago

@DevSeg Please take out the Dry Run option from Live and Training server.