Improvements in data ingestion process

akvo / akvo-flow

A data collection and monitoring tool that works anywhere.

http://akvo.org/products/akvoflow/

GNU Affero General Public License v3.0

65 stars 31 forks source link

Improvements in data ingestion process #790

Closed muloem closed 4 years ago

muloem commented 10 years ago

A lot of support issues are reported that are linked to failures in the file processing when we are ingesting files with survey responses, .e.g. the file contains IDs that are not present in the system, the process fails somewhere along the way and data is duplicated, and empty files. We need to refactor the data ingestion process to limit as much as possible erroneous data from entering the system, while still allowing correct data in the same files to be processed without failure.

muloem commented 10 years ago

Some issues with the processing:

Data read from the file is stored in multiple locations and causes duplication of efforts / inaccuracies in data in later processing. e.g. the surveyedLocale identifiers stored here --> https://github.com/akvo/akvo-flow/blob/v1.7.4/GAE/src/org/waterforpeople/mapping/dao/SurveyInstanceDAO.java#L168
Sometimes GAE a device was not able to complete the file upload and notifies GAE anyway that there is one to process.

muloem commented 9 years ago

We should consider using some kind of checksum for the data sent by the devices, to verify that the response files were correctly processed.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.