search

akvo / akvo-flow-services

Akvo Flow service applications for reporting, bulk uploads and others
Other
4 stars 1 forks source link

Bulk upload code doesn't handle correctly the sharded structure of surveyal folder #146

Closed janagombitova closed 4 years ago

janagombitova commented 8 years ago

Original issue

https://github.com/akvo/akvo-flow/issues/430

[iperdomo commented on Nov 7, 2013]

The bulk-upload process tries to process all the zip files in a folder (and subfolders) searching for duplicates and generating only one wfpGeneratedxxxx.zip.

The current process can't handle the structure of the typical surveyal folder, and generates one wfpGeneratedxxx.zip per zip found.

Possible solutions:

Keep the current behavior, but instead of generating a wfpGenerated, just upload the zip file, and leave the duplication detection in the backend Fix the pre-processing part, searching recursively across all subfolders

[mtwestra commented on Nov 7, 2013]

A typical surveyal folder will contain many duplications, possibly dozens for every surveyInstance. I think the dedupe should therefore happen in the upload process, to avoid uncessary stressing the backend. The main reason I guess is that the backend might spawn different processes to handle a large number of zip files, recreating possible problems

[iperdomo commented on Nov 7, 2013]

mtwestra OK, agreed. The 2nd option is the designed and intended one.

[ichinaski commented on Nov 7, 2013]

Regarding the duplications, we should consider avoiding unnecessary exports as specified in akvo/akvo-flow-mobile#39

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.