Refactor bulk upload

[iperdomo]

The current bulk upload relies on the applet code. The process is straight forward and can be re-implemented in Clojure.

• After combining all the chunks sent by the client,
• Search for all zip files contained in the original surveyal.zip
• Unzip each file and loop through the content of data.txt, we need to skip processing the zip files names wfpGenerated*.zip
• For each survey instance (identified by uuid) in data.txt, check if that survey instance exists in the datastore by making a Remote API call (NOTE: this is a new step, not present in the old process)
• If the survey instance doesn't exist in the datastore, make a zip with just one survey instance, upload it to S3 and notify the backend for ingestion.
• We need to keep track of the uploaded survey instance for this particular bulk-upload execution, to avoid reprocessing a survey instance. (We can't rely on the Remote API lookup)

[iperdomo]

As @ichinaski suggests in https://github.com/akvo/akvo-flow/issues/652 if we find an image, we just need to upload it, no re-sizing is needed.

[iperdomo]

Some idea for detecting duplicate lines

(defn get-data [f]
  (let [zf (ZipFile. f)
        entry (.getEntry zf "data.txt")]
    (if entry
      (line-seq (io/reader (.getInputStream zf entry)))
      "")))

(reduce conj #{} (mapcat get-data (fs/find-files "/path/to/folder" #".*\.zip$")))

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.