Bulk document uploader

[lippytak]

In addition to filling out the basic application form, users have to upload personal documents to go along with their application. This usually includes about 4-10 scanned documents (PDFs) or photos taken from a phone/tablet. At first we tried a one view per document approach and a circular flow ("do you want to upload anything else?") but a bulk uploader would be a lot better.

Let's try making one:

• Easy to add multiple files from one screen
• Auto upload (doesn't require a click per doc)
• Thumbnail preview
• Can remove files
• Setup docs: https://github.com/blueimp/jQuery-File-Upload/wiki/Setup
• Needs to work on
  • IE 8+ desktop
  • iOS
  • Android
  • Windows RT (we can test this)
• Bonus
  • Uploads to S3
  • Client side image resizing to get under ~3mb per file
  • Open the camera for multiple photos (vs. only file picker)

Resources:

• Let's use the jQuery-File-Upload
• Setup docs: https://github.com/blueimp/jQuery-File-Upload/wiki/Setup

Moved final launch checklist to: https://github.com/codeforamerica/clean/issues/504

[daguar]

If anyone is interested in taking a stab at this, please ping me here first — this might have some complexity in terms of integrating with the larger app, so want to chat about it first.

[lippytak]

We should try out filepicker.io. Just tried it out on Instacart:

Note the 'take a picture' option even on Desktop...huge perk!

S3 instructions: https://developers.filepicker.io/page/s3/

[lippytak]

Filepicker.io support says:

• If you send to S3 files are never stored on their servers
• Supports Windows RT browser

[lippytak]

Moved to current sprint. At the very least let's spec out this feature together this week.

[alanjosephwilliams]

Is a $49 - $99 fixed monthly cost in any way a blocker to this approach?

On Mon, Mar 2, 2015 at 1:47 PM, Jake Solomon notifications@github.com wrote:

Moved to current sprint. At the very least let's spec out this feature together this week.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/365#issuecomment-76832960 .

t: @alanjosephwilli p: 817 713 6264

[daguar]

Heroku has 4 options for image processing (one is Filepicker):

From https://addons.heroku.com/

[lippytak]

Price isn't a blocker

On Tuesday, March 3, 2015, Dave Guarino notifications@github.com wrote:

Heroku has 4 options for image processing (one is Filepicker):

[image: screen shot 2015-03-03 at 1 35 55 pm] https://cloud.githubusercontent.com/assets/994938/6472891/66c9c88c-c1aa-11e4-9e0b-0f4d59f3911a.png

From https://addons.heroku.com/

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/365#issuecomment-77040751 .

[daguar]

I'm currently assessing the world of options across UX and technical needs.

[daguar]

So, here's my proposal for what I think we want to use for each of the components:

• For AJAXy upload of multiple documents on same page
  • jquery-file-upload (Javascript library)
  • AJAX POST to Rails route, eg POST /document/new, #{ case_id: '1', file_blob: FileObject }
• Store individual documents EITHER in Postgres or Amazon S3
  • Simply a Rails model (a case has many documents), with either a URL to S3 or a binary column
  • May use a gem for this (or may do it more explicitly):
  • Paperclip (uses S3 for storage)
  • CarrierWave (can use Postgres — I like this gem less, though)

(In UX, User finalizes application and clicks submit here)

• In background job, process (image -> PDF) and combine all files (cover letter, application, info release, documents) into a single PDF
  • Use imagemagick in our application for this (maybe the Ruby wrapper, RMagick)
  • Store on S3
• URL for full application + documents PDF is secured with basic auth, but the file is hosted on a (secure) S3 bucket on the other end; we provide the URL in the email to the agency

[Mr0grog]

Might recommend shoving the images into redis over PG. You could store the images for each app in a list. But if you want to keep them around, S3 is probably best.

Will try and check out jquery-file-uploader tomorrow if I have time.

[lippytak]

https://support.box.com/hc/en-us/articles/200519718-How-do-I-upload-files-to-a-Box-folder-from-another-website-with-the-Upload-Widget-

Better late than never?

[daguar]

Epic todo checklist

(work being done on the doc-upload-redux branch):

• [x] Basic front-end AJAX interaction and upload flow (with backend for storing)
• [x] Change routes to make the doc page /application/documents
• [x] Make it automatically upload (no start button)
• [x] Fix progress bar
• [x] Attach unique key (per application) to the documents uploaded

Routing

• [x] Make previous screen's next step take you to /application/documents (two places: a button link in add'l household member QUESTION view, and the view after you've already done one additional household member)

UI

• [x] change buttons to make them mobile responsive (done on @alanjosephwilliams's branch)
• [x] Add button for "next step" (just a link to next page in the flow)

File processing

• [x] Resize images so they're smaller (on a single page)
• [x] Append all document images to the end of the application
• [x] Store final PDF on S3
• [x] Email link to S3 file (using middle-man route)
• [x] Make Paperclip use S3 instead of local file storage (production only)
• [x] Secure uploads so that they're not publicly accessible

insert YAKS / UNDERPANTS GNOMES here

• [x] Improve image quality (resize using MiniMagick rather than Prawn#fit, and then use Prawn to draw to a correctly-sized page)

Cleanup

• [ ] Remove any unwanted routes
• [ ] Delete dead JS from doc upload approach exploration
• [ ] Confirm no degradation from asset changes (JS, CSS)
• [ ] Document setting up S3/uploads in the README

MAYBE:

• [ ] Rename Upload model to Document model
• [ ] Make standalone doc upload page (use the same view, but add views and have diff backend and instantly send, etc)

[daguar]

Added:

• [ ] Delete dead JS from doc upload approach exploration
• [ ] Confirm no degradation from asset changes (JS, CSS)

[daguar]

Done: Make it automatically upload (no start button)

[daguar]

Added: fix progress bar

[daguar]

Fixed the progress bar in a slightly hacky way — swapped out the jquery-file-upload plugin's progress bar with the simple jquery-ui one: https://jqueryui.com/progressbar/

Experience/design could use some improvement, but the plumbing is there now.

[daguar]

Edited the storage item — let's not use Postgres since Heroku's $50 tier caps at 64gb of storage, and S3 will work fine (even though I don't love schlepping files across HTTP twice.)

[daguar]

Added: - [ ] Secure uploads so that they're not publicly accessible

[daguar]

Finished: Attach unique key (per application) to the documents uploaded

[daguar]

Reorganized under some broad buckets

[daguar]

First pass at processing done

[daguar]

Done with links from prior screens

[daguar]

w00t! done with "Append all document images to the end of the application"

First successful use of it on my local box!

[daguar]

Link for "next step" done (but need to make it a nice button; ugh, css)

[daguar]

Added: Resize images so they're smaller (on a single page)

[daguar]

Done with processing images for PDF appending! The strategy is this:

• If the doc is a PDF, simply append it
• If the doc is an image (JPG, PNG, GIF), draw it on its own PDF page (resizing it to fit) and then append that PDF page

Tested manually successfully with an iPhone.

[alanjosephwilliams]

woooot

On Thu, Apr 9, 2015 at 10:05 AM, Dave Guarino notifications@github.com wrote:

Done with processing images for PDF appending! The strategy is this:

• If the doc is a PDF, simply append it
• If the doc is an image (JPG, PNG, GIF), draw it on its own PDF page (resizing it to fit) and then append that PDF page

Tested manually successfully with an iPhone.

— Reply to this email directly or view it on GitHub https://github.com/codeforamerica/clean/issues/365#issuecomment-91293211 .

t: @alanjosephwilli p: 817 713 6264

[daguar]

Done with:

X - Store the location of the PDF in the database somewhere X - Make case has a Paperclip attachment with the final PDF X - Add has_attached_file to model X - Write a migration X - Try out the interface (what object does it expect?) X - Try it in full flow X - Delete zipping and attachment code X - Modify email body to put link to file

[daguar]

Done with using S3 for doc and PDF storage

[daguar]

Done with securing S3 URLs

[daguar]

Done with basic auth URL to give access to secured application PDFs for EWs

[daguar]

Merged in @alanjosephwilliams's front end work via #484

[daguar]

Successfully used on dev!

Two issues that came out of it (adding above as new tasks):

• Need to maintain image quality better — using Prawn's fit method to resize is causing quality issues which made 1 test doc illegible; since we use MiniMagick now, the best approach is probably leaning on that for the resizing, and then leaning on Prawn to draw it to a PDF page with normal dimensions
• The emails are not HTML, so the raw HTML is shown and our URL (while a link in Gmail) is shown twice (in the href and then inner element)

[lippytak]

Small user pain today = when user had a problem and emailed RMInquiries about it instead of us. Oops! This isn't RM's job so let's do our best to get their email address down from the public asap. Onwards!

[lippytak]

@daguar in the interest of chunkification, what are the remaining steps to launch? Check the QA issue for what I found: https://github.com/codeforamerica/clean/issues/486. LMK if you'd like me to make separate issues for the P1/P2 issues.

[daguar]

Checked off improvement of image quality (was #487)

[daguar]

Moved release checklist to https://github.com/codeforamerica/clean/issues/504