search

boyle / 2018-measure-stress

Using machine learning to investigate sympathetic activation of the autonomic nervous system (SAANS) during the treatment of mild traumatic brain injury, chronic pain, and post-traumatic stress disorder.
3 stars 1 forks source link

better upload for large files #25

Closed boyle closed 5 years ago

boyle commented 5 years ago

We may see very large file uploads. It would be nice for that to provide feedback on the upload.

Looks like one easy to install approach with a nice implementation would be https://github.com/flowjs/flow.js

which has an easy-to-adapt-to-flask python implementation on the server-side https://pypi.org/project/flowjs/

boyle commented 5 years ago

Flow.js was used. Hacking within the code, I can see there might be some failure mechanisms and currently very large uploads will stall at the end as the server merges all the uploaded segments into a single file. The upload will work, but a bit of patience is required.

A potential fix for this stall is to distribute the assembly over the upload so every N segments a merge happens. This would leave a small merge at the end and not be apparent to the user.

In addition, there may be problems with chunks colliding of there are many users uploading files with similar names. We mostly have one user doing uploads so we never expect this scenario, but the code is not inherently safe, unfortunately. As usual, to bullet-proof this is a major re-write of flow.js. Note that there is also a library on which flow.js was based (its not very well advertised within the code unfortunately), called resumable.js. Resumable may possibly be better maintained. A merger of the best of both of these codes might lead to a more robust solution.

None the less, this code does work, has been tested in production over the last two months, and so far no issues have arisen. We'll close for now as "fixed enough," despite this not being a perfect or entirely future-proof solution.