search

boyle / 2018-measure-stress

Using machine learning to investigate sympathetic activation of the autonomic nervous system (SAANS) during the treatment of mild traumatic brain injury, chronic pain, and post-traumatic stress disorder.
3 stars 1 forks source link

Files > 500 MB do not upload #45

Closed boyle closed 5 years ago

boyle commented 5 years ago

Increase file size limit. Currently 500 MB.

boyle commented 5 years ago

Hi Alistair,

I attempted to upload the vicon data for our pilot session on April 5th, participant 803. I received this message:

Request Entity Too Large The data value transmitted exceeds the capacity limit.

The properties of the file are below:

687 MB [image skipped]

If the Vicon data will be used in this form files this size are to be expected. Will this be a problem?

Andrew

boyle commented 5 years ago

I was able to upload all of N808 data including Vicon data (properties below).

492 MB [image skipped]

Andrew

boyle commented 5 years ago

Andrew,

Yup. File size limit is set to 500MB. Its easy to increase. I'll fix that ASAP.

I'll set the limit to double our expected max size. What is the largest file you would expect to upload?

Alistair

boyle commented 5 years ago

I have to get onto the CAREN to determine this as it depends on the duration of the application. I will run a test at 3:30pm to get a proper estimation and then get back to you.

Andrew

boyle commented 5 years ago

What is the largest file you would expect to upload?

I have to get onto the CAREN to determine this as it depends on the duration of the application. I will run a test at 3:30pm to get a proper estimation and then get back to you.

I imagine the file size is linear with time. How long are those two files you have (in time)? Probably ~20MB/min, I'd guess? Would we ever have a extended sessions (>1h appt or maybe healthy controls)?

boyle commented 5 years ago

Yes, it would be linear with time, and depend on the number of tracking markers. I asked Janet what is the longest a patient would last in an application she said 5 minutes. Based on N803 I would estimate max file with a 120 second buffer would be ~340000 KB.

APP Time (s) Size (KB)
20190405_0950_Visual_Sway_N803_SESSION_1_TR01 48 57492
20190405_1012_CCOP_ML_N803_SESSION_1_TR01 140 123386
20190405_1004_ROPEBRIDGE_N803_SESSION_1_TR01 209 179610
20190405_0954_PARK_222_N803_SESSION_1_TR01 295 245124
Regression   y = 763.45x + 19327
Correlation   R² = 0.9994
Estimated max 300 248362
Max time with 120 s buffer 420 339976

Andrew

boyle commented 5 years ago

This estimate was per/app sorry I misunderstood. Vicon data looks to be about 65 MB/minute. For 1 hr it would be ~2800 MB

Andrew

boyle commented 5 years ago

Ok, thanks. I'll update the limits. This is the quick fix. Hopefully that won't blow up the server. It currently stores everything in memory as it uploads. I think this will work well enough for now. The KISS design.

The longer short-term fix is to change the upload mechanism (Issue #25; to flow.js) to get small chunks transferred. You also get progress bars (upload speed), pause and resume. Essentially no change from a user point of view, except that it looks less ugly. Now I can justify fixing that. :)