skoobe / riofs

Userspace S3 filesystem
GNU General Public License v3.0
393 stars 60 forks source link

Content-type not set correctly for large files (ie, multipart uploads) #87

Open jamshid opened 9 years ago

jamshid commented 9 years ago

I built riofs with ./configure --with-libmagic because I want my S3 objects to have a correct MIME type.

When riofs uploads a large file, e.g., after saving youtube-dl 'https://www.youtube.com/watch?v=tMoTfLykVcA' in /tmp then cp /tmp/cops.mp4 /mnt/riofs/, I see it does a multipart upload to S3, splitting the file into 3 parts. I see after the POST to complete the multipart upload, it does a followup PUT with copy with Content-type: application/octet-stream. I expected that to be video/mp4.

The problem seems to be that it doesn't try to determine the MIME type until the last part is sent. Is it really only passing that last part's bytes to libmagic?

04:20:07 [fio] (fileio_release_on_part_con_cb file_io_ops.c:359) Guessed MIME type of /cops.mp4?partNumber=3&uploadId=76fdb47d9ddc5400940aeb2c255efce9 as application/octet-stream

The file is correctly identified by "file --mime":

root@c076e670d07a:/# file --mime /mnt/riofs/cops.mp4 
/mnt/riofs/cops.mp4: video/mp4; charset=binary
wizzard commented 9 years ago

Hello,

could you please send me a sample of such file?

Thank you!

jamshid commented 9 years ago

Can you install youtube-dl, e.g. "brew install youtube-dl" (http://rg3.github.io/youtube-dl/download.html)?

Then just use it to download a .mp4 file I happened to notice the problem with, ~15MB:

$ youtube-dl -o cops.mp4 'https://www.youtube.com/watch?v=tMoTfLykVcA'