search

photo / mobile-android

Trovebox mobile application for Android
Apache License 2.0
120 stars 64 forks source link

Sync: corrupted uploaded photos #290

Open httpdispatch opened 11 years ago

httpdispatch commented 11 years ago

User on twitter claimed about corrupted uploaded photos https://twitter.com/thaddeusmt/status/290813328786329600 "Evan Johnson ‏@thaddeusmt @OpenPhoto Notice anything funny about these images synced from Android to Dropbox? http://i.imgur.com/pkDYo.jpg http://i.imgur.com/WW9Pe.jpg Cause?"

User uses dropbox as a storage. @jmathai @patricksan do you have any ideas where the issue is?

jmathai commented 11 years ago

I've seen this problem. It's not related to Dropbox but with the generated photo. I believe it happens when...

  1. You request a size of a photo that doesn't already exist
  2. Display that image which has a create URL
  3. The photo is generated correctly but is sent to the user as corrupt
  4. The corrupt photo is cached and the non-corrupt photo is never requested from disk

This is a valid issue we should look into.

httpdispatch commented 11 years ago

What do you mean cached and sent to the user? It is android app sending photos from device to server and it is how photos look on the web.

jmathai commented 11 years ago

The Android app uploads the original. The server stores it and a lower resolution at around 1280x1280. Every other size is generated later when it is requested. It's these lower resolution photos that appear to be corrupt.

httpdispatch commented 11 years ago

Are the original uploads corrupted or they are OK?

httpdispatch commented 11 years ago

@jmathai does server resize all the uploaded images and doesn't store the original iteslf?

thaddeusmt commented 11 years ago

I sent the original tweet, thanks for opening up an official bug report. Here are a few things I noticed:

  1. This started when I selected ~400 photos in the Android App's "Sync" screen to, well, sync
  2. I got a lot of "Upload failed" notifications on my phone while it was syncing.
  3. It appeared to re-try the failed images and eventually completed the sync
  4. I'm not sure if the corrupted ones were the ones failing, I did not think unfortunately to check that before clearing out the phone notifications.
  5. I'm assuming that's the case though - if an image was corrupted there would be 2-6 versions of it, usually one of which was non-corrupted, making it appear that it kept re-trying them after failing
  6. I looks like it would fail, not clean up the corrupted image from the failure, then try and fail again, until it finally had a non-corrupted image, leaving all of the corrupted ones on Dropbox still.
  7. I did check the "Auto-upload" and "Wi-Fi only" option on Android, but have not changed any other settings from the defaults (I did not change image size or anything)
  8. I was using the web interface while the Android app was still syncing
  9. I also had my desktop Dropbox folder open, spying on how it was organizing my uploads in directories
  10. I do not see where it is storing multiple image sizes in Dropbox - all I see (now that I deleted the duplicate corrupted ones) are a single copy of each photo in the original (full) size
  11. The full-sized images are corrupt, not the resized ones (I don't see any resized ones, as I said)
  12. Not all of the images had corrupt versions, some uploaded correctly the first time, no problems.
  13. The corruption is weird - sometimes it's a color shift, sometimes it's a weird repeating-crop thing, sometimes it's both

Here is an example of a full-sized corrupt image: IMAG0359-50f37d1a0fffb

I hope that helps, thanks for looking in to it!

jmathai commented 11 years ago

@httpdispatch The server stores the original and additionally a 1280x1280 version. Then all other sizes are created as needed.

jmathai commented 11 years ago

@thaddeusmt Thanks for the info. It contradicts what I had originally thought so it's very helpful. A couple of things I'm curious about.

  1. There should never be the same photo uploaded twice which exists on the file system (Dropbox, here) and in the database. Do you see duplicate versions of the same photo through the web interface (this would imply there are multiple records for the same photo in the database). If so, could you let me know which ones?
  2. We don't store low resolution photos in your Dropbox account. Dropbox doesn't fare well as a CDN so we cache those files to deliver photos for viewing in the web and mobile apps.
  3. What Android device and OS version are you using?

Thanks!

thaddeusmt commented 11 years ago

1 Yes, I see multiple of the same photo in the web interface. I just made a couple of the files public so you can take a look that they are indeed viewable through the web interface: http://thaddeusmt.openphoto.me/photos/list They have slightly different names in Dropbox with a unique hash at the end, like IMAG0359-50f3899085886.jpg and IMAG0359-50f37d1a0fffb.jpg

  1. Thanks for the info about why Dropbox doesn't have all the image sizes
  2. I am running Windows 7 and my phone is an HTC Incredible 2 running Android 2.3.4
httpdispatch commented 11 years ago

@jmathai How i understand from the @thaddeusmt description android app reports about failed upload and tries to reupload photos. But server accepts even partially uploaded photos. Is it correct?

httpdispatch commented 11 years ago

@jmathai any ideas?

jmathai commented 11 years ago

@httpdispatch The server doesn't validate that the image isn't corrupt but it waits until the upload is complete. If it's terminated early it should return a 500 or empty response and not store the photo.

I've seen this issue with uploads locally not using Android. I'm not sure the app is the culprit.