Backup+mount+compression script

[wenluhu]

ln -s works on /data but not /sdcard. Some people say that the reason is /sdcard is FAT.

Trying to find a way to check /sdcard file system type. It doesn't show up in mount.

[bamos]

Hi @wenluhu - this is indeed an interesting error.

root@maguro:/mnt/shell/emulated/0 # ln -s /data test
link failed Function not implemented

Interestingly, DCIM on the sdcard gets routed back to /data/media/0/DCIM. However, symlinking on top of this still seems to cause an error when accessing through the sdcard interface. /sdcard is doing some routing I don't understand.

root@maguro:/data/media/0 # mv DCIM/ DCIM-old                                  
root@maguro:/data/media/0 # ln -s /data DCIM 
root@maguro:/data/media/0 # ls /sdcard/DCIM/                                   
/sdcard/DCIM/: Function not implemented
1|root@maguro:/data/media/0 # ls /sdcard/DCIM/                                 
/sdcard/DCIM/: Function not implemented
root@maguro:/data/media/0 # ls DCIM/                                           
anr
app
app-asec
app-lib
app-private
backup
bugreports
cal.bin
dalvik-cache
data
dontpanic
drm
gray
local
lost+found
lvm
media
mediadrm
misc
nbdroid
property
radio
resource-cache
security
smc
ssh
system
tombstones
user

Even trying to link to the original DCIM folder doesn't let me take pictures. The camera gives an error saying to mount the storage:

root@maguro:/data/media/0 # ln -s $PWD/DCIM-old/ DCIM                          
root@maguro:/data/media/0 # ls DCIM/                                           
100ANDRO
Camera
root@maguro:/data/media/0 # ls -l
lrwxrwxrwx root     root              2014-11-14 16:19 DCIM -> /data/media/0/DCIM-old/
drwxrwxr-x media_rw media_rw          2014-09-15 17:57 DCIM-old
10|root@maguro:/data/media/0 # chown media_rw:media_rw DCIM

I'm hopeful that mounting something to /data/media/0/DCIM or /sdcard/DCIM will work. Otherwise, the alternatives I can think of are diagnosing the symlink error on /sdcard/ and possibly patching this code, or using a non-standard camera application that saves to a directory of our choosing, since the default app doesn't seem to give an option of where to save the file.

\cc @theonewolf, weird Android errors we're working around.

[wenluhu]

That's interesting. How did you find out that your /sdcard is routed to /data/media/0/DCIM? This is how I find the route, and it ends at /mnt/shell/emulated/0.

root@maguro:/mnt/shell/emulated # ls /sd* -l                                   
lrwxrwxrwx root     root              2014-11-14 15:55 sdcard -> /storage/emulated/legacy
root@maguro:/mnt/shell/emulated # ls /storage/emulated/l* -l                   
lrwxrwxrwx root     root              2014-11-14 15:55 legacy -> /mnt/shell/emulated/0
root@maguro:/mnt/shell/emulated # ls /mnt/shell/emulated -l                    
drwxrwxr-x root     sdcard_rw          2014-11-14 16:12 0
drwxrwxr-x root     sdcard_rw          2014-11-07 14:02 legacy
drwxrwxr-x root     sdcard_rw          2014-11-07 14:01 obb

[wenluhu]

This is really interesting. ln -s succeeds only in /data/. And the symlink shows up is ls /sdcard, but not ls -l /sdcard. I suspect it's the problem of file ownership, but for some reason I cannot change it with chown.

root@maguro:/mnt/shell/emulated/0/DCIM # ln -s target Camera
link failed Function not implemented
root@maguro:/mnt/shell/emulated/0/DCIM # cd /data/media/0/DCIM/ 
root@maguro:/data/media/0/DCIM # ln -s target Camera                           

root@maguro:/data/media/0/DCIM # ls
100ANDRO
Camera
Camera-Backup
target
root@maguro:/data/media/0/DCIM # ls /sdcard/DCIM/                              
100ANDRO
Camera
Camera-Backup
target
root@maguro:/data/media/0/DCIM # ls /sdcard/DCIM/ -l                           
drwxrwxr-x root     sdcard_rw          2014-11-14 12:21 100ANDRO
drwxrwxr-x root     sdcard_rw          2014-11-14 16:29 Camera-Backup
drwxrwxr-x root     sdcard_rw          2014-11-14 16:29 target
root@maguro:/data/media/0/DCIM # ls -l
drwxrwxr-x media_rw media_rw          2014-11-14 12:21 100ANDRO
lrwxrwxrwx root     root              2014-11-14 16:36 Camera -> target
drwxrwxr-x media_rw media_rw          2014-11-14 16:29 Camera-Backup
drwxrwxr-x media_rw media_rw          2014-11-14 16:29 target

root@maguro:/data/media/0/DCIM # chown media_rw:media_rw Camera
root@maguro:/data/media/0/DCIM # ls -l
drwxrwxr-x media_rw media_rw          2014-11-14 12:21 100ANDRO
lrwxrwxrwx root     root              2014-11-14 16:36 Camera -> target
drwxrwxr-x media_rw media_rw          2014-11-14 16:29 Camera-Backup
drwxrwxr-x media_rw media_rw          2014-11-14 16:29 target

[bamos]

I found the route by brute force: adb shell ls -R /data | grep DCIM. I thought to run this because /sdcard didn't map to a block device. Try using chown on /data/..., but I think you'll still get the link failed error when trying to access over /sdcard. I wonder if we can just remove the weird sdcard interface and symlink /sdcard to somewhere in /data.

[bamos]

I still think the most favorable option is mounting something in /data/... and being to access it over the vanilla /sdcard/.... Try this first.

[wenluhu]

I'm trying to mount a device directly on /sdcard/DCIM/Camera. It is successfully mounted. If I take a picture right away, it works but the picture doesn't show up in that folder. If I wait a while and start the camera App, I see "Mount USB storage before using the camera" on the phone screen and I can't take a picture. Still working on it.

BTW, the way to change a symlink ownership is chown -h. However, Android doesn't recognize this option or -R

root@maguro:/data/media/0/DCIM # chown -h media_rw:media_rw Camera          
No such user '-h'

[wenluhu]

I can take pictures if I mount the device on /data/media/0/DCIM/Camera instead. I've tried both /dev/block/ram* and /dev/block/loop*, and have not seen any warnings or errors. But I cannot view the pictures either on the phone, or through adb. It seems like they are not properly written to the disk.

Do you guys have any thoughts? @bamos @theonewolf

[bamos]

Do the block devices you're mounting have a filesystem on them? What if you add a picture from the old camera directory to the newly mounted one, will the camera show it as a previously taken picture?

[wenluhu]

Yes, the device I'm mounting has a filesystem on it. Previously taken pictures, corrupted or not, show up in the camera app even if I don't move them to the Camera/ folder. I can move pictures to the Camera/ folder, but it doesn't make any difference to what I see in the camera app.

root@maguro:/data # ls /data/media/0/DCIM/Camera/                              
Dir_Created_in_Android
Dir_Created_in_Ubuntu
lost+found

[bamos]

Hmm, I wonder if the camera caches pictures. What if you restart the phone, then mount something on top of the Camera directory, then load the camera?

Maybe we can use a modified camera like https://github.com/almalence/OpenCamera to read and save images to a different locations. Not sure if that project supports it directly, but it shouldn't be hard to modify since the source is available.

[wenluhu]

I guess the camera app stores somewhere else a registry of the photos it has taken. It's really weird that in the camera app I can still see old pictures that I have deleted from DCIM/Camera. I have restarted the app and they are still there! I'm totally confused where it reads the old pictures.

dev/block/loop* has EXT4 filesystem on it, while dev/block/ram* has EXT2 filesystem on it. Could this be the problem? If the underlying filesystem is not the type the app is expecting, would the writes fail?

[bamos]

Hm, I don't think the filesystem type should impact it, but not 100% sure.

[theonewolf]

Isn't it supposed to be cloud-backed? Is Google auto-backing these up to the cloud at all?

Also, it would make sense that the VFS is caching them including metadata pointing to their original locations potentially. If you delete pictures and then reboot the phone, do they remain?

On Fri, Nov 14, 2014 at 10:12 PM, Brandon Amos notifications@github.com wrote:

Hm, I don't think the filesystem type should impact it, but not 100% sure.

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/22#issuecomment-63157985 .

Wolf

[wenluhu]

Old pictures disappear after I delete them and reboot. I doubt that Google is auto-backing them up as I don't use any Google accounts on this phone. I'll try to see if I can make this work in a hour. If I still can't get a clue, I'll use OpenCamera instead.

[wenluhu]

I took some pictures, restarted the phone, mount a loop device on /data/media/0/DCIM/Camera, started Camera App. I can view the pictures on the phone although they don't show up in ls /sdcard/DCIM/Camera. Then I force stop Gallery App and start it again, the pictures are still there.

I then took some more pictures when the loop device is mounted. I can see place holders for them in the Camera and Gallery App, instead of the real pictures taken, an all-black frame is displayed. I restarted the App and the place holders are still there.

I think caching can explain what I observed. The cache is established while booting, before running the camera app. And the cache is persistent to app restart.

I got the Camera app source code from git clone https://android.googlesource.com/platform/packages/apps/Camera.git to see if it is really using caching. Although I haven't find the exact code for caching yet, grep shows a lot of description about caching.

~/D/Camera> ls
Android.mk  CleanSpec.mk  jni/  MODULE_LICENSE_APACHE2  NOTICE  res/  src/  tests/
~/D/Camera> grep cache * -r -i
jni/feature_stab/doc/dbreg_API_doxyfile:# The SYMBOL_CACHE_SIZE determines the size of the internal cache use to
jni/feature_stab/doc/dbreg_API_doxyfile:# When the cache is full, less often used symbols will be written to disk.
jni/feature_stab/doc/dbreg_API_doxyfile:# probably good enough. For larger projects a too small cache size can cause
jni/feature_stab/doc/dbreg_API_doxyfile:# If the system has enough physical memory increasing the cache will improve the
jni/feature_stab/doc/dbreg_API_doxyfile:# memory usage. The cache size is given by this formula:
jni/feature_stab/doc/dbreg_API_doxyfile:# 2^(16+SYMBOL_CACHE_SIZE). The valid range is 0..9, the default is 0,
jni/feature_stab/doc/dbreg_API_doxyfile:# corresponding to a cache size of 2^16 = 65536 symbols
jni/feature_stab/doc/dbreg_API_doxyfile:SYMBOL_CACHE_SIZE      = 0
jni/feature_mos/doc/feature_mos_API_doxyfile:# The SYMBOL_CACHE_SIZE determines the size of the internal cache use to
jni/feature_mos/doc/feature_mos_API_doxyfile:# When the cache is full, less often used symbols will be written to disk.
jni/feature_mos/doc/feature_mos_API_doxyfile:# probably good enough. For larger projects a too small cache size can cause
jni/feature_mos/doc/feature_mos_API_doxyfile:# If the system has enough physical memory increasing the cache will improve the
jni/feature_mos/doc/feature_mos_API_doxyfile:# memory usage. The cache size is given by this formula:
jni/feature_mos/doc/feature_mos_API_doxyfile:# 2^(16+SYMBOL_CACHE_SIZE). The valid range is 0..9, the default is 0,
jni/feature_mos/doc/feature_mos_API_doxyfile:# corresponding to a cache size of 2^16 = 65536 symbols
jni/feature_mos/doc/feature_mos_API_doxyfile:SYMBOL_CACHE_SIZE      = 0
res/values/styles.xml:        <item name="android:colorBackgroundCacheHint">@android:color/black</item>
src/com/android/camera/PreferenceInflater.java:                // Class not found in the cache, see if it's real, and try to
tests/src/com/android/camera/activity/CameraTestCase.java:                return getInstrumentation().getTargetContext().getCacheDir();

~/D/Camera> grep DCIM * -r -i
src/com/android/camera/Storage.java:    public static final String DCIM =
src/com/android/camera/Storage.java:            Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DCIM).toString();
src/com/android/camera/Storage.java:    public static final String DIRECTORY = DCIM + "/Camera";
src/com/android/camera/Storage.java:     * OSX requires plugged-in USB storage to have path /DCIM/NNNAAAAA to be
src/com/android/camera/Storage.java:        File nnnAAAAA = new File(DCIM, "100ANDRO");
tests/src/com/android/camera/stress/ShotToShotLatency.java:            Environment.getExternalStorageDirectory().toString() + "/DCIM/Camera/";

This doesn't look promising. I should probably try OpenCamera first.

• [x] Try OpenCamera. Make sure this approach works.
• [ ] If time permits, look into Camera App source code to figure out what exactly is happening.

[wenluhu]

OpenCamera uses API level 21 (Android 5.0). An issue says this new version of code should work on old devices. I have to upgrade my Eclipse and JDK, install NDK, to be able to build it. Now it works on the phone!

• [x] Change the picture saving location.
• [x] See if pictures can be saved to a device we mount

One other option is to roll back to old versions. But the repository has no releases, I'm not exactly sure which commit to roll back too.

[wenluhu]

OpenCamera has an user interface to choose a custom folder to save the pictures. But the folders are limited to those in /storage, which is read-only. The only top-level option is /storage/sdcard0. I thought about remounting this partition as read-write. But /storage doesn't show up in mount.

So I still mount the loop device on /data/media/0/DCIM/Camera and try to see what errors I get. First I got FileNotFound for /storage/emulated/0/DCIM/Camera. I go to the directory and found out that /storage/emulated/0/ has disappeared! So I force the app to write to /data/media/0/DCIM/Camera. I still get the same error. I've checked write permissions in the manifest and didn't see anything wrong. I'm going to try the seemingly-weird suggestions in this post.

11-16 16:47:11.627: W/System.err(6736): java.io.FileNotFoundException: /data/media/0/DCIM/Camera/20141116_164711.jpg: open failed: EACCES (Permission denied)
11-16 16:47:11.627: W/System.err(6736):     at libcore.io.IoBridge.open(IoBridge.java:409)
11-16 16:47:11.627: W/System.err(6736):     at java.io.FileOutputStream.<init>(FileOutputStream.java:88)
11-16 16:47:11.627: W/System.err(6736):     at java.io.FileOutputStream.<init>(FileOutputStream.java:73)
11-16 16:47:11.627: W/System.err(6736):     at com.almalence.opencam.PluginManager.saveResultPicture(PluginManager.java:2424)
11-16 16:47:11.627: W/System.err(6736):     at com.almalence.plugins.export.standard.ExportPlugin.onExportActive(ExportPlugin.java:75)
11-16 16:47:11.627: W/System.err(6736):     at com.almalence.opencam.PluginManager$ProcessingTask.doInBackground(PluginManager.java:2024)
11-16 16:47:11.627: W/System.err(6736):     at com.almalence.opencam.PluginManager$ProcessingTask.doInBackground(PluginManager.java:1)
11-16 16:47:11.627: W/System.err(6736):     at android.os.AsyncTask$2.call(AsyncTask.java:287)
11-16 16:47:11.627: W/System.err(6736):     at java.util.concurrent.FutureTask.run(FutureTask.java:234)
11-16 16:47:11.627: W/System.err(6736):     at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
11-16 16:47:11.627: W/System.err(6736):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
11-16 16:47:11.627: W/System.err(6736):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
11-16 16:47:11.635: W/System.err(6736):     at java.lang.Thread.run(Thread.java:841)
11-16 16:47:11.635: W/System.err(6736): Caused by: libcore.io.ErrnoException: open failed: EACCES (Permission denied)
11-16 16:47:11.635: W/System.err(6736):     at libcore.io.Posix.open(Native Method)
11-16 16:47:11.635: W/System.err(6736):     at libcore.io.BlockGuardOs.open(BlockGuardOs.java:110)
11-16 16:47:11.635: W/System.err(6736):     at libcore.io.IoBridge.open(IoBridge.java:393)
11-16 16:47:11.635: W/System.err(6736):     ... 12 more
11-16 16:47:11.635: W/System.err(6736): java.io.FileNotFoundException: /storage/emulated/0/DCIM/Camera/20141116_164711.jpg: open failed: EACCES (Permission denied)
11-16 16:47:11.635: W/System.err(6736):     at libcore.io.IoBridge.open(IoBridge.java:409)
11-16 16:47:11.635: W/System.err(6736):     at java.io.FileOutputStream.<init>(FileOutputStream.java:88)
11-16 16:47:11.635: W/System.err(6736):     at java.io.FileOutputStream.<init>(FileOutputStream.java:73)
11-16 16:47:11.635: W/System.err(6736):     at com.almalence.opencam.PluginManager.saveResultPicture(PluginManager.java:2437)
11-16 16:47:11.635: W/System.err(6736):     at com.almalence.plugins.export.standard.ExportPlugin.onExportActive(ExportPlugin.java:75)
11-16 16:47:11.635: W/System.err(6736):     at com.almalence.opencam.PluginManager$ProcessingTask.doInBackground(PluginManager.java:2024)
11-16 16:47:11.635: W/System.err(6736):     at com.almalence.opencam.PluginManager$ProcessingTask.doInBackground(PluginManager.java:1)
11-16 16:47:11.635: W/System.err(6736):     at android.os.AsyncTask$2.call(AsyncTask.java:287)
11-16 16:47:11.635: W/System.err(6736):     at java.util.concurrent.FutureTask.run(FutureTask.java:234)
11-16 16:47:11.635: W/System.err(6736):     at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:230)
11-16 16:47:11.635: W/System.err(6736):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
11-16 16:47:11.643: W/System.err(6736):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
11-16 16:47:11.643: W/System.err(6736):     at java.lang.Thread.run(Thread.java:841)
11-16 16:47:11.643: W/System.err(6736): Caused by: libcore.io.ErrnoException: open failed: EACCES (Permission denied)
11-16 16:47:11.643: W/System.err(6736):     at libcore.io.Posix.open(Native Method)
11-16 16:47:11.643: W/System.err(6736):     at libcore.io.BlockGuardOs.open(BlockGuardOs.java:110)
11-16 16:47:11.643: W/System.err(6736):     at libcore.io.IoBridge.open(IoBridge.java:393)
11-16 16:47:11.643: W/System.err(6736):     ... 12 more

[wenluhu]

Maybe I should not play with the code now. It might be easier to just mount the device somewhere in /sdcard and point the app there through user interface.

It works perfectly when I point it to a normal directory. But when I point it to the mounted point, it says "Sorry, we can't put your photos here :(". Fixed the owner of the folder, but still.

[wenluhu]

Hooray!!! It works!!

It's a stupid mistake. I forgot to check the permission of the folder. Just changed it to 775, and it works. I have pulled the jpg from the phone and I'm able to view it. So it is properly written to the loop device.

I still can't view the photo on the phone though. By default, OpenCamera calls native Gallery to view the photos. It's the problem of Gallery. But I guess this is not as big a problem for the demo. As long as it's properly written to disk, we should be able to view it on GammaRay.

[bamos]

Great to hear! Are you using OpenCamera?

[wenluhu]

Yes, it works with OpenCamera now. I don't have a clear clue on how to make native Camera work. I'm going to poke around and see what I've missed.

For native camera,

• [x] check folder permission and ownership after mounting the device

[wenluhu]

For native camera app, I fixed the permission and ownership of /sdcard/DCIM/Camera, and force stopped a couple of stuff. I can take pictures and view pictures on the phone! BUT... the pictures don't show up in the loop device. After I umount the device, all the pictures I just took show up in /sdcard/DCIM/Camera! I'm feeling like the camera app is operating at one layer below the file system.

If I don't force stop anything, I got "Your USB storage is running out of space. Change the quality setting or delete some images or other files". My loop device has about 20MB free space.

[theonewolf]

This might be due to some safety feature inside the Android OS actually trying to avoid security vulnerabilities.

I think perhaps the Camera app or something it interfaces with has an open handle to that folder/a pointer to the original folder. Thus, its writes go into the original file system which becomes invisible to your ls tools etc. because they don't have a "connection" to anything---they start from scratch when they execute, and they go into a newly mounted file system.

Now, you've definitely tried this right:

1. Turn off Camera native app
2. Mount the file system to /sdcard/DCIM/Camera (set permissions etc.)
3. Turn on Camera native app, take pics

If they still aren't showing up there, than my suspicion is that the Camera app is saving them via some already-running (continuously running?) "service" and that "service" has a direct handle into the original folder. Thus, it can not be easily re-directed or subverted.

On Sun, Nov 16, 2014 at 6:47 PM, Wenlu Hu notifications@github.com wrote:

For native camera app, I fixed the permission and ownership of /sdcard/DCIM/Camera, and force stopped a couple of stuff. I can take pictures and view pictures on the phone! BUT... the pictures don't show up in the loop device. After I umount the device, all the pictures I just took show up in /sdcard/DCIM/Camera! I'm feeling like the camera app is operating at one layer below the file system.

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/22#issuecomment-63245900 .

Wolf

[theonewolf]

I wish we could "strace" the Camera or Gallery apps to see how they save those photos/see what they might communicate with.

At this point though, if you have OpenCamera working, then that is good enough for a demo :-) Focus on making OpenCamera smooth.

On Sun, Nov 16, 2014 at 8:23 PM, Wolfgang Richter wolf@cs.cmu.edu wrote:

This might be due to some safety feature inside the Android OS actually trying to avoid security vulnerabilities.

I think perhaps the Camera app or something it interfaces with has an open handle to that folder/a pointer to the original folder. Thus, its writes go into the original file system which becomes invisible to your ls tools etc. because they don't have a "connection" to anything---they start from scratch when they execute, and they go into a newly mounted file system.

Now, you've definitely tried this right:

1. Turn off Camera native app
2. Mount the file system to /sdcard/DCIM/Camera (set permissions etc.)
3. Turn on Camera native app, take pics

If they still aren't showing up there, than my suspicion is that the Camera app is saving them via some already-running (continuously running?) "service" and that "service" has a direct handle into the original folder. Thus, it can not be easily re-directed or subverted.

On Sun, Nov 16, 2014 at 6:47 PM, Wenlu Hu notifications@github.com wrote:

For native camera app, I fixed the permission and ownership of /sdcard/DCIM/Camera, and force stopped a couple of stuff. I can take pictures and view pictures on the phone! BUT... the pictures don't show up in the loop device. After I umount the device, all the pictures I just took show up in /sdcard/DCIM/Camera! I'm feeling like the camera app is operating at one layer below the file system.

— Reply to this email directly or view it on GitHub https://github.com/wenluhu/gammaray-android/issues/22#issuecomment-63245900 .

Wolf

Wolf