ext3/4 support

[arcaine2]

Does Dagang support ext3/4 as external media? I have a FAT32 formatted pendrive (brand-new) connected to USB port and enabled timelapse to save JPGs there. This works fine but there are always some issues while deleting old images (tons of them each day, deleting them takes a longer while) that lead to filesystem corruption. The more data to delete, the more often it corrupts, switch filesystem to read-only mode what have to be fixed manually. fsck on the camera often segfaults (possibly not enough free RAM). I recently switched from pictures based timelapse to my own recording script that records one hour long video and switches to a new file every hour. This gives me 24 files per day but there are still times when filesystem switches to ro mode while removing old files. Maybe ext filesystem would be more stable but i don't see required mkfs utility available.

[saipsa]

Can you share your recording script?

Thanks!

[arcaine2]

Sure, but don't judge, it needs some rewriting. I essentially modified some other script:

#!/bin/sh                                                                                                                                                                                                  
PIDFILE="/run/recording.pid"                                                                                                                                                                               
BASE_SAVE_DIR="/system/media/pendrive/DCIM/VIDEO"                                                                                                                                                          

SAVE_DIR_PER_DAY=1                                                                                                                                                                                         
SAVE_DIR_PER_HOUR=0                                                                                                                                                                                        
AUTO_DELETE_OLD_RECORDINGS_DAYS=5                                                                                                                                                                          
SLEEP_INTERVAL=1                                                                                                                                                                                           

if [ ! -d "$BASE_SAVE_DIR" ]; then                                                                                                                                                                         
  mkdir -p $BASE_SAVE_DIR                                                                                                                                                                                  
fi                                                                                                                                                                                                         

status()                                                                                                                                                                                                   
{                                                                                                                                                                                                          
  pid="$(cat "$PIDFILE" 2>/dev/null)"                                                                                                                                                                      
  if [ "$pid" ]; then                                                                                                                                                                                      
    kill -0 "$pid" >/dev/null && echo "PID: $pid" || return 1                                                                                                                                              
  fi                                                                                                                                                                                                       
}                                                                                                                                                                                                          

start()                                                                                                                                                                                                    
{                                                                                                                                                                                                          
  if [ -f $PIDFILE ]; then                                                                                                                                                                                 
    echo "Recording already running";                                                                                                                                                                      
  else                                                                                                                                                                                                     
    echo "Start recording"                                                                                                                                                                                 
    while true; do                                                                                                                                                                                         
      CURRENT_TIME=$(date +%M%S)                                                                                                                                                                           
      if [ $CURRENT_TIME == "0000" ]; then                                                                                                                                                                 
        echo "Full hour, reset the recording"                                                                                                                                                              
        stop                                                                                                                                                                                               
      elif [ ! -f $PIDFILE ]; then                                                                                                                                                                         
        if [ $SAVE_DIR_PER_HOUR == 1 ]; then                                                                                                                                                               
            SAVE_DIR="$BASE_SAVE_DIR/$(date +%Y-%m-%d_%H.00)"                                                                                                                                              
        elif [ $SAVE_DIR_PER_DAY == 1 ]; then                                                                                                                                                              
            SAVE_DIR="$BASE_SAVE_DIR/$(date +%Y-%m-%d)"                                                                                                                                                    
        fi                                                                                                                                                                                                 
        if [ ! -d "$SAVE_DIR" ]; then                                                                                                                                                                      
            mkdir -p $SAVE_DIR                                                                                                                                                                             
        fi                                                                                                                                                                                                 
        FILE_NAME="REC_$(date +%Y-%m-%d_%H%M%S.mp4)"                                                                                                                                                       
        RECORDING_PATH="$SAVE_DIR/$FILE_NAME"                                                                                                                                                              
        echo "Recording $FILE_NAME to $RECORDING_PATH"                                                                                                                                                     
        /system/sdcard/bin/busybox nohup /system/sdcard/bin/avconv -rtsp_transport tcp -y -i rtsp://0.0.0.0:8554/unicast -vcodec copy -c:a aac -strict experimental "$RECORDING_PATH"  &>/dev/null &       
        echo "$!" > "$PIDFILE"                                                                                                                                                                             
      fi                                                                                                                                                                                                   
      #echo "Waiting $SLEEP_INTERVAL"                                                                                                                                                                      
      sleep $SLEEP_INTERVAL                                                                                                                                                                                
    done                                                                                                                                                                                                   
  fi                                                                                                                                                                                                       
}                                                                                                                                                                                                          

stop()                                                                                                                                                                                                     
{                                                                                                                                                                                                          
  pid="$(cat "$PIDFILE" 2>/dev/null)"                                                                                                                                                                      
  if [ "$pid" ]; then                                                                                                                                                                                      
    kill "$pid" &&  rm "$PIDFILE"                                                                                                                                                                          
    killall avconv                                                                                                                                                                                         
    echo "Stopped recording"                                                                                                                                                                               
  else                                                                                                                                                                                                     
    echo "Could not find a running recording to stop."                                                                                                                                                     
  fi                                                                                                                                                                                                       
}                                                                                                                                                                                                          

if [ $# -eq 0 ]; then                                                                                                                                                                                      
  start                                                                                                                                                                                                    
else                                                                                                                                                                                                       
  case $1 in start|stop|status)                                                                                                                                                                            
      $1                                                                                                                                                                                                   
      ;;                                                                                                                                                                                                   
  esac                                                                                                                                                                                                     
fi                               

As for deleting old data, i have this added to crontab and run every night:

#!/bin/sh

BASE_SAVE_DIR='/system/media/pendrive/DCIM/VIDEO/'
#SAVE_DIR_PER_DAY=0
#SAVE_DIR_PER_HOUR=1
AUTO_DELETE_OLD_RECORDINGS_DAYS="+5"

if [ $AUTO_DELETE_OLD_RECORDINGS_DAYS -gt 0 ]; then
  echo "Deleting directories older than $AUTO_DELETE_OLD_RECORDINGS_DAYS in $BASE_SAVE_DIR"
  echo `date` >> /system/media/pendrive/last_cleaned.txt
  #MAXDEPTH 1 for scan only the current dir not everything inside that take ages
  #find $BASE_SAVE_DIR -maxdepth 1 -type d -mtime $AUTO_DELETE_OLD_RECORDINGS_DAYS -print -exec rm -rf "{}" \;
  find $BASE_SAVE_DIR -maxdepth 2 -mtime $AUTO_DELETE_OLD_RECORDINGS_DAYS -print -exec rm -rf "{}" \;
fi

[jmtatsch]

Not sure about ext3 support. Try adding some swap maybe that helps. Or use the rootfs - that is based on ext3.

[arcaine2]

Swap doesn't help to prevent filesystem corruption. It could help with fsck but i had 128MB swap added (on microSD so it wasn't on read-onny disk) and fsck still crashed.

Or use the rootfs - that is based on ext3.

Can i create such filesystem directly on dafang or how to do that?

[jmtatsch]

@arcaine2 Look at Dafang-Hacks/rootfs and checkout my pullrequest.

[arcaine2]

@jmtatsch, ah, i misunderstood you. To use that "rootfs" and its kernel i just have to copy the files directly to sdcard assuming i already have custom bootloader flashed?

[jmtatsch]

Exactly

[chisht]

@arcaine2 , You can easily compile ext3/ext4 modules and insmod them on the target. This is what I did,

1. Checked out the kernel code.
2. Enabled ext3 and ext4 support as modules in config
3. Compiled kernel and the modules.
4. Sideloaded modules files to the target and insmod them manually.

And now it show ext3 and ext4 as supported filesystem. Though I did not format my usb to either and test, but I see no reason why it won't work.

[root@DAFANG:sdcard]# cat /proc/filesystems
nodev   sysfs
nodev   rootfs
nodev   bdev
nodev   proc
nodev   cgroup
nodev   tmpfs
nodev   debugfs
nodev   sockfs
nodev   pipefs
nodev   anon_inodefs
nodev   rpc_pipefs
nodev   configfs
nodev   devpts
        squashfs
nodev   ramfs
        vfat
        msdos
nodev   nfs
nodev   jffs2
nodev   mtd_inodefs
        ext3
        ext2
        ext4
[root@DAFANG:sdcard]# lsmod
ext4 388494 0 - Live 0xc0934000
jbd2 78413 1 ext4, Live 0xc0898000
ext3 147548 0 - Live 0xc084d000
mbcache 6453 2 ext4,ext3, Live 0xc0814000
jbd 61463 1 ext3, Live 0xc0797000
crc16 1261 1 ext4, Live 0xc077c000
sensor_jxf22 8976 1 - Live 0xc061e000 (O)
tx_isp 363556 3 - Live 0xc05ae000 (O)
sinfo 7814 0 - Live 0xc053d000 (O)
sample_motor 11889 0 - Live 0xc0531000 (O)
audio 60487 1 - Live 0xc0517000 (O)
8189es 1233474 0 - Live 0xc033e000 (O)

[arcaine2]

@chisht you only compiled new kernel and used it instead of "stock" one with no other changes to the filesystem etc? Could you upload those modules somewhere? I could use the new rootfs but i saw it was not entirely complete, with some paths missing etc.

To be honest, recording on dafang is a bit unstable for me in general. I use camera as a simple monitoring and have a preview opened nearly 24/7 on a tablet (using tinycam). If i have recording turned on with avconv on the camera itself, every once in a while (it's random, sometimes every couple hours, sometimes days) my preview is not available with an error message that server is not working. If i stop recording script or kill avconv preview on tablet comes back almost instantly. I don't anything weird in logs, i don't have to restart rtsp server on the camera.

[chisht]

@arcaine2 , Sorry can't help with instability with preview. I only started playing around with this camera. As for the ext3/ext4, no I did not load the whole kernel. That'd be risky. Instead I only ftped the kernel modules and insmoded them on the target manually. This way, it either works or it does not work. Can't break anything.

If you want to try my compiled modules, I uploaded them to my my fork https://github.com/chisht/Xiaomi-Dafang-Software/blob/chisht-patch-1/ext3_4_modules.zip Extract it to the target on any folder and then insmod them as,

insmod crc16.ko
insmod jbd.ko
insmod jbd2.ko
insmod mbcache.ko
insmod ext3.ko
insmod ext4.ko

And now you should have ext3/ext4 support. You can confirm it by doing, cat /proc/filesystems

[arcaine2]

Those modules loads fine. Still need to format the drive and test if the issue i had with that pendrive formated as FAT32 persist.

[arcaine2]

While i still did not manage to find out why sometimes avconv hangs when recording locally, on the camera and after all, moved on from recording directly on camera to recording remote stream on OrangePi Zero in the same network. I do record 15 minutes streams, using ffmpeg and script put in crontab and while i get less corrupted video than when i was doing the same straight on the Dafang, i still see them. This seems to be related with camera having possibly unstable WiFi connection when someone is moving nearby but i'm not 100% sure. If someone has an idea how to verify this, let me know.

Ext3 and ext4 support is fine with those modules loaded, so that's solved and issue can be closed.

[Yannik]

@chisht which kernel source did you use to compile these modules? can you perhaps upload the patched kernel config?

[chisht]

@chisht which kernel source did you use to compile these modules? can you perhaps upload the patched kernel config?

I don't have that setup anymore. But I'm pretty sure I used the kernel from https://github.com/Dafang-Hacks/Main And I believe .config in there is already set for the camera.