Record video on camera on-demand

[thematrixdev]

I have been storing purely video (with no audio) from the camera. Recently I want to store both video and audio.

This is my situation. https://stackoverflow.com/questions/72135598/recording-rtsp-steam-with-python

I cannot find any good solution to do the detection and store the file.

May I know if it is possible to store file on-demand on the camera? That is, by calling a REST-API to the camera, the camera starts recording, stop recording and send out the video.

[mnakada]

I think the REST-API for camera start,stop takes too much time from detection to start recording to achieve the expected behavior.

Is it possible to receive the mp4 file via webhook and then filter it with human detection in post-processing?

If you set the camera settings to motion detection on and human detection and pet detection off, all motion detected files will be transferred, so the amount of transfer will be higher, but the first part of the file will not be missed.

[thematrixdev]

As I remember, the duration of video taken by the camera is limited to 12 seconds, right?

[mnakada]

The file recorded on the "alarm_record directory" side is 12 seconds long. But the file on the "record directory" side that is recorded at the same time appears to be recorded continuously during motion detection. The file is split every minute, though, so it needs to be post-processed.

[thematrixdev]

When video is taken by the camera, it is uploaded to Atom's server (should be AWS S3), and notification will be shown on Atom app. Do you think we can skip uploading to Atom's server? If there is no proper way, maybe I can point DNS request to AWS from camera to a dummy IP.

[mnakada]

Any reason you don't want to upload to AtomTech's AWS? Is it because there are too many notifications to the app?

[thematrixdev]

Exactly...

[thematrixdev]

And recently I suspect Atom is a Chinese founded company.

[mnakada]

And recently I suspect Atom is a Chinese founded company.

That may be true. But it may be difficult to run the camera without using the app.

As for video uploads, I have already included a mechanism in curl.c in libcallback to skip re-posts within 5 minutes. This is to avoid overloading AtomTech's AWS, as the tools reduce the motion detection insensitivity time to 30 seconds. It looks like it could be done by modifying this, but if you want to stop all communication, you might want to redirect DNS.

[thematrixdev]

May I know where is the code to skip re-posting within 5 minutes? I may try setting in to be a very big number.

[mnakada]

https://github.com/mnakada/atomcam_tools/blob/bdf6deeeaa1dd4ec36c7ef77ba18620be2ae1f3c/libcallback/curl.c#L103

[mnakada]

I have put in a mechanism to stop the uploading of videos to aws. Add the following to line 11 of /scripts/atom_init.sh

export ATOMTECH_AWS_ACCESS=disable_video

This will not stop the exchange of settings, but it will stop the video upload.

[thematrixdev]

Do you like to add this as an option on WEB-UI?

[mnakada]

Yes. I will add it in the next UPDATE.

[thematrixdev]

Thank you very much.

[thematrixdev]

Excuse me, what options are needed to enable in order to record video (with audio) into "record" directory?

[mnakada]

From the AtomCam app settings, turn on "録画及びストレージ管理 -> SDへのローカル録画". "録画モード" should be "モーション検知時のみ". App is not WebUI.

[thematrixdev]

Okay. How about on the WEB-UI? What options are necessary to be switched on?

[mnakada]

If it is a file on an SD-Card that you need, turn on "SD-Card録画". In that case, reading the file via http is the lightest load. Open the "ファイル一覧" SD-Card button and check the URL of the file.

If you need the file on the NAS, turn on "NAS録画".

[thematrixdev]

So both "SDへのローカル録画" on app and "SD-Card録画" needed to be turned on in order to record to "record" folder? Only turning one single option won't trigger the recording? Sorry but I am a bit confused.

[mnakada]

"SDへのローカル録画" in the AtomCam-App settings turns on the writing operation of iCamera_app to the SD-Card.

If the setting is on, iCamera_app will mv to SD-Card upon completion of a 1 minute mp4 file recorded to /tmp.

This hack replaces mv with script and sidesteps it.

In it, it records to SD-Card, records to NAS, and sends WebHook events according to the WebUI settings.

For this reason, both the app settings and the WebUI settings need to be turned on.

[thematrixdev]

This is my camera setting:

And in Web-UI:

For example, I waved my hand in front of the camera at 21:30:30. In SMB, there is a file called 213000.mp4, which contains 30 seconds of static image (no motion) and then my waving hand. Is it normal? I think it starts recording just after there is motion.

And there is no new video if there is no motion detected.

[mnakada]

No, iCamera_app generates an mp4 file in /tmp every minute from 00 seconds.

When there is motion detection, it sends 12 seconds of video from the detection to Cloud and at the same time moves 1 minute of video including that time to SD-Card. (when "SDカードへのローカル録画" is on and "録画モード" is set to "アラート検知時のみ")

[thematrixdev]

So one single motion may be recorded in two sequential files. It seems to be hard to use. But I think it is fine to have MediaPipe detecting from RTSP, and use the recorded file from SMB when needed. Thanks a lot.

[thematrixdev]

On Web-UI, NASモーション検知録画 does not create any file in SMB. Do I need to turn on 検出通知のローカル録画 as well?

[mnakada]

The current version requires both to be set to on.

The two settings exist for historical reasons, but they are not necessary. I will fix this in the next UPDATE.

[thematrixdev]

Now I have "transfer to SMB", "RTSP" and "PING" turned on. It seems the services are sometimes unstable. Sometimes I cannot read RTSP stream, and I cannot SSH to one of my cameras. I see you have mentioned it is not recommended to have RTPS turned on with other functions. Maybe could you publish a version with no logging at all, so to minimize the I/O load?

[mnakada]

I think the heaviest logging is writing to SD-Card. I think atomhack.log is once an hour, and healthcheck.log is only when a network error occurs. The network accesses in healthcheck.sh might be heavier. Is there any other logging that seems heavy?

RTSP, samba and NAS are heavy processes. Is there any of these that can be stopped?

For example, can you load video over http on a motion detection webhook event?

[thematrixdev]

By the way these days I have found one of my AtomSwing cameras reboots during during overnight when I am asleep. It has happened 3 to 4 times. The next day I have found the camera is reset to default position so I know that.

Below is the log showing the reboot.

Sat May 14 00:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      67028       7164          0        140      17852
-/+ buffers/cache:      49036      25156
Swap:       131068      14252     116816
tmpfs                    37096      4072     33024  11% /tmp
Sat May 14 01:10:01 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      62728      11464          0         28      12348
-/+ buffers/cache:      50352      23840
Swap:       131068      13436     117632
tmpfs                    37096      3992     33104  11% /tmp
Sat May 14 02:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      61632      12560          0         36      12508
-/+ buffers/cache:      49088      25104
Swap:       131068      13808     117260
tmpfs                    37096      4120     32976  11% /tmp
Sat May 14 03:10:01 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      60912      13280          0         36      12404
-/+ buffers/cache:      48472      25720
Swap:       131068      13708     117360
tmpfs                    37096      4240     32856  11% /tmp
Sat May 14 04:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      65708       8484          0         92      16856
-/+ buffers/cache:      48760      25432
Swap:       131068      13000     118068
tmpfs                    37096      4056     33040  11% /tmp
Sat May 14 05:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      62288      11904          0         40      12956
-/+ buffers/cache:      49292      24900
Swap:       131068      12980     118088
tmpfs                    37096      4280     32816  12% /tmp
Sat May 14 06:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      66012       8180          0        212      17012
-/+ buffers/cache:      48788      25404
Swap:       131068      12956     118112
tmpfs                    37096      4392     32704  12% /tmp
2022/05/14 07:02:08 : Reboot & Start watchdog
Sat May 14 07:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      70796       3396          0        536      22428
-/+ buffers/cache:      47832      26360
Swap:       131068        880     130188
tmpfs                    37096      2764     34332   7% /tmp
Sat May 14 08:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      70136       4056          0        660      21596
-/+ buffers/cache:      47880      26312
Swap:       131068        880     130188
tmpfs                    37096      2800     34296   8% /tmp
Sat May 14 09:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      70316       3876          0        660      21632
-/+ buffers/cache:      48024      26168
Swap:       131068        880     130188
tmpfs                    37096      2836     34260   8% /tmp
Sat May 14 10:10:00 JST 2022
             total       used       free     shared    buffers     cached
Mem:         74192      70260       3932          0        660      21676
-/+ buffers/cache:      47924      26268
Swap:       131068        880     130188
tmpfs                    37096      2872     34224   8% /tmp

I don't have schedule-reboot set on Web-UI. After reboot, "録画及びストレージ管理" is disabled in the app. I have to enable it again.

And it is strange that, 2 of my 3 AtomSwing cameras do not restore to a proper position after reboot. I have to manually point the camera to the position I want.

[thematrixdev]

I think the heaviest logging is writing to SD-Card. I think atomhack.log is once an hour, and healthcheck.log is only when a network error occurs. The network accesses in healthcheck.sh might be heavier. Is there any other logging that seems heavy?

RTSP, samba and NAS are heavy processes. Is there any of these that can be stopped?

For example, can you load video over http on a motion detection webhook event?

I am using RTSP together with SAMBA for the ease of doing least modification. Since those video in SAMBA are separeted into minutes, I have to pick the correct videos, merge, detect, trim and then send. If the resources problem cannot be solved easily I have to do in this way anyway.

[mnakada]

I don't have schedule-reboot set on Web-UI. After reboot, "録画及びストレージ管理" is disabled in the app. I have to enable it again.

And it is strange that, 2 of my 3 AtomSwing cameras do not restore to a proper position after reboot. I have to manually point the camera to the position I want.

Neither of them seem to be updating /atom/configs/.user_config properly, but I can't reproduce it at my place.

Maybe you need to rewrite the user_config in /scripts/webcmd.sh and then do the sync process. Can you confirm this by adding the following to line 111?

    echo 3 > /proc/sys/vm/drop_caches

[thematrixdev]

It becomes

if [ "$cmd" = "posrec" ]; then
    pos=`/scripts/cmd move`;
    awk '
    /slide_x/ {
      pan=POS;
      gsub(/ .*$/, "", pan);
      printf("slide_x=%d\n", int(pan * 100 + 0.5));
      next;
    }
    /slide_y/ {
      tilt=POS;
      gsub(/^.* /, "", tilt);
      printf("slide_y=%d\n", int(tilt * 100 + 0.5));
      next;
    }
    {
      print;
    }
    ' POS="$pos" /atom/configs/.user_config > /atom/configs/.user_config_new
    mv -f /atom/configs/.user_config_new /atom/configs/.user_config
    echo 3 > /proc/sys/vm/drop_caches
    echo "$cmd OK" >> /var/run/webres
    cmd=""
  fi

Is it correct?

What should I do after modifying this?

[mnakada]

Yes. Correct. I feel like maybe if I turn the power back on right after moving the camera direction it will reproduce. I would like confirmation that the problem will occur if not corrected, and that it is fixed when it is corrected.

[thematrixdev]

Do I need to save the settings on Atom App or Web UI again now?

[mnakada]

Check if the camera orientation is saved. Settings do not have to be made again.

[thematrixdev]

Okay I will keep an eye on it.

By the way, on recordEvent webhook request, may you add the saved filename please? Only the basename (yyyymmdd.mp4) would be fine. So I can use a script to pick the video file for processing. I have tried modifying myself but it is not correct:

if [ "$STORAGE_CIFS" = "on" -o "$STORAGE_CIFS" = "record" ] && /tmp/system/bin/mount_cifs && [ ! -f /tmp/disable_cifs ] ; then
      TIME=`echo $2 | sed -e 's|^/media/mmc/record/||' -e 's|/||g' -e 's|.mp4$||'`
      OUTFILE=`date -d $TIME + "/mnt/$HOSTNAME/record/$STORAGE_CIFS_PATH.mp4"`
      DIR_PATH=${OUTFILE%/*}
      mkdir -p $DIR_PATH
      cp $TMPFILE $OUTFILE

      if [ "$WEBHOOK" = "on" ] && [ "$WEBHOOK_URL" != "" ]; then
        if [ "$WEBHOOK_RECORD_EVENT" = "on" ]; then
          FILENAME=`echo $OUTFILE | sed -e 's|${DIR_PATH}||'`
          LD_LIBRARY_PATH=/tmp/system/lib:/usr/lib /tmp/system/lib/ld.so.1 /tmp/system/bin/curl -X POST -m 3 -H "Content-Type: application/json" -d "{\"type\":\"recordEvent\", \"device\":\"${HOSTNAME}\", \"file\": \"${FILENAME}\"}" $WEBHOOK_URL > /dev/null 2>&1
        fi
      fi
    fi

[mnakada]

How about the following code? I haven't checked it works.

/sceipts/mv.sh line72-

if [ "$FMT" != "" ]; then
  TMPFILE="/tmp/mv_`cat /proc/sys/kernel/random/uuid`"
  mv $1 $TMPFILE
  (
    TIME=`echo $2 | sed -e 's|^/media/mmc/record/||' -e 's|/||g' -e 's|.mp4$||'`
    FILE=`date -d $TIME +"$STORAGE_CIFS_PATH.mp4"`
    OUTFILE="/mnt/$HOSTNAME/record/$FILE"
    if [ "$STORAGE_CIFS" = "on" -o "$STORAGE_CIFS" = "record" ] && /tmp/system/bin/mount_cifs && [ ! -f /tmp/disable_cifs ] ; then
      DIR_PATH=${OUTFILE%/*}
      mkdir -p $DIR_PATH
      cp $TMPFILE $OUTFILE
    fi

    if [ "$STORAGE_SDCARD" = "on" -o "$STORAGE_SDCARD" = "record" ]; then
      /bin/busybox mv $TMPFILE $2 || /bin/busybox rm $TMPFILE
    else
      /bin/busybox rm $TMPFILE
      LD_LIBRARY_PATH=/tmp/system/lib:/usr/lib /tmp/system/lib/ld.so.1 /tmp/system/bin/find_libc /media/mmc/record -depth -type d -empty -delete
    fi
    if [ "$WEBHOOK" = "on" ] && [ "$WEBHOOK_URL" != "" ]; then
      if [ "$WEBHOOK_RECORD_EVENT" = "on" ]; then
        LD_LIBRARY_PATH=/tmp/system/lib:/usr/lib /tmp/system/lib/ld.so.1 /tmp/system/bin/curl -X POST -m 3 -H "Content-Type: application/json" -d "{\"type\":\"recordEvent\", \"device\":\"${HOSTNAME}\", \"file\":\"${FILE}\"}" $WEBHOOK_URL > /dev/null 2>&1
      fi
    fi
  ) &

[thematrixdev]

My AtomSwing camera still loses all the motion-detection and file-recording settings after reboot.

[mnakada]

I found the cause. I fixed it in Ver. 1.3.5.

[thematrixdev]

Updated. All of my AtomSwing cameras reboot and goes back to previous position. Thanks.

[thematrixdev]

I have written a script to merge, detect motion, trim and then send out the video. So I am turning off RTSP function to make sure the cameras are stable.

When someone is in front of the camera, the camera starts recording. If I open the Atom app and watch the camera, does it affect the recording process like RTSP? (I am not sure how the camera sends video to the app)

[mnakada]

Looking at the log, it looks like it is periodically throwing videos through the cloud. The upload doesn't seem to occur unless the app is open, so I think it's lighter than RTSP, but there is a load when the app is open.

[thematrixdev]

Do I need to enable anything so as to receive the image from 動体検知静止画転送?

[mnakada]

If "モーション検知" in Mobile App is on, you are good to go. It is sent by curl in /scripts/rm.sh:line 81. When the following message appears in /tmp/log/atom.log, rm.sh is executed.

 cmd:[rm -rf /tmp/alarm.jpg]

[thematrixdev]

Yes I have "motion detect" on Mobile App enabled. It seems I don't have any "image/jpeg" webhook triggered.

Below is my Python webhook script. Receiving "recordEvent" webhook works. I have been using it to receive webhook-video before. Just changed to image.

@app.route('/atom', methods=['GET', 'POST'] , strict_slashes=False)
async def atom():
    try:
        if flask.request.json and 'device' in flask.request.json:
            device = flask.request.json['device']
        elif flask.request.form and 'device' in flask.request.form:
            device = flask.request.form['device']
        else:
            return "", 500

        if flask.request.json and 'type' in flask.request.json:
            if flask.request.json['type'] == 'recordEvent' and 'cifsFile' in flask.request.json:
                print(device, 'VIDEO', flush=True)

                file_path = '/recordings/' + device + '/' + flask.request.json['cifsFile']

                db.session.add(AtomCamRecordingLog(
                    device_name=device,
                    time=int(time.time()),
                    file_path=file_path,
                    processed=0,
                ))
                db.session.commit()
        elif flask.request.form:
            if 'image' in flask.request.files:
                print(device, 'IMAGE', flush=True)

                now = datetime.datetime.now(pytz.timezone('Asia/Hong_Kong'))
                file_path = '/' + '/'.join(['recordings', device, str(now.year), str(now.month), str(now.day)])
                pathlib.Path(file_path).mkdir(parents=True, exist_ok=True)
                file_path = file_path + "/" + now.strftime("%H:%M:%S") + ".jpg"
                flask.request.files['image'].save(file_path)

                db.session.add(AtomCamImageLog(
                    device_name=device,
                    time=int(time.time()),
                    file_path=file_path,
                    processed=0,
                ))
                db.session.commit()
    except Exception as err:
        pass
    finally:
        db.session.close()
        engine_container.dispose()
        return "", 200

[mnakada]

OK. It's not working right. I will check it.

[mnakada]

I fixed it. It will be in the next UPDATE.

[thematrixdev]

I have updated v1.3.6 on AtomCam2 and AtomSwing. recordEvent was triggered. image/jpeg upload was not.

[mnakada]

I am able to receive it at my place. It's Node-RED though, not python.

I think the interpretation of the data received is different because image/jpeg is multi-part/form-data unlike other messages.

[thematrixdev]

Sorry my fault. I have got the uploaded image. I have found the image is taken so early that there is no person in it. I think I should enable "human detect" in Atom app.

Is it called "AI detection alarm" in the app now? I cannot find "human detection" or "pet detection" anymore...

[mnakada]

It's gone. But it wasn't working very well, so I guess it doesn't make much difference.