Extracting Frame by Frame

[rootninja]

Tip of the hat to you as this is a FANTASTIC tool! I've a drive in hand that was pulled during operation however have confirmed the last append to the .ubv file was written to less +/- 59 seconds prior to physical removal. As a result, a large portion of the recordings are not extracted even though the cam was set for Full Time recording at 30 fps. Would it be possible to extract each frame/still from a single 1GB ubv file as the data is extremely crucial, not worried about sound at this point. Log file attached and due to the sensitivity of this am willing to compensate as well. I've tried with and without each combination of the flags available however still only exports the mp4 (with -mp4=false) Thank you!

logs_20230511.log

[petergeneric]

Can you capture the output of the following command and attach it to this ticket to see what frames the .ubv file does have in an easily extractable form?

ubnt_ubvinfo -P -H -s -f file.ubv 2>&1 | tee all-frames.log

The output of this command will be large - it's essentially position data on each frame of video available in the .ubv. It's all generic metadata, there's no personally identifying information in the output. The -H arg requests a human-readable timestamp for each frame, and you should be able to use that to work out from that whether there are video frames near to the time of physical removal. Alternatively, if you attach that log to this ticket + tell me the time of removal, I can have a look instead if you'd prefer.

If there aren't any frames with timestamps near to the time of physical removal then it may be that the data either wasn't written, or that the data was written but some headers are missing making it inaccessible. If the latter then you may be better talking to a specialist data forensics company, since they should have tooling to examine a file and extract any sections that look like H.264 bitstreams (the .ubv has the raw AAC and H.264 bitstreams in plaintext, it's just an unusual container format).

P.S. Ubiquiti have started shipping their own UBV -> MP4 tool with recent Protect releases. I have not fully investigated it but I think it would be worth running it to see if it can extract any more data than my tool does. It's used as follows (if you don't currently have access to working Ubiquiti hardware to run that tool you can run it using the same qemu wrapper as I suggest in the readme for ubnt_ubvinfo). If you do find it manages to extract more data than my tool please let me know -- so far with my corrupt sample files I've found it doesn't export as much data as my software, but if this has changed I'd be quite interested.

/usr/share/unifi-protect/app/node_modules/.bin/ubnt_ubvexport -s your-file.ubv -d my_output_prefix

[rootninja]

Thank you! I have attached the log tarred up (4.5MB extracted) and drive was physically removed at 17:15 8.1.23 however last time stamp on file was 17:14. I do have a new DMP that I can offline restore and insert offline cloned drive all-frames.tar.gz

Also feel free to drop me an email to my user at protonmail and let me know where I can comp you for your time on this, super appreciate your repo and willingness to help!

[rootninja]

Restored the DMP offline and ran the above which produced 33 mp4 (including one of 0 bytes) files vs the original 26 from your repo however total size of exported files held a delta of only 0.2MB (in favor of ubnt_ubvexport).

For good measure ran this the above mentioned tool against all 10 available backup files just because; all of which produced the same results unfortunately. The corresponding log attached is from only 1 of the 10 as the results were all identical. Prefix used (as part of -d) is in reference to the UniFi OS Backup file used relative to backup date i.e. AUG03BKP represents the auto backup file taken by UniFi on that date.
ubnt_ubvexport.log

[petergeneric]

From the ubnt_ubvinfo log you provided, it appears that there's video frame data available up to 17:13:48, although 17:13:44 is the last keyframe, so footage after that point may be garbled (I'm assuming you're in/around the PST timezone).

Based on that log file, both remux and ubnt_ubvexport should have been able to extract all the way up to that final set of frames in the log, and av+header data is interleaved in .ubv, so the fact that the headers for thoe frames have been written should mean the data was successfully written...

So, a few questions for you:

1. What's the date+time of the last frame the tools have been able to extract video for?
2. Was this a wifi camera?
3. Can you confirm the version number output by ffmpeg -v? Just the major.minor output on the first line
4. What model was the camera?

I'm somewhat confused by the number of timecode discontinuities in the output, that usually indicates something was malfunctioning (and the Application provided duration: -9223372036854775808 / timestamp: -9223372036854775808 is out of range for mov/mp4 format error repeating for each remux further adds to that mystery).

As a sanity check, I think it would be a good idea to run remux --mp4=false file.ubv - this should output a series of .h264 files containing the raw video bitstream within the .ubv but before they were processed by FFmpeg to create an MP4. If you can then compare the sizes of the .h264 files against the .mp4 files, that should give an indication of whether FFmpeg ran into issues during the MP4 creation stage (the MP4 might be up to 1-2MB larger than the .h264, it should never be smaller). Can you also confirm the version number output by ffmpeg -v?

[rootninja]

Thanks for your response:

1. 19:13 was last extracted however the important events prior to are missing yet some are not (camera set to FT record @30FPS)
2. Wired on dedicated vlan (G3 Dome)
3. ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
4. G3 Dome

I too am confused over the timecode as well and am wondering if I shouldn't use a portion of the file header from another intact UBV file and dd it over to the UBV that was being written to while in operation?

I've tried the --mp4=false however mp4 files are the only output which too has me confused but could be due to the file being written to while in operation? I've a VM up and running in the cloud running Ubuntu but have also attempted this on local hardware both on and off the DMP from the restored backup. Thanks again for your time, want to make this worth your while.

[petergeneric]

That's strange, did you add the --mp4=false argument before the .ubv filename?

Do you know the dates+times you're interested in footage from? We can see from the log whether there are video frames covering that time period (if you can give the dates+times in UTC that would be helpful, otherwise let me know the timezone too)

I worry that the camera was experiencing issues prior to removal - generally large numbers of discontinuities in the feed means that the camera was misbehaving and the Protect system didn't receive any frames for a time period from the camera. I've seen this happen on cameras periodically, a camera reboot is usually required to get them back into normal operation. Based on your ubnt_ubvinfo file, the 1GB file was only 1/3rd full, having around 335MB of video+audio data

Edit: P.S. due to the way ubv files are structured, moving a header from an existing file wouldn't help: data within a UBV file are structured as a series of [length][timecode info][h264 or aac bitstream] elements, so the metadata is closely interleaved with the video/audio bitstream, rather than in a different position in the file.

[petergeneric]

Also, on compensation: I'd usually charge $200 for investigation of this level, I'd really appreciate if you could donate that amount to help Ukraine at https://u24.gov.ua/ They have a variety of different funds so that you can ensure your donation goes to an area you care about. Defence is vitally important, of course, but Medical Aid is also of critical importance.

[rootninja]

Consider it done! Would you prefer I donate to Defense or Medical? Say the word and I'll send now.

[petergeneric]

Defense preferable, thank you!

[rootninja]

To answer your question of: That's strange, did you add the --mp4=false argument before the .ubv filename?

Yes, I did both pre and post the ubv file to rule out any syntax error on my part. I've started to tinker with using a "good" UBV file and using dd to see if I cannot reconstruct portions of the UBV file that was in operation while drive removed. Sending donation to defense now in your github name as I type this

[rootninja]

Donated :) 205 (5 to cover transaction costs and be glad to send you copy)

[petergeneric]

Thanks!

The fact that you're not getting the .h264 files out is very odd - I am just producing a special build of the tool that will only output .h264 files, should have it ready shortly

[petergeneric]

I've uploaded a custom version to https://github.com/petergeneric/unifi-protect-remux/releases/tag/v0.0.0-custom.1 - used as demux file.ubv. It should only output .h264 files, and it should explain timecode information for each one prior to extraction. Let me know what that's able to output - if it still doesn't output anything it might be worth exploring having it ignore partitions and discontinuities (although I think that would result in garbled video at best)

[rootninja]

Same number of output files (26) and downloading now to review. I think you may be onto something via ignoring partitions :+1: Log file attached from demux as well demux_.log

[rootninja]

Thanks for uploading the custom demux! After reviewing all the h264 files they are the same as the mp4 outputs so perhaps your earlier idea as to exploring ignoring partitions is best?

[petergeneric]

Ok, looking at the "Frames:" parts of the output of demux and summing it up, the figure comes to 34650, which is the same as the number of video frames that ubnt_ubvinfo is reporting (and comes to about 19 minutes of video at 30fps).

I don't see any major gaps in offsets reported within all-frames.log file, so at this point I'm pretty sure that the video frames that aren't in the .h264 files weren't recorded by Protect (or weren't successfully delivered by the camera).

It might be an idea to try the equivalent _2_rotating_ file and see if any of the lower-resolution proxy frames for the missing time period are present?

[rootninja]

All files created that day (and all previous days) all contain the MACADDRESS_0_rotating format without an increment in the center number to 1, 2 and so on. My best guess is protect appends to the UBV file until the file is full and rotates to create a new file. Confirming with higher frame rate cameras on premise generate more 1GB UBV files than the lower frame rate cameras all of which follow the same naming convention mentioned above which is suffixed with a random number I am not able to decipher.

[petergeneric]

Yes that's correct - the _0_rotating files are full-quality, the _2_rotating files are low-res browse proxy files (used when you're scrubbing through video in the app, so the image quality isn't great but it's still worth checking in case you can get anything useful). I forget what the _1_rotating files hold. All files rotate once they hit 1GB. The large number at the end is a millisecond timestamp (unixtime with millisecond precision)

So I'd try running remux against those MACADDRESS_2_rotating_*.ubv files - it's possible that whatever camera/network issue was causing the discontinuities in the _0_rotating file didn't impact it - I think that's the last chance to salvage footage for the time period(s) you need

[petergeneric]

Were you able to extract any additional footage from the _2_rotating files?

[rootninja]

I had to run through the testdisk suite to recovery the _2_rotating and just recovered two of which were staggered 3 days apart. Running all variants of both revisions of your tool (thank you again btw) and for good measure the native ubnt tool and will report back shortly. Still curious, due to your notes on the camera issues relative to frames, I'm hopeful you can point me in the right direction for addressing the partition issues however I've yet to dive into comparing the file headers of the UBV file (from intact UBV files from same camera) in contrast to the potentially damaged UBV file from the drive being pulled while in operation. It's been a task to recover these files but will report back as soon as these results are in. Thanks again!

[rootninja]

Results are sadly the same as prior to across all three methods of the native ubnt_ubvexport and both variants of your tools. Partitions? In the meantime I am going to attempt to peer into the files with a hexeditor by contrasting the good intact ubv file from the same camera but am hopeful you'll have an idea for the partitioning you mentioned earlier.

[petergeneric]

So to temper your expectations - I don't think the frames are in there, based on the ubvinfo output you provided. If there were additional video essence packets present but unindexed then you'd be able to see them because the OFFSET col values from ubnt_ubvinfo would show a big discontinuity, but the discontinuities are only the size of the packet headers (and the sum of the size col is consistent with the amount of video you got out). My understanding of what they do is: open the file, set the size to 1GB (so the end of the file will be all zeroes), open a partition and then write packets to it sequentially. if opening an existing file, they close off the current partition and start a new one (pretty sure they also do this in the event of a timecode discontinuity).

However, the format is reasonably simple, I did a bit of work reverse engineering it and had a half-documented grammar built in Synalyze It (unfortunately since lost)... all lengths are stored as simple 64-bit ints, the structure is roughly:

UBV [length of header or maybe start offset of partition 1] [ubv header data]
0..n Partition [length in bytes]
    0..n Packet [ length in bytes] [frame header] [frame data]
    Seeking index structure (if memory serves)

The seeking index structure I didn't look into, but I'm pretty sure it will be a simple table of keyframe timecodes and file positions (or possibly partition-relative offsets) for that packet - it's quite small so can't contain actual AV essence.

Pretty sure the header structures were all fixed-lengths (albeit different per header) I don't recall whether they're included in the [length in bytes] totals. A packet describes the track number (7 for video, 1000 for audio) and the timecode (samples and sample rate, pretty sure everything uses a fixed sample rate of 90k but I could be misremembering).

If I recall correctly, any length in bytes value is aligned to 4 byte (I assume for simplicity so they can mmap the .ubv). Within the frame, the data segment for a packet is either a raw AAC bitstream, or for H.264 it's encoded as [NAL Length] [data] (I convert these to [NAL Separator] [data], see /demux/demux.go:68.