When trying to export recordings they seem missing, but appear in Smart Client

[JaumeSEneTelecom]

Our company oppened a ticket about a month and a half ago with the account of @MontseMunozFaus for an issue with the codec.

During the last weeks we have been trying to export video recordings to mkv from 24/7 recording cameras.

The problem is that there are many days that haven't exported anything and since it worked the first days we don't know what could have caused that the script stopped working.

We use codec H.265 and with the new module (MilestonePSTools 23.3.1) the exportation process worked fine at the beginning, but after some days now we aren't able to export a single recording even those one from yesterday.

Note that we user a software called Tiger to upload the recordings to AWS immediate tier, but they are still reachable (it has been working the entire time). Its just that after changing the MilestonePSTools module for the new version 23.3.1 it worked for like 3 or 4 days and then it is not exporting anymore. Remember that if we enter from the smart client the recordings appear but it looks like from MilestonePSTools we are not able to export them.

If you need any type of additional information please request it.

Thanks,

[Tarterman]

What product (Corporate, Expert, etc.) of Milestone are you connecting to and what version? Also, I don't know if it will help, but MilestonePSTools 23.3.2 is the latest version. I don't recall what was changed so it probably won't affect what you are doing but it would be worth upgrading.

Do you know what version of MilestonePSTools you were using before upgrading to 23.3.1? If so, try using that version again just to verify whether upgrading to 23.3.1 is actually what caused the change or if it is something else.

If you aren't sure what version you were using, you can check the install location for MilestonePSTools which will be in one of two locations:

C:\Users\\Documents\WindowsPowerShell\Modules\MilestonePSTools C:\Program Files\WindowsPowerShell\Modules\MilestonePSTools

For example, I can see on one of my systems (which also hasn't been upgraded to 23.3.2 yet) that I was using 23.2.3 before upgrading.

With that knowledge, I can open a fresh PowerShell session and load the old version and then run whatever script I want. For example, to load 23.2.3, I would do this:

Import-Module -Name MilestonePSTools -RequiredVersion 23.2.3

Beings that it worked for a few days after upgrading, I'm wondering if there is something else contributing. Also, you mention exporting in MKV. If you try exporting in XProtect format, does it work fine?

[JaumeSEneTelecom]

Hello,

Anwsering your questions:

• The product is XProtect Corporate 2023 R2
• Previously we used version 23.2.1 of MilestonePSTools Module, but we had to change it because we couldn't export recordings in codec H.265
• From the smart client exporting in XProtect format is also working

Here is some additional information that might help:

• As mentioned previously we are using the Tiger plugin to use an immediate tier storage from AWS to store the recordings so we are exporting them from there
• Also we have the export sample created by you, and after trying to do an exportation with it, it worked. So right now I am a bit lost since if your sample works our script should also work.

This is the line of code that executes the export just in case something is missing: Start-Export -CameraIds $cameraID -StartTime $startTime -EndTime $endTime -Path $path -Name $exportFileName -Format mkv -ErrorAction Stop

[JaumeSEneTelecom]

Are there any news or something that we can try? commands to see if configurations are correctly or whatever could be causing the error.

[Tarterman]

In regards to exporting in XProtect format, I was more wondering if exporting in XProtect format using PSTools works or if it fails for both XProtect format and MKV format.

I'd also be interested if adding -Verbose and/or -Debug to the end of your command returns anything that might be useful.

[joshooaj]

Hi @JaumeSEneTelecom, sorry I didn't have a moment to spare last week to reach out for a remote session. I'd love to connect with you this week - can you propose a time and we'll send a Teams invite?

I know we may need to be flexible on timezones but for reference I am in the US on the west coast where the offset is GMT-7.

[MontseMunozFaus]

Hi @joshooaj , we've checked with @JaumeSEneTelecom agenda. Our time is GMT+2. Please would it be suitable for you to meet at 9:00am your time? Available days: this Friday or next week any day except Monday. Thank you very much

[joshooaj]

Sounds good, let's do Tuesday at 9am Pacific - I'll send a Teams invite separately!

[JaumeSEneTelecom]

Josh, we have news, The client told us that they changed the cameras in dublin to codec H.265, but after revisisng we've seen that they were not changed. So it looks like the problem is with the codec H.265 Avigilon cameras.

[joshooaj]

Thank you for the update! I am working on tracking down an engineer that can help me understand why we would get this error in PowerShell and not the MIP SDK ExportSample UI.

Are you familiar enough with Wireshark to capture a trace of the failing export request in PowerShell and another trace of a successful request in the ExportSample UI? The successful request should probably be stopped as soon as the export looks like it's started so that we don't end up with a giant PCAP file.

If this is something we can do but you'd like a hand to capture the Wireshark trace, let me know when you'd be free for another remote session.

[JaumeSEneTelecom]

I am not very familiar with wireshark, so we can schedule a meeting. The only issue is that I'm not sure if we can install wireshark in the server or if there are any firewall rules that could affect what we are able to see in the trace of requests. I will ask our client. Regarding the meeting, you can choose the time, I'm available today or next week. The only thing is that it should not be later than 10:00 your time (if it is impossible for you maybe I can a bit later).

[joshooaj]

Invitation sent!

[JaumeSEneTelecom]

Hello Josh,

Instead of tomorrow, can we do the meeting on thursday? since our client won't have Wireshark installed in the machine until then. It can be the same time or maybe its better for us bit later.

Regards and sorry for the inconvenience,

Jaume

Sensitive information removed by Tarterman

[joshooaj]

Can you confirm whether any cumulative patches have been applied? Looking at the logs from the diagnostics tool, I don't see that any patches have been installed. Are you able to apply the latest patches to see if it resolves the issue? Unfortunately I can't say whether I expect to see difference, but there is a note about a fix for an H.265 decoding issue and as you'll see in the article below, there are a lot of fixes available for 2023 R2.

https://supportcommunity.milestonesys.com/s/article/XProtect-2023-R2-cumulative-patch-installers?language=en_US

Looking at the Wireshark traces, so far I see the following pattern for the ExportSample UI:

1. Request connection to the specific camera database on the recording server
2. Response indicates success
3. Request to "goto" the desired timestamp
4. Response includes binary video data
5. Request for next chunk of data

The PowerShell export request looks exactly the same, except there's no "step 5: request next chunk of data". It's like the data received was somehow invalid or did not actually make it all the way up through the network stack back to the MilestonePSTools PowerShell session. I still can't quite explain it.

My initial instinct is to look at antivirus / endpoint protection software if present, and see if it could be interfering at all with PowerShell as it's not uncommon to see that, but if that was it I wouldn't expect to see a difference between an H.264 export and H.265 export.

[joshooaj]

There was a new release of the VideoOS.Platform.SDK DLL a couple days ago resolving a video rendering/decoding issue reported by another customer, so MilestonePSTools v23.3.3 has been released with that version of the DLL.

I do recommend you install the cumulative patches for your Milestone servers, but you might try MilestonePSTools v23.3.3 first to see if that resolves the issue.

[joshooaj]

Hi @JaumeSEneTelecom, it seems like I can reproduce this on my lab system now. Here's what I'm seeing:

✅ Export H.264 from PowerShell on virtual machine ✅ Export H.265 from ExportSample on virtual machine ✅ Export H.265 from PowerShell on laptop

❌Export H.265 from PowerShell on virtual machine

I'm not entirely certain where the issue is yet and I will be asking our MIP team to look at this with me, but until then, is there a chance you can attempt to export from a "bare metal" host?

[JaumeSEneTelecom]

Sorry for the delay Josh, we had a 3 day weekend and were not able to answer.

Responding to your questions:

• No, the cumulative patches have not been applied, but we will look into it and try to update the XProtect, also we are going to update the MilestonePSTools module to version 23.3.3.
• Regarding the "bare metal" host, I don't think it will be possible, maybe we could do some test but not have it as a final solution.

Thanks for all the news, it looks like we are going in the right direction, if something works I will tell you.

[joshooaj]

Regarding the "bare metal" host, I don't think it will be possible, maybe we could do some test but not have it as a final solution.

Thanks, I definitely wouldn't expect this to be a final solution - more of a stop-gap to enable you to complete the required exports before the data is moved to the glacier tier if necessary. I'll keep you posted on our efforts to debug the issue on our end.

[JaumeSEneTelecom]

Thanks, I hope that today we can update the XProtect and the MilestonePSTools module, I will let you know if something works.

[joshooaj]

Hi @JaumeSEneTelecom, good news! I've been able to track down the reason this fails in PowerShell but not in the ExportSample UI and I have a temporary fix for you to try. Either manually copy the files described below, or run the script that will locate and copy them for you. Please let me know how this works out for you as soon as possible please, and thank you!

Temporary Fix

1. Locate the subfolder bin\15dd936825ad475ea34e35f3f54217a6 which is found in the MilestonePSTools module directory. Typically this will be at a path like C:\Users\jhadmin\Documents\WindowsPowerShell\Modules\MilestonePSTools\23.3.3\ or C:\Program Files\WindowsPowerShell\Modules\MilestonePSTools\23.3.3\ depending on if MilestonePSTools is installed for the current user, or for all users.
2. Copy the whole 15dd936825ad475ea34e35f3f54217a6 folder to C:\Windows\System32\WindowsPowerShell\v1.0 so that the file mfxplugin64_hevcd_sw.dll can be found at C:\Windows\System32\WindowsPowerShell\v1.0\15dd936825ad475ea34e35f3f54217a6\mfxplugin64_hevcd_sw.dll and the corresponding plugin.cfg file is present in the same folder.

Import-Module MilestonePSTools
$modulePath = ([io.fileinfo](Get-Module MilestonePSTools).Path).DirectoryName
$hevcLibraryPath = Join-Path $modulePath 'bin\15dd936825ad475ea34e35f3f54217a6'
Copy-Item $hevcLibraryPath C:\Windows\System32\WindowsPowerShell\v1.0\ -Recurse

Root Cause

The file mfxplugin64_hevcd_sw.dll is the Intel HEVC / H.265 software decoding library and is used when hardware acceleration is unavailable. The MIP SDK locates this file by looking for the folder 15dd936825ad475ea34e35f3f54217a6 as a subfolder of the currently executing application.

For a standalone application like the ExportSample, the EXE file and this subfolder are both in the same directory. But a PowerShell module like MilestonePSTools runs inside the PowerShell process, and in the case of Windows PowerShell 5.1, that location is C:\Windows\System32\WindowsPowerShell\v1.0\.

The MIP SDK uses AppDomain.CurrentDomain.BaseDirectory to determine the "base path" to look for the Intel software decoding library. In PowerShell, you can see the value of this with [AppDomain]::CurrentDomain.BaseDirectory.

Long-term Solution

I don't have an answer for you yet on a long-term fix for this as I only just discovered it. Maybe the logic in our MIP SDK can be updated to search for this DLL under the file path for one of the other MIP SDK assemblies/dlls, or maybe we can provide our own "base path" as an argument in a future release - in which case we would update the PowerShell module to set that path automatically. I'll keep you posted as I learn more.

[joshooaj]

Hi @JaumeSEneTelecom,

I think I have a better temporary fix. It would appear that the Intel MediaSDK library also looks for a file named mfxplugin64_sw.dll in the same directory as the MIP SDK. When I copy 15dd936825ad475ea34e35f3f54217a6\mfxplugin64_hevcd_sw.dll to the bin folder containing the rest of the MIP SDK dll's, I'm able view and export video from PowerShell without adding or modifying files in the C:\Windows directory.

I'm going to chat a bit more with some colleagues in Denmark to see if this is a safe solution. If so, we'll release an updated version of MilestonePSTools with that file already in place. Hopefully this fix works for you until then. Here's a script to find and copy the dll for you, as well as clean up the folder from the previous recommendation if present.

# Delete the folder from the previous recommendation if present
$folderToDelete = Join-Path ([appdomain]::CurrentDomain.BaseDirectory) '15dd936825ad475ea34e35f3f54217a6'
Remove-Item $folderToDelete -Recurse -Force -ErrorAction SilentlyContinue

Import-Module MilestonePSTools
$modulePath = ([io.fileinfo](Get-Module MilestonePSTools).Path).DirectoryName
$splat = @{
  Path = Join-Path $modulePath 'bin\15dd936825ad475ea34e35f3f54217a6\mfxplugin64_hevcd_sw.dll'
  Destination = Join-Path $modulePath bin\mfxplugin64_sw.dll
}
Copy-Item @splat

[JaumeSEneTelecom]

Thanks Josh, today we are going to implement these changes, I will let you know if it works for us. Thanks again

[joshooaj]

Thanks Josh, today we are going to implement these changes, I will let you know if it works for us. Thanks again

Hi @JaumeSEneTelecom, just checking in - how'd it go?

[JaumeSEneTelecom]

We tested during this weekend and found an error, but it turned out to be something from amazon, so this morining I checked and it looks like it has been working fine.

We will stay tunned in case you release a module that doesn't require to move any folders.

Thanks for everything.

[joshooaj]

That's great to hear! MilestonePSTools version 23.3.4 is available on PowerShell Gallery now with this fix.

[JaumeSEneTelecom]

Perfect we will try it in the next meeting with our client to see if it works as expected.

Thank you very much.

[JaumeSEneTelecom]

Hi Josh,

When we tried to export the files recorded when the exporting started to fail (like at february 27 the recordings from 12 to 4 AM we were able to export them, but the next 4 hour time frame didn't work) we noticed that during the time frame when the issue with the exportation started, the recording of all cameras within that time frame now have a size of about 40 GB Which in some cases is about 50 times more than the regular size of the camera.

Just to clarify if I did not explained myself properly, I would like to know what could have made the files to increase their size about 50 reaching 40GB per 4 hours of recordings, since these files are only the ones from the time frame when the issue started probably it is related. I don't know if you have any idea of what could it be or if we should open a ticket with milestone directly.

If it can be fixed it would be better, but since it is not so critical just by knowing the cause so we can avoid it is what our client would want to know.

Thanks,

[JaumeSEneTelecom]

I will add a screenshot so maybe it is easier to understand.

[joshooaj]

The answer should be "yes" but I'll ask just to be sure - does an MKV export from Smart Client also show the same issue with the large file size?

If so, my guess is that the codec was changed in XProtect. Maybe for testing purposes the codec was changed, or maybe it was accidentally changed to MJPEG?

If you have done any codec configuration changes from PowerShell, it's also possible that because of a bug in the Configuration Api related to setting validation, someone changed the codec to "H265" when the Management Server expected "h265". Because the values match when you ignore case, the Management Server may accept the value but the Recording Server fails to map it to an available codec and defaults to MJPEG. When that happens, the Management Client shows the expected "H.265" codec but the actual codec received is MJPEG.

These are just some guesses since MilestonePSTools doesn't do any video manipulation. It uses the MKVExporter class in our SDK, and that class also shouldn't be manipulating the video. It should be taking the raw video data received from the Recording Server and storing it in an MKV "container". So these should be the actual recordings from February made by the Recording Server.

You can explore this a bit further on your own if you like, and then if you need to dig deeper you can contact our tech support team. They may not be able to help much if there are no logs available for the relevant time periods though.

• In Smart Client, enable the Video Diagnostic Overlay in Settings / Advanced. Then playback the video for that time period. Check the codec reported by Smart Client, and if it's H264/H265, check the frame rate and "GOP" size. Normally the GOP or "Group of Pictures" size/length should match the frame rate, resulting in one "GOP" per second, or one "key frame" per second. With "smart" codecs, the GOP might also be dynamic, resulting in a much longer GOP size - one every 10-20 seconds for example in an idle scene. If the GOP is really short like "1", then that could account for the significant increase in size, but I couldn't tell you why the GOP was that short if that is the case.
• Examine the MKV file with VLC media player or the ffmpeg / ffprobe command-line tools to extract information about the video stream. It should hopefully tell you the codec, frame rate, and duration. Double check that the duration is what you expected.

[Tarterman]

I'll jump in with one more thought. Are you doing any privacy masking on the cameras that are creating large export file sizes? This would include privacy masking that is configured in the Management Client and privacy masking that you configure when doing an export. The reason I bring this up is that, if there is a privacy mask, the video is converted to MJPEG which will drastically increase the file size.

[JaumeSEneTelecom]

After checking the recordings from the smart client, during that time frame there was a change in codec. So that would be the reason?

[Tarterman]

If the codec was changed to MJPEG, that would definitely explain the large difference in file sizes.

[JaumeSEneTelecom]

Well, the change was from H.264 to H.265 also I saw that the GOP size was about 7-8 all the time and the framerate 30

[Tarterman]

Hmm, if anything, changing from H.264 to H.265 would decrease the file size. The 7-8 second GOP size on a 30fps clip would increase the file size but I wouldn't expect as big of an increase as you are seeing in most situations. There could be some edge situations though.

Were you able to verify that there wasn't any sort of privacy masking on the camera? When exporting in our XProtect format or MKV format, we don't compress, re-encode, etc. the video at all unless there is a privacy mask.

[JaumeSEneTelecom]

Yes, I verified the privacy mask, but there are no cameras woth privacy masks. Anyway, since it is a very strage event, don't worry to much, I have talked to the client and they understood that it was not normal.

Thanks for your time and help guys

[joshooaj]

Cheers @JaumeSEneTelecom, I’m happy to see we’ve reached a solution finally - sorry that it took so long!

Keep in touch and let us know if you have any new issues or feedback for us or the module.