Lots of strange errors when migrating existing "Live Archive" assets

schuettecarsten commented 4 months ago

We have a bunch of "Live Archive" assets that we need to migrate before AMS is retired. They were all recorded by AMS using RTMP ingest during the last few years and can be played without any known issues. But migration fails with different errors on most of these assets.

For example:

[7:20:01 PM Fail] Failed to migrate asset c39a2b04-9bb1-4672-a279-d87f7c4fb3e2 Error:System.IO.InvalidDataException:
Unexpected, sampleDurationPresent must be present
   at AMSMigrate.Transform.TransMuxer.UpdateTrackRunDuration(String fileName) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\transform\TransMuxer.cs:line 256
   at AMSMigrate.Transform.BasePackager.DownloadAsync(String workingDirectory, String file, IList`1 tracks,
CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\transform\BasePackager.cs:line 295
   at AMSMigrate.Transform.BasePackager.<>c__DisplayClass66_0.<<DownloadInputsAsync>b__0>d.MoveNext() in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\transform\BasePackager.cs:line 215
--- End of stack trace from previous location ---
   at AMSMigrate.Transform.BasePackager.DownloadInputsAsync(String workingDirectory, CancellationToken
cancellationToken) in D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\transform\BasePackager.cs:line 216
   at AMSMigrate.Transform.PackageTransform.TransformAsync(AssetDetails details, ValueTuple`2 outputPath,
CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\transform\PackageTransform.cs:line 200

Some of them are complaining about gaps and missing chunks, log does not show on which asset which makes it difficult to figure out if this error will cause whe whole asset to fail:

[6:47:31 PM Warn] Missing Chunk at time 135611267260906 for stream video_550000. Ignoring gap by skipping to next.
[6:47:31 PM Fail] Failed to download chunk video_550000/135611267260906 for live stream: video_550000. Error:
System.IO.InvalidDataException: Unexpected missing chunk indicated by manifest is currently not supported
   at AMSMigrate.Pipes.MultiFileStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\pipes\MultiFileStream.cs:line 102
[6:47:40 PM Warn] Missing Chunk at time 135611267440906 for stream video_850000. Ignoring gap by skipping to next.
[6:47:41 PM Fail] Failed to download chunk video_850000/135611267440906 for live stream: video_850000. Error:
System.IO.InvalidDataException: Unexpected missing chunk indicated by manifest is currently not supported
   at AMSMigrate.Pipes.MultiFileStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\pipes\MultiFileStream.cs:line 102
[6:47:54 PM Warn] Missing Chunk at time 135611267440906 for stream video_1350000. Ignoring gap by skipping to next.
[6:47:54 PM Fail] Failed to download chunk video_1350000/135611267440906 for live stream: video_1350000. Error:
System.IO.InvalidDataException: Unexpected missing chunk indicated by manifest is currently not supported
   at AMSMigrate.Pipes.MultiFileStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\pipes\MultiFileStream.cs:line 102
[6:47:57 PM Warn] Missing Chunk at time 135611267620906 for stream video_2200000. Ignoring gap by skipping to next.
[6:47:57 PM Fail] Failed to download chunk video_2200000/135611267620906 for live stream: video_2200000. Error:
System.IO.InvalidDataException: Unexpected missing chunk indicated by manifest is currently not supported
   at AMSMigrate.Pipes.MultiFileStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\pipes\MultiFileStream.cs:line 102
[6:49:01 PM Warn] Missing Chunk at time 135611267440906 for stream video_3500000. Ignoring gap by skipping to next.
[6:49:01 PM Fail] Failed to download chunk video_3500000/135611267440906 for live stream: video_3500000. Error:
System.IO.InvalidDataException: Unexpected missing chunk indicated by manifest is currently not supported
   at AMSMigrate.Pipes.MultiFileStream.DownloadAsync(Stream stream, CancellationToken cancellationToken) in
D:\NoAV\Tools\Microsoft\azure-media-migration\migrationTool\pipes\MultiFileStream.cs:line 102

Any ideas how to mitigate these errors? Let me know if you need sample assets.

pohhsu commented 4 months ago

@schuettecarsten,

let me add some initial comments first to explain the issues.

Unexpected, sampleDurationPresent must be present Currently we expect sampleDuration to be present in the trun box, I think some of your contents doesn't have that so it failed.
Unexpected missing chunk indicated by manifest is currently not supported Currently we expect if a chunk is specified by manifest then it must exist. The reason is we have to determine whether we need to do some form of healing (e.g. insert silence for missing audio) by analyzing the manifest without having to download and check each fragments.

unfortunately, in both of these case, we're not sure if there are easy/good solutions right now...
cc: @duggaraju, @weibz

schuettecarsten commented 4 months ago

Unexpected, sampleDurationPresent must be present Currently we expect sampleDuration to be present in the trun box, I think some of your contents doesn't have that so it failed.

I've looked into the source code (TransMuxer.cs, line 254). For me it looks like the exception occurs when the migrator tries to fix issues with the stream length, which requires sampleDuration for calculation. I think that the exception is not neccessary here, a warning that the stream cannot be checked is enough - and of course only do the checks when possible?

pohhsu commented 4 months ago

hi @schuettecarsten,

@duggaraju gave a great suggestion on issue 1.

First one looks like a bug in our code? Are we assuming that a 'trun' box is always present even if all samples have same duration and the duration is specified in the tfhd box?

so maybe this can be fixed easily...

i'm wondering if you can provide sample for issue 1 to check this?

schuettecarsten commented 4 months ago

@pohhsu @duggaraju

i'm wondering if you can provide sample for issue 1 to check this?

Sure, but I cannot post this here - please drop me an email schuettecarsten at googlemail.com and I will send you a SAS link to the asset.

pohhsu commented 4 months ago

Also, if you are interested, I put up a draft PR to address the ‘Unexpected, sampleDurationPresent must be present‘ issue but I couldn’t test it out.

https://github.com/Azure/azure-media-migration/compare/main...pohhsu/test_tfhd

schuettecarsten commented 4 months ago

Also, if you are interested, I put up a draft PR to address the ‘Unexpected, sampleDurationPresent must be present‘ issue but I couldn’t test it out.

main...pohhsu/test_tfhd

@pohhsu @duggaraju I've send you an email with some sample assets.

The issue with the missing sampleDurationPresent is fixed with the draft PR and the resulting asset looks good.

For the missing chunks issue, a lot of our assets are affected by this and we need to migrate them. So, maybe it's an option to simply ignore the gaps, add black there or repeat the previous chunk?

Azure / azure-media-migration

Lots of strange errors when migrating existing "Live Archive" assets #251