System. AccessionViolationException when ShowFramePrev

[void-soul]

I have integrated a MIDI device into the software, which has a scroll wheel, When the roller rotates in the forward direction, it maps to ShowFrameNext of FlyLeaf When the roller rotates in reverse, it maps to the SeekBackward of FlyLeaf (because ShowFramePrev is slow, I have learned the reason behind it through other issues) Current issues: When the roller rotates rapidly (10 times per second), FlyLeaf will crash because: An unhandled exception of type 'System. AccessionViolationException' occurred in FlyeafLib.dll Attempted to read or write protected memory This is often an indication that other memory is corrupt This error can occur in multiple files, such as: Demuxer.cs Line 1531 Ret=avunread_frame (fmtCtx, packet); VideoDecoder.cs 1006 Av_packet_unref (demuxer. packet); My video is 4K 120FPS, which may also be one of the reasons for the crash. I can set the frequency for the scroll wheel, but I have tested it multiple times and even if it is set to 500MS, it still prompts this error --

[SuRGeoNix]

My understanding is that your issue is not with ShowFramePrev (as you don't use it) but with ShowFrameNext, right?

From a quick look it seems that as you don't use ShowFramePrev the combination between SeekBackward / ShowFrameNext causes the issue as they don't use the same lock (seeks / lockActions).

The other (performance) issue that I see, is that probably you mess with HEVC that FFmpeg fails to seek on keyframe and makes it much more slower.

Generally, there is a lot of space for improvements for Frame Stepping / Reverse Playback etc.. but it will require a lot of recoding (already included in v4 but still needs time to be completed). It will use a two way packet queue and two way frame queue.

Another issue with the current implementation I think is that it will not cancel a Frame Stepping request so you will have to wait to finish the previous one. I will try to do some testing and see if I can improve the current implementation but if it needs a lot of work I will leave this for v4.

[void-soul]

Perhaps as you said, there is a performance issue with my HEVC encoding. I use ffmpeg.autogen to cache files and use flyleaf to play them, but using the fast ShowFrameNext will result in errors. I implemented a lock using Metarama to ensure that the method can only be executed once at the same time. If you are interested, you can take a look at my ffmpeg configuration. For ffmpeg, I am a beginner. Although these configurations took me a lot of time, I think there are still issues:

ssp_client.dll is supplied by the camera manufacturer, and the function of this code is to call the methods of ssp_client.dll and process the data in the callback.

First of all, OnMediaInfoReceived will be triggered first, you can know some basic information about video and audio from here; Next, OnVideoDataReceived and OnAudioDataReceived will be triggered, it will pass in the basic video data (SspH264Data), audio data (SspAudioData)

This code will initialize the ffmpeg parameters when the user calls StartCache and cache the data as a file in the next OnVideoDataReceived and OnAudioDataReceived.

The camera's internal time base is 1000000 and the encoding format is HEVC. The cached generated file plays fine using potplayer, vlc, and flyeaf, and I don't know how to test the playback performance for a file that has been generated.

Also, SSPStream was mentioned in my last issue, specifically for live play streaming. I can upload a cached video file if you want to know more.

using System;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO;
using System.Runtime.InteropServices;
using System.Threading;
using System.Windows.Controls;
using Vortice.MediaFoundation;
using Windows.Storage.Streams;

namespace SSP
{
    public struct SspData
    {
        public string IPStr;
        public int Port = 9999;
        public Action<Stream> Live = null;
        public SspData()
        {
        }
    }
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct SspVideoMeta
    {
        public uint width;
        public uint height;
        public uint timescale;
        public uint unit;
        public uint gop;
        public uint encoder;
    };
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct SspAudioMeta
    {
        public uint timescale;
        public uint unit;
        public uint sample_rate;
        public uint sample_size;
        public uint channel;
        public uint bitrate;
        public uint encoder;
    };
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct SspMeta
    {
        public bool pts_is_wall_clock;
        public bool tc_drop_frame;
        public uint timecode;
    };
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct SspH264Data
    {
        public IntPtr data;
        public ulong len;
        public ulong pts;
        public ulong ntp_timestamp;
        public uint frm_no;
        public uint type;
    };
    [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
    public struct SspAudioData
    {
        public IntPtr data;
        public ulong len;
        public ulong pts;
        public ulong ntp_timestamp;
    };
    public delegate void VideoFN(SspH264Data data);
    public delegate void AudioFN(SspAudioData data);
    public delegate void MediaFN(SspVideoMeta v, SspAudioMeta a, SspMeta m);
    public delegate void ConnectFN();
    public delegate void DisConnectFN();
    public delegate void ExceptionFN(int code, string description);
    public delegate void BufferFullFN();

    internal unsafe class SSPStream : Stream
    {
        private readonly NLog.Logger Logger = NLog.LogManager.GetLogger("steam");
        private readonly ConcurrentQueue<byte[]> data = new();
        private byte[] remainingData = null;
        private byte[] readBuffer = null;
        private int remainingDataOffset = 0;
        public override bool CanRead => true;
        public override bool CanSeek => false;
        public override bool CanWrite => true;
        public override long Length => 0;
        public override long Position { set { } get { return 0; } }
        public override void Flush() { }
        public override int Read(byte[] buffer, int offset, int count)
        {
            int read = 0;
            Stopwatch sw = new();
            sw.Start();
            while (read <= 0 && sw.ElapsedMilliseconds < 100)
            {
                if (remainingData != null)
                {
                    int remainingDataLen = remainingData.Length - remainingDataOffset;
                    if (remainingDataLen > 0)
                    {
                        int copyLen = count;
                        if (copyLen > remainingDataLen)
                        {
                            copyLen = remainingDataLen;
                        }
                        Array.Copy(remainingData, remainingDataOffset, buffer, offset, copyLen);
                        remainingDataOffset += copyLen;
                        read += copyLen;
                    }
                    if (remainingDataOffset >= remainingData.Length)
                    {
                        remainingData = null;
                        remainingDataOffset = 0;
                    }
                }

                while (read < count && ((remainingData != null && remainingData.Length != 0) || !data.IsEmpty))
                {
                    if (data.TryDequeue(out byte[] bytes))
                    {
                        int copyLen = count - read;
                        if (copyLen > bytes.Length)
                        {
                            copyLen = bytes.Length;
                        }
                        Array.Copy(bytes, 0, buffer, offset + read, copyLen);
                        read += copyLen;
                        if (copyLen < bytes.Length)
                        {
                            remainingData = bytes;
                            remainingDataOffset = copyLen;
                        }
                    }
                }
            }

            Logger.Info($"R:{data.Count}");
            return read;
        }
        public override long Seek(long offset, SeekOrigin origin) => 0;
        public override void SetLength(long value) { }
        public override void Write(byte[] buffer, int offset, int count)
        {
            if (count > 0)
            {
                byte[] bytes = new byte[count];
                Array.Copy(buffer, offset, bytes, 0, count);
                data.Enqueue(bytes);
                Logger.Info($"W:{data.Count}");
            }
        }
        protected override void Dispose(bool disposing)
        {
            data?.Clear();
            remainingData = null;
            base.Dispose(disposing);
        }

    }
    public unsafe partial class Ssp(SspData sspData)
    {
        #region common attributes
        private readonly NLog.Logger Logger = NLog.LogManager.GetLogger(sspData.IPStr);
        private SspVideoMeta videoMeta;
        private SspAudioMeta audioMeta;
        private SspMeta meta;
        private AVRational vRationaReal = new() { num = 1, den = 1000000 };
        private AVRational vRationalTarget = new() { num = 1, den = 90000 };
        private AVRational framerate;
        private readonly string deviceIP = sspData.IPStr;
        private readonly int devicePort = sspData.Port;
        private readonly string deviceKey = $"{sspData.IPStr}-{Guid.NewGuid()}";
        private bool deviceConnected = false;
        private int deviceFirstFrameNo = 0;
        private AVOutputFormat* outputFormat = ffmpeg.av_guess_format("mpegts", null, null);
        #endregion

        #region attributes for cache
        private Timer cacheTimer = null;
        private long cacheStartTime = 0;
        private TimeSpan cacheDealTime;
        private bool caching = false;

        private AVFormatContext* cacheOutputFormatContext;
        private AVCodecContext* cacheVideoCodecContext;
        private AVStream* cacheVideoStream;
        private AVCodec* cacheVideoCodec;
        private AVCodecParameters* cacheVideoCodecParams;
        private AVCodecContext* cacheAudioCodecContext;
        private AVStream* cacheAudioStream;
        private AVCodec* cacheAudioCodec;
        private AVCodecParameters* cacheAudioCodecParams;
        #endregion

        #region attributes for live
        private SSPStream liveVideoMEMStream = null;
        #endregion

        private void OnMediaInfoReceived(SspVideoMeta _videoMeta, SspAudioMeta _audioMeta, SspMeta _meta)
        {
            Logger.Info($"Video: width={_videoMeta.width} height={_videoMeta.height} timescale={_videoMeta.timescale} unit={_videoMeta.unit} gop={_videoMeta.gop} encoder={_videoMeta.encoder}");
            Logger.Info($"Audio: timescale={_audioMeta.timescale} unit={_audioMeta.unit} sample_rate={_audioMeta.sample_rate} sample_size={_audioMeta.sample_size} channel={_audioMeta.channel} bitrate={_audioMeta.bitrate} encoder={_audioMeta.encoder}");
            Logger.Info($"Meta: pts_is_wall_clock={_meta.pts_is_wall_clock} tc_drop_frame={_meta.tc_drop_frame} timecode={_meta.timecode}");

            videoMeta = _videoMeta;
            audioMeta = _audioMeta;
            meta = _meta;
            framerate = new() { num = (int)videoMeta.gop, den = 1 };

            StartLive();
        }
        private void OnVideoDataReceived(SspH264Data data)
        {
            if (!deviceConnected || data.data == IntPtr.Zero || data.len == 0) { deviceFirstFrameNo = (int)data.frm_no; return; }
            byte[] byteArrayVideo = new byte[data.len];
            fixed (byte* pBuffer = byteArrayVideo)
            {
                #region live play steam
                Marshal.Copy(data.data, byteArrayVideo, 0, (int)data.len);
                liveVideoMEMStream.Write(byteArrayVideo, 0, byteArrayVideo.Length);
                #endregion
                #region cache file
                if (caching)
                {
                    Marshal.Copy(data.data, byteArrayVideo, 0, (int)data.len);
                    AVPacket* inputPacket = ffmpeg.av_packet_alloc();
                    inputPacket->data = pBuffer;
                    inputPacket->size = (int)data.len;
                    inputPacket->pts = (long)data.pts;
                    inputPacket->dts = inputPacket->pts;
                    ffmpeg.av_packet_rescale_ts(inputPacket, vRationaReal, vRationalTarget);
                    inputPacket->stream_index = 0;
                    ffmpeg.av_interleaved_write_frame(cacheOutputFormatContext, inputPacket);
                    ffmpeg.av_packet_unref(inputPacket);
                }
                #endregion

            }
        }
        private void OnAudioDataReceived(SspAudioData data)
        {
            if (!deviceConnected || data.data == IntPtr.Zero || data.len == 0 || !caching) { return; }
            byte[] byteArrayAudio = new byte[data.len];
            fixed (byte* pBuffer = byteArrayAudio)
            {
                Marshal.Copy(data.data, byteArrayAudio, 0, (int)data.len);
                AVPacket* inputPacket = ffmpeg.av_packet_alloc();
                inputPacket->data = pBuffer;
                inputPacket->size = (int)data.len;
                inputPacket->pts = (long)data.pts;
                ffmpeg.av_packet_rescale_ts(inputPacket, vRationaReal, vRationalTarget);
                inputPacket->dts = inputPacket->pts;
                inputPacket->stream_index = 1;
                ffmpeg.av_interleaved_write_frame(cacheOutputFormatContext, inputPacket);
                ffmpeg.av_packet_unref(inputPacket);
            }
        }

        public void Start()
        {
            if (!deviceConnected)
            {
                Connect(CommonUtil.FromStr(deviceIP), CommonUtil.FromStr(deviceKey), devicePort, OnVideoDataReceived, OnAudioDataReceived, OnMediaInfoReceived, OnConnection, OnDisConnect, OnException, OnBufferFull);
            }
        }
        public void Stop()
        {
            if (deviceConnected)
            {
                DisConnect(CommonUtil.FromStr(deviceKey));
                OnDisConnect();
            }
        }

        private void StartLive()
        {
            liveVideoMEMStream = new();
        }
        private void StopLive()
        {
            if (liveVideoMEMStream != null)
            {
                liveVideoMEMStream.Dispose();
                liveVideoMEMStream = null;
            }
        }

        public void StartCache(string fileName)
        {
            cacheStartTime = ((DateTimeOffset)DateTime.Now).ToUnixTimeMilliseconds();

            cacheOutputFormatContext = ffmpeg.avformat_alloc_context();
            cacheOutputFormatContext->oformat = outputFormat;
            ffmpeg.avio_open(&cacheOutputFormatContext->pb, fileName, ffmpeg.AVIO_FLAG_WRITE);

            cacheVideoCodecContext = ffmpeg.avcodec_alloc_context3(null);
            cacheVideoStream = ffmpeg.avformat_new_stream(cacheOutputFormatContext, null);

            cacheAudioStream = ffmpeg.avformat_new_stream(cacheOutputFormatContext, null);
            cacheAudioCodecContext = ffmpeg.avcodec_alloc_context3(null);

            cacheVideoCodec = ffmpeg.avcodec_find_decoder(AVCodecID.AV_CODEC_ID_HEVC);
            cacheVideoCodecParams = ffmpeg.avcodec_parameters_alloc();
            ffmpeg.avcodec_parameters_from_context(cacheVideoCodecParams, cacheVideoCodecContext);

            cacheVideoCodecParams->codec_type = AVMediaType.AVMEDIA_TYPE_VIDEO;
            cacheVideoCodecParams->width = (int)videoMeta.width;
            cacheVideoCodecParams->height = (int)videoMeta.height;
            cacheVideoCodecParams->codec_id = AVCodecID.AV_CODEC_ID_HEVC;

            cacheVideoStream->codecpar = cacheVideoCodecParams;
            cacheVideoStream->time_base = vRationalTarget;
            cacheVideoStream->avg_frame_rate = framerate;
            cacheVideoStream->r_frame_rate = framerate;

            cacheVideoCodecContext->gop_size = (int)videoMeta.gop;
            cacheVideoCodecContext->framerate = framerate;
            cacheVideoCodecContext->time_base = vRationalTarget;

            ffmpeg.avcodec_parameters_to_context(cacheVideoCodecContext, cacheVideoCodecParams);
            ffmpeg.avcodec_open2(cacheVideoCodecContext, cacheVideoCodec, null);

            cacheAudioCodec = ffmpeg.avcodec_find_decoder(AVCodecID.AV_CODEC_ID_AAC);
            cacheAudioCodecParams = ffmpeg.avcodec_parameters_alloc();
            ffmpeg.avcodec_parameters_from_context(cacheAudioCodecParams, cacheAudioCodecContext);

            ffmpeg.av_channel_layout_default(&cacheAudioCodecParams->ch_layout, (int)audioMeta.channel);
            cacheAudioCodecParams->codec_id = AVCodecID.AV_CODEC_ID_AAC;
            cacheAudioCodecParams->sample_rate = (int)audioMeta.sample_rate;
            cacheAudioCodecParams->bit_rate = (int)audioMeta.bitrate;
            cacheAudioCodecParams->codec_type = AVMediaType.AVMEDIA_TYPE_AUDIO;
            cacheAudioCodecParams->frame_size = (int)audioMeta.sample_size;

            cacheAudioStream->codecpar = cacheAudioCodecParams;
            cacheAudioStream->time_base = vRationalTarget;

            cacheAudioCodecContext->time_base = vRationalTarget;

            ffmpeg.avcodec_parameters_to_context(cacheAudioCodecContext, cacheAudioCodecParams);
            ffmpeg.avcodec_open2(cacheAudioCodecContext, cacheAudioCodec, null);
            ffmpeg.avformat_write_header(cacheOutputFormatContext, null);

            caching = true;
        }

        public void StopCache()
        {
            caching = false;
            if (cacheOutputFormatContext != null)
            {
                if (cacheOutputFormatContext->pb != null)
                {
                    int result = ffmpeg.av_write_trailer(cacheOutputFormatContext);
                    if (result == 0)
                    {
                        ffmpeg.avio_close(cacheOutputFormatContext->pb);
                    }
                }
                ffmpeg.avformat_free_context(cacheOutputFormatContext);
                cacheOutputFormatContext = null;
            }
            if (cacheVideoCodecContext != null)
            {
                fixed (AVCodecContext** ptr = &cacheVideoCodecContext)
                {
                    ffmpeg.avcodec_free_context(ptr);
                }
                cacheVideoCodecContext = null;
            }
            if (cacheAudioCodecContext != null)
            {
                fixed (AVCodecContext** ptr = &cacheAudioCodecContext)
                {
                    ffmpeg.avcodec_free_context(ptr);
                }
                cacheAudioCodecContext = null;
            }

            cacheVideoStream = null;
            cacheVideoCodec = null;
            cacheVideoCodecParams = null;
            cacheAudioStream = null;
            cacheAudioCodec = null;
            cacheAudioCodecParams = null;
        }

        #region Protocol Event Handling
        private void OnConnection()
        {
            Logger.Info("ConnectFN Received");
            deviceConnected = true;
        }
        private void OnDisConnect()
        {
            Logger.Info("DisConnectFN Received:");
            if (caching)
            {
                StopCache();
            }
        }
        private void OnException(int code, string description)
        {
            Logger.Error($"ExceptionFN Received: {code}, {description}");
        }
        private void OnBufferFull()
        {
            Logger.Info("BufferFullFN Received:");
        }
        #endregion

        [LibraryImport("ssp_client.dll")]
        [UnmanagedCallConv(CallConvs = new Type[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })]
        private static partial void Connect(IntPtr ip, IntPtr key, int port, VideoFN videoFN, AudioFN audioFN, MediaFN mediaFN, ConnectFN connectFN, DisConnectFN disConnectFN, ExceptionFN exceptionFN, BufferFullFN bufferFullFN);
        [LibraryImport("ssp_client.dll")]
        [UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
        private static partial void DisConnect(IntPtr key);
    }
}