[Request]: Is it possible to use a live streaming source instead of webcam?

[sautechgroup]

MediaPipe4U Version

2022.0.1

UE Version

5.0.x

UnrealEngine Type

Compiled From Source Code

What happened?

Hello everyone, We have an rtmp stream from a remote pc and we are able to play this stream in a media player by attaching it to a stream media source. We would to know if it is possible to give in input this stream media source to MediaPipe4U instead of the webcam or some video file. We read the documentation and seems to be possible by blueprint only with a video file. Thank you in advance. Regards.

[endink]

It is not support now. But if you can write C++ code, I can tell you how to do it. It is easy.

[sautechgroup]

Hello, yes I think it's the only way we can take. Can you tell me how to do it? Thank you and have a nice day.

[endink]

Alright, first of all, I have to apologize for the lack of documentation in this section.

You need implement a "ImageSourceComponent" by your self. Although this sounds very complicated because of the design to flow and multithreading， but most of the work in this implementation has already been done by the plugin.

1. Write a a "Component " that inherits from UMediaPipeImageSourceComponent.
2. Override function IsSupportHorizontalFlip, IsSupportLimitResolution (return false is OK).
3. In your "Component" tick event, read a image and push to "Stream".

How to push image:

1. Invoke PrepareTexturePool function to tell MediaPipe4U image size and format.
2. Invoke Stream().Push method

A simple example code:

auto* self  = this;
Streaming()->PushFrame([self](MediaPipeTexture* PooledTexture)
{
  .... // set PooledTexture data

  self->NotifyTexture2DCreated(PooledTexture);
  return true;
});

How to read image to MediapipeTexture?

it has some help function:

• bool AdoptCvMat(void* SrcMat);
• bool CopyFromCvMat(void* SrcMat);
• bool CopyFromData(void* InData, SIZE_T DataSize);
• bool CopyFromImage(IImageWrapper* InImage, int32 BitDepth = 8);

CvMat means an opencv mat class (cv::Mat)
InData means an uint8 array, DataSize means array length Adopt means zero copy, but now, only cv::Mat can "zero copy"

you can use CopyFromData, read unit8 array from MediaTexutre, and copy the array to MediaPipeTexture

[sautechgroup]

I'll try this solution and recompile the plugin and I will let you know if I'll have good news. Thank you and best regards.

[endink]

sorry again, i will write doc for this asap.

[sautechgroup]

I upgrated to latest version of MediaPipe4U and I saw that there is a MediaPlayerImageSourceComponent.h completely commented. I use a media player to show the stream in unreal. Is there a possibility to reuse this component to make my job?

[endink]

Sorry, you cant. MediaPlayer can't decode some video formats well,even decoding MP4 will get an error sometimes. GStreamer is the perfect replacement (NVIDIA also uses Gstreamer for video AI, so it's definitely a trustworthy project).

If you need RTMP stream right now, you'll have to do it yourself, and I'm marking this issue as an "enhancement", so if I have the time, I might use GStreamer to implement RTMP, in fact, everything Media Player can do easily with Gstreamer.

[endink]

BTW, push stream

Alright, first of all, I have to apologize for the lack of documentation in this section.

You need implement a "ImageSourceComponent" by your self. Although this sounds very complicated because of the design to flow and multithreading， but most of the work in this implementation has already been done by the plugin.

1. Write a a "Component " that inherits from UMediaPipeImageSourceComponent.
2. Override function IsSupportHorizontalFlip, IsSupportLimitResolution (return false is OK).
3. In your "Component" tick event, read a image and push to "Stream".

How to push image:

1. Invoke PrepareTexturePool function to tell MediaPipe4U image size and format.
2. Invoke Stream().Push method

A simple example code:

auto* self  = this;
Streaming()->PushFrame([self](MediaPipeTexture* PooledTexture)
{
  .... // set PooledTexture data

  self->NotifyTexture2DCreated(PooledTexture);
  return true;
});

How to read image to MediapipeTexture?

it has some help function:

• bool AdoptCvMat(void* SrcMat);
• bool CopyFromCvMat(void* SrcMat);
• bool CopyFromData(void* InData, SIZE_T DataSize);
• bool CopyFromImage(IImageWrapper* InImage, int32 BitDepth = 8);

CvMat means an opencv mat class (cv::Mat) InData means an uint8 array, DataSize means array length Adopt means zero copy, but now, only cv::Mat can "zero copy"

you can use CopyFromData, read unit8 array from MediaTexutre, and copy the array to MediaPipeTexture

BTW, the "PushFrame" function has an overload that supports pushing a media texture directly, but I haven't tested it carefully, and maybe it will help you get uint8 arrays from a MediaTexture (for MediaPlayer output)

[sautechgroup]

Hello, I'm trying to use CopyFromData function, but I don't know how to pass the MediaTexture object as an uint8 array. Is there any conversion I need to make? Thank you.

[endink]

After CopyFromData, you dont need to do anything.

[sautechgroup]

you can use CopyFromData, read unit8 array from MediaTexutre, and copy the array to MediaPipeTexture How can I read MediaTexture as an uint8 array?

[endink]

How to read data (uint8 array ) from MediaTexture, you need google it by your self

[sautechgroup]

If I understood well, self->NotifyTexture2DCreated(PooledTexture) fires the VideoTextureCreated event. How can i call this function if I use the PushFrame function that takes in input directly a MediaTexture? I googled for making an uint8 array from a MediaTexture, but the only way I found to make an array is the ReadPixels function that creates a TArray of type Fcolor Thank you for the support.

EDIT: Right now I found the way to convert an TArray Fcolor in TArray uint8. I'm almost done, I only miss the part where I display the video into the widget.

[endink]

Oops... Sorry, PushFrame use a mediaptexture cant notify,Its my fault, I forgot that in my code, You are right, NotifyTexture2D is the key for your rendering.

Fcolor is 4 bytes, the order is related to the image format. In fact, it is easy to convert them, it seems that you are not familiar with the knowledge of pixels, I will give you the code directly later.

[endink]

@sautechgroup

UMediaTexture* Texture = XXXX....
MediaPipeTexture* MediaPipeTexture = XXXX..

TArray<FColor> colors;
FMediaTextureResource* TexResource = static_cast<FMediaTextureResource*>(Texture->GetResource());
TexResource->ReadPixels(colors);
MediaPipeTexture->Width = Texture->GetWidth();
MediaPipeTexture->Height = Texture->GetHeight();
//mediaptexture is RGBA8
MediaPipeTexture->Format = MediaPipeImageFormat::SRGB;
MediaPipeTexture->CopyFromData(colors.GetData(), colors.Num()*4);
return true;

Because FColor is order B->G->R->A, so its rgba8 format, you juse use GetData it will be return a memory array, its not only a FColor array for struct, but also a unit8 array in memory.

But, its a little problem:

The data of the media texture resides on the GPU. To modify/access it on the CPU, you must lock the bits. I haven't looked into mediatexture carefully, so I can't tell you the specific steps and how to lock mediatexture , the above code is for reference only.

It is worth mentioning, copy memory from gpu to cpu is slow and not recommended.

If you're new to RHI and image technology, I still don't recommend manually working with pixels.

[sautechgroup]

Hello, thank you for the support, yes I'm new to c++ programming on Unreal Engine (I usually don't use c++). I'll try and let you know. Thank you and have a nice day

[sautechgroup]

I documented about the GPU -> CPU issue and as I understood when I call ReadPixels the array is saved into a pointer so I think that I have no problem once I have the pixels array. I followed your advice, and I attached my new ImageSourceComponent to the "Start Image Source" blueprint function. After that I set a bool for starting to call PushFrame.

Here's the code:

` void URemoteImageSourceComponent::TickComponent(float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction) {

auto* self = this;
if (bCanPush && MediaPlayer->IsReady()) {

    Streaming()->PushFrame([self](MediaPipeTexture* PooledTexture)
    {
        PooledTexture = new MediaPipeTexture(self->MediaTexture->GetWidth(), self->MediaTexture->GetHeight(), MediaPipeImageFormat::SRGB);
        TArray<FColor> PixelData;
        UE_LOG(LogTemp, Warning, TEXT("PixelData"));
        FMediaTextureResource* TextureResource = static_cast<FMediaTextureResource*>(self->MediaTexture->Resource);
        UE_LOG(LogTemp, Warning, TEXT("TextureResource"));
        if (TextureResource)
        {
            TextureResource->ReadPixels(PixelData);
            UE_LOG(LogTemp, Warning, TEXT("ReadPixels"));
            if (PixelData.Num() > 0)
            {
                UE_LOG(LogTemp, Warning, TEXT("PrepareTexturePool"));
                PooledTexture->CopyFromData(PixelData.GetData(), PixelData.Num() * 4);
                UE_LOG(LogTemp, Warning, TEXT("CopyFromData"));
            }
        }

        return true;
    });
    self->NotifyTexture2DCreated(self->MpTexture);

    bCanPush = false;
}

}

where bCanPush is the bool i set to true in blueprint, MpTexture is the MediaPipeTexture I have inside my ImageSourceComponent, MediaTexture, MediaPlayer and StreamingMediaSource are initialized inside blueprint.

When I call Start Image Source block in blueprint UE crashes and I have this log inside Visual Studio:

[2023.03.29-16.14.06:631][417]LogMediaPipe: Display: +FDynamicTexture 00000609BE90C6C0 [2023.03.29-16.14.06:631][417]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::Stopping [2023.03.29-16.14.06:631][417]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::Stopped [2023.03.29-16.14.06:631][417]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::Start [2023.03.29-16.14.06:632][417]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::Start OK [2023.03.29-16.14.06:633][417]LogTemp: Warning: PixelData [2023.03.29-16.14.06:633][417]LogTemp: Warning: TextureResource [2023.03.29-16.14.06:636][417]LogMediaPipe: Display: [MediaPipe API] Enter WorkerThread: -1328396028 [2023.03.29-16.14.06:636][417]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::RunImageImpl [2023.03.29-16.14.06:636][417]LogMediaPipe: Warning: [MediaPipe API] FaceGeometry is enabled, auto disable refine_face_landmarks options. [2023.03.29-16.14.06:636][417]LogMediaPipe: Display: [MediaPipe API] MediaPipe graph nodes: 15 [2023.03.29-16.14.06:639][417]LogTemp: Warning: ReadPixels [2023.03.29-16.14.06:639][417]LogTemp: Warning: PrepareTexturePool [2023.03.29-16.14.06:639][417]LogTemp: Warning: CopyFromData [2023.03.29-16.14.06:702][421]LogMediaPipe: Display: [MediaPipe API] CalculatorGraph::Initialize OK [2023.03.29-16.14.06:702][421]LogMediaPipe: Display: [MediaPipe API] +UmpObserver output_video [2023.03.29-16.14.06:702][421]LogMediaPipe: Display: [MediaPipe API] CalculatorGraph::StartRun (pollers: 0, observers: 7) enable_segmentation : bool input_horizontally_flipped : bool input_rotation : int input_vertically_flipped : bool model_complexity : int refine_face_landmarks : bool smooth_landmarks : bool smooth_segmentation : bool use_prev_landmarks : bool

[2023.03.29-16.14.06:705][421]LogMediaPipe: Display: [MediaPipe API] ------------> Start Loop Work Thread <------------ [2023.03.29-16.14.06:705][421]LogMediaPipe: Display: [MediaPipe API] UmpPipeline::AddImageFrameIntoStream (in loop) OK. [2023.03.29-16.14.06:705][421]LogMediaPipe: Display: [MediaPipe API] Notify image size received ( w: 2, h: 2 ). [2023.03.29-16.14.06:705][421]LogMediaPipe: Display: [MediaPipe API] First look workflow is done ! Exception thrown at 0x00007FFA015340AC in UnrealEditor.exe: Microsoft C++ Exception: cv::Exception at memory location 0x0000003717BFDC60. Exception thrown at 0x00007FFA015340AC (KernelBase.dll) in UnrealEditor.exe: 0x00004000 (parameters: 0x0000003717BFC7D0). Additional Crash Context (Key="Breadcrumbs_RHIThread_0", Value="Breadcrumbs 'RHIThread' ")'UnrealEditor.exe' (Win32): caricamento di 'C:\Windows\System32\psapi.dll' completato. [2023.03.29-16.14.08:803][477]LogRHI: Error: Breadcrumbs 'RHIThread'

[2023.03.29-16.14.08:803][477]LogWindows: FPlatformMisc::RequestExit(1) [2023.03.29-16.14.08:803][477]LogWindows: FPlatformMisc::RequestExitWithStatus(1, 3) [2023.03.29-16.14.08:803][477]LogCore: Engine exit requested (reason: Win RequestExit) Il thread 0x2aa8 è terminato con il codice 0 (0x0). Il thread 0xbcdc è terminato con il codice 3 (0x3). Il thread 0xa9f8 è terminato con il codice 3 (0x3). Il thread 0x4d10 è terminato con il codice 3 (0x3). Il thread 0x8b68 è terminato con il codice 3 (0x3). Il thread 0xb84c è terminato con il codice 3 (0x3).

I inserted logs like "LogTemp: Warning: CopyFromData" to see the workflow of PushFrame lambda function. I don't understand if I need to call NotifyTextureCreated after the PushFrame or I don't need to call it. I reset to false the flag for having a clean log. Can you tell me why is thows this exception? Thank you.

[endink]

First of all, you can't save an mp texture as a class member variable, there is a complex design behind it, as the parameter name "pooled texture" , behind it is a pool, you have to return it when you run out, there are a lot of thread, RHI, mediapipe, render,game thread etc, only if all steps are successful you can return true, Notify is to transfer texture to texture 2d, which has RHI thread and switching game threads, for that you don't have to care about these details, so the PushFrame function looks a little strange, but this is the simplest.

It's hard to know the problem from the error message because of the lack of English text, but I'm guessing that RHI pixels (mediatexture) can only be read in RHI threads, "ticks" is in game thread, I advise you to stop trying, there is a lot of complex things here that will waste a lot of your time, I strive for the next release to implement an rtmp protocol (by GStreamer)

[endink]

If you are interested in extending M4U with C++, I recommend you read here to help understand the design behind M4U：

https://opensource-labijie-com.translate.goog/Mediapipe4u-plugin/extensions/image_consumer.html?_x_tr_sl=zh-CN&_x_tr_tl=en&_x_tr_hl=zh-CN&_x_tr_pto=wapp

[sautechgroup]

I have a function that direcly converts MediaTexture in UTexture2D. You said that NotifyTexture is needed to convert the MediaPipeTexture in Texture2D, so I imagine that MediaPipe processes the Texture2D object, right? If yes, can I use directly the UTexture2D object? Thank you, I know that it's a complex field, expecially for who is not familiar with image manipulation.

[endink]

emmm, NotifyTextureCreated will auto convert MediaPipeTexture to Texture2D, if you dont need M4U TextureCreatedEvent, you can ignore this and draw texture to your UI by yourself.

[sautechgroup]

Ok I'm trying to use directly the UTexture2D object and use the PushStream override method. My last chance :D I found a function called StartLoopThread that I saw in webcam logs that is called before starting the webcam. I tried to call it in my function (once not at every tick) and I have this log: [2023.03.30-16.16.01:109][507]LogMediaPipe: Display: ImageSourceLoop Start! ThreadID = 32664 [2023.03.30-16.16.01:110][507]LogMediaPipe: Display: ImageSourceLoop thread enter. [2023.03.30-16.16.01:110][507]LogMediaPipe: Display: Handle loop break. [2023.03.30-16.16.01:110][507]LogMediaPipe: Display: ImageSourceLoop thread exit.

I saw that this call let me see the first frame into the VideoImage object into the widget and fires the VideoTextureCreated event in blueprint. But I don't know why it breaks. Any suggestions? Thank you for the support and sorry for lack of knowledge in this technology.

[endink]

If your image source is an infinite loop call, then a separate thread may be required to execute it, you can see some function that the named "Loop" are for this, if you use tick, then the loop thread is not needed. The implemention for webcam Probably like this：

while(runFlag)
{
   ReadFrame... 
   PushFrame... 
}

So it use startLoop, onLoop and these functions simplify creating a thread to execute this loop, you can also write your own, there is nothing mysterious about them.

[endink]

Do you have a rtmp address for testing? I can write a GStreamer implemention.

[sautechgroup]

Hello, sorry for the late answer, jet lag problems.. No I don't have an rtmp stream, but if you use SRS (Simple Realtime Server) and install docker with this giude: https://ossrs.io/lts/en-us/docs/v4/doc/getting-started

You can use rtc2rtmp configuration: https://ossrs.io/lts/en-us/docs/v4/doc/webrtc#rtc-to-rtmp and see your webcam by "Publish by WebRTC" link provided (that is localhost). Finally you can see your webcam with rtmp://localhost/live/show. Thank you.

[endink]

Alright, I have used SRS and it is a Chinese project : )

[sautechgroup]

If you used SRS it also uses WebRTC. I choosed RTMP conversion because it's easily streammable into a UMediaPlayer object, but I know that GStreamer also supports WebRTC. So it does not matter for me if it uses RTMP or WebRTC, indeed it would be better use WebRTC. Thank you and have a nice day.

[endink]

WebRTC is hard to implement，It's not a question about the code, it's about webrtc spec.

The WebRTC specification does not specify how the signalling server needs to be implemented. It is perfectly valid to exchange the SDP and ICE candidates via email for example.

Unless I'm binding a signalling server implementation, or reserving an interface to be implemented by the user himself, it doesn't seem like a UE plugin's job anyway.

BTW：I have done "URI image" support, not only rtmp, but also RTSP, M3U8, etc., as long as the network protocols supported by GStreamer can be used as "image source" , which will be released with the next M4U version.

[endink]

RTMP is supported now (please download the latest version), the parameter of the GStreamerImageSourceComponent::Start can be a file path or URIs, URIs can be any protocol, http/rtsp/rtmp, etc.

[sautechgroup]

Hello, I'm really thankful about this. Thank you and have a nice day.

[endink]

Although the issue was closed, I just mark it, the latest version of the plugin already supports MediaPlayer as an image source:

Document is here