search

endink / Mediapipe4u-plugin

371 stars 52 forks source link

Network replication / multiplayer issues #20

Closed 4thWallInt closed 1 year ago

4thWallInt commented 1 year ago

When I am testing this in the UE editor with two players (multiplayer mode) , with Play as Client or Play as Listen Server option, the program crashes, and it gives an error at the MediaPipeHolisticComponent cpp class, which we do not have access to. Also it doesn't seem to work with network replication at all, the pose solvers that we have set up in the animation blueprints, are moving all the players together in multiplayer mode, where actually it should just move the local player."

endink commented 1 year ago

  1. Regarding the MediaPipe error, post the log file at here when an error occurs, which is usually in the saved/logs folder so that I can find out what the problem is.

  2. Network, I don't think multiplayer animation has much to do with MediaPipe4U, it should be a server streaming animation data and client-side display problem, I believe UnrealEngine has the example of that. You can find UE document or market to get the sample.

  3. Important note about the MediaPipeAminInstance (your animation blueprint). At the same time in the world outline you must make sure it has only one MediaPipeAminInstance , If you want to sync animation to many actors, you can use IK Reatarget to anothers. Because the landmarks data of mediapipe that made by MediaPipeHolisticComponent can be deal one time, multiple consumers (MediaPipeAminInstance ) will be get a low animation frame rate (Multiple MediaPipeAminInstance will be competition for data).

    BTW: sorry for my bad English , this is a question of the structure of the program, which is difficult to explain in a short answer.

sameekue5 commented 1 year ago

MediaPipe4U_Demo.log

Hey Anders, this is Sameek , and I am working with Giovanni on this project. As you requested , here is the saved log file for the error which is occuring when I am switching the number of players to more than 1 in the editor, for multiplayer testing.

I dont understand what you meant on the second point, because all the other animations are working for the player at both client and server. Only the Mediapipe Solver nodes that we have setup in the animation Blueprints, are controlling all the players together instead of just the client player.

Also , we are not looking for syncing animations. Our use case is pretty simple, the mediapipe solvers should move only the local player , that is obvious right ? Because if I am waving my hand, I only want my player to wave its hand, and not the server player. And also if I am waving my hand, the other players shall also see my hand movement, currently if the opponent players are waving their hands, I am not able to view it.

endink commented 1 year ago

From the logs, it can be seen that the component was reloaded or re-initialized for some reason. when mediapipe started, you can not uninitialize it or re-initialized it.

The following invocations should not appear after you start capture.

UMediaPipeHolisticComponent::CreateObserver() 
UMediaPipeHolisticComponent::SetupObservers() 
UMediaPipeHolisticComponent::InitializeComponent()  <---------Here, why we need Initialize ? because uninitialize its owner Actor?

I think this is because you break the Actor for some reason, and moving the MediaPipe component to another Actor might solve this problem

sameekue5 commented 1 year ago

What do you mean ? This is happening from your code only, we did not even have access to the source code, so we did not modify it.

endink commented 1 year ago

I guess the actor is re-initialized, which causes the component attached by the actor to be re-nitialize. This is due to inappropriate usage, and it is possible that the code did not handle this error and caused the crash. But arguing about whose code it was generated did nothing to solve the problem.

sameekue5 commented 1 year ago

The crash is causing as soon as I am calling the StartCamera Blueprint node from The MediaPipeHolistic component. If you dont want to solve the issues generated from your code, you can just say so that you dont want to provide support. But you are behaving as if this code is not written by you.

4thWallInt commented 1 year ago

Hey I wanted to come in and just say that there is no need to argue over these things. On our side, we are just trying to do the best we can with the available information to resolve these issues. On Anders' side we completely understand that he is dealing with many requests and has a lot of work on his plate.

Let's keep the conversation civil please

endink commented 1 year ago

From the logs, it can be seen that the component was reloaded or re-initialized for some reason. when mediapipe started, you can not uninitialize it or re-initialized it.

The following invocations should not appear after you start capture.

UMediaPipeHolisticComponent::CreateObserver() 
UMediaPipeHolisticComponent::SetupObservers() 
UMediaPipeHolisticComponent::InitializeComponent()  <---------Here, why we need Initialize ? because uninitialize its owner Actor?

I think this is because you break the Actor for some reason, and moving the MediaPipe component to another Actor might solve this problem

I'm helping you, but you've been dissatisfied, it's not the attitude of discussing problems, I've told you how to try to solve it, but you don't want to hear it at all, you're just expressing dissatisfaction.

moving the MediaPipe component to another Actor might solve this problem !!!!!

sameekue5 commented 1 year ago

image It is still the same, it does not matter where I am moving the MediaPipe component, as soon as I am calling the StartCamera function, it crashes for more than 1 player. For just 1 player it is fine.

endink commented 1 year ago

Is there this line in every crash log file?

Error: [mediapipe_api.dll] Invalid state: pipeline running

You can give me a crash log file, after you moved MediaPipe Component to another Actor .

sameekue5 commented 1 year ago

MediaPipe4U_Demo.log Here it is.

Also please reproduce this at your end first. It is very easy to reproduce and also you will have better understanding at what is causing the issue

  1. Change the Number of Players to 2 , and PlayAsListenServer
  2. Call the StartCamera function of MediaPipeHolistic component in Blueprints
4thWallInt commented 1 year ago

Anders, could you maybe try to reproduce this error on your end by just calling the StartCamera function while keeping the number of players to more than 1?

endink commented 1 year ago

Okay, I will try to reproduce 2 players later.

Error: [mediapipe_api.dll] Invalid state: pipeline running

One thing I know from this line is that when you call StartCamera the MediapipeHolisticComponent has not finished initializing, or the level is streaming (If async load level).You call StartCamera is too early, it should be in begin play or later, I will add a check in the next release to prevent this call ...

I will try reproduce it.

endink commented 1 year ago

Anyway, I'll replicate the multiplayer mode. And as a way to try it for now, inherit UMediaPipeHolisticComponent (C++, Blueprints can do the same), override its InitializeComponent function, add a feild as a flag (e.g. bInited = true) after the base UMediaPipeHolisticComponent ::InitializeComponent function call, and don't call camera when bInited not true:

YourComponent.h

UCLASS(ClassGroup="Default", meta=(BlueprintSpawnableComponent))
class UYourComponent : public UMediaPipeHolisticComponent
{
     UPROPERTY(EditAnywhere, BlueprintReadWrite)
     bool bInited = false;

     virtual void InitializeComponent() override;
}

YourComponent.cpp

void UYourComponent ::InitializeComponent()
{
    Super::InitializeComponent();
        bInited = true;
}

if bInited is not true, dont call StartCamera

endink commented 1 year ago
image

when i play PlayAsListenServer, every thing is OK. But, you can't use one camera to capture two windows at the same time， But even if you do that, it won't cause a crash

endink commented 1 year ago

Maybe it's a buggy for a specific version, I'm packing a new version that can be downloaded later today.

BTW: In my case, I didn't crash regardless of whether the number of players was 2 or 3, but my machine was stuck over 3

endink commented 1 year ago

Download a new version,it has been tested with multi player network mode.

sameekue5 commented 1 year ago

No, you just tested it with the default project as it it. But you need to test it with third person player character, without possessing the default pawn, because clearly the multiplayer here means that we are controlling the two characters.

So to reproduce the issue -

  1. Remove the mannequin bp from the level
  2. Add your defaultpawn class as the mannequin character in the world settings
  3. Replace both the game pawn and the character reference in the mediaplayer runtime BP with "Get Player Pawn" or "Get Player Character" instead.

I am still getting the error, attached are the screenshot and the log as well image MediaPipe4U_Demo.log

endink commented 1 year ago

OK~Wait for my testing

endink commented 1 year ago

@sameekue5 One thing, let me know when this crash happened, from the logs it looks like stopping the game

4thWallInt commented 1 year ago

Hey Anders, were you able on your end to reproduce the issue?

"So to reproduce the issue -

  1. Remove the mannequin bp from the level
  2. Add your defaultpawn class as the mannequin character in the world settings
  3. Replace both the game pawn and the character reference in the mediaplayer runtime BP with "Get Player Pawn" or "Get Player Character" instead. "
endink commented 1 year ago

I've been fixing it for 3 hours, it's a bit of a troublesome problem, It's almost done. But I'm afraid I don't have time to test this problem, when I fix it is done, please test it, and then feedback here. because In China, it's early in the morning.

sameekue5 commented 1 year ago

The crash happened as soon as I press the Play button, when it calls the StartCamera function in BP.

endink commented 1 year ago

I have done without full testing, just hot fix :

https://1drv.ms/u/s!AkmROUeQfSBjzgEz6gu8FCr7ibot?e=x79QpF

BTW: I'm going to bed, it's too late and too late

sameekue5 commented 1 year ago

image

The crash is not happening, but the video tracking is coming as black, and now the hand tracking is stopped as well. Kindly check

endink commented 1 year ago

Which version worked well with the video tracking on your PC？

sameekue5 commented 1 year ago

Unreal 5.0. It was working fine before with single player, but after your updates it stopped displaying the video

4thWallInt commented 1 year ago

Hey Anders, just wondering if you are still going to be helping us tomorrow

endink commented 1 year ago

Hi,

If plugin has BUG, post issue, I will fix it asap.

If it's just a technical discussion, I can't continue. I don't think discussing so much today are plug-in-related, and I don't have any plans to develop this piece of content.

BTW: I don't think a complex multiplayer game can be developed with Blueprints only, It's impossible to use Blueprints and wait for Unreal to do everything automatically, Epic can't do everything. Blueprints are just a designer tool. For my side, energy is limited. Sorry for that. Good luck !

Anders Xiao.

发件人: @.> 发送时间: 2023年2月12日 20:28 收件人: @.> 抄送: Anders @.>; State @.> 主题: Re: [endink/Mediapipe4u-plugin] Network replication / multiplayer issues (Issue #20)

Hey Anders, just wondering if you are still going to be helping us tomorrow

― Reply to this email directly, view it on GitHubhttps://github.com/endink/Mediapipe4u-plugin/issues/20#issuecomment-1427020447, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAB2UJEBTPGINGCRXPODD3TWXDJNFANCNFSM6AAAAAAUSQ7MWI. You are receiving this because you modified the open/close state.Message ID: @.***>