Sometimes json deserialize fail

[lucasjinreal]

LogTemp: Warning: json解码错误啊兄弟！！ {
  "version" : "1.2.5",
  "userName" : "iPhone",
  "sessionUUID" : "15FD7C25-412F-43E7-840F-E518230E4CB0",
  "deviceName" : "iPhone13,4"
}C?8????мC?8 
Live Link: Warning: （01） PoseAI: failed to deserialize json object from 192.168.18.235:62108

I just sucessed once, then everytime connect it fails.

[PoseAI]

Hi that is strange. Can you tell us what the packet format is set to in the handshake? The only time we have seen such a failure was when using format 0 (i.e. verbose) and the user's router was truncating the large packets. Most routers can handle large UDP packets but some truncate and leave the packet useless. Can you try changing the packet format in the handshake (if you are using 0 change to 1, or if you are using 1 change to 0) and see if that resolves?

If that doesn't resolve, please use our python script to inspect the packets, print the entire packet and post here so we can see what the problem is with the packet. That may help us identify the source of the issue.

In terms of multiple connections, often the issue is having multiple sources on Unreal LiveLink open. We have some nodes which manage open / close in blueprints, but if you are using the Unreal LiveLink UI to manually manage sources, make sure you close the extra ones. finally we have some troubleshooting tips in the info section on our website www.poseai.co.uk

On Sun, Jul 17, 2022 at 5:25 AM JinTian @.***> wrote:

LogTemp: Warning: json解码错误啊兄弟！！ {

"version" : "1.2.5",

"userName" : "iPhone",

"sessionUUID" : "15FD7C25-412F-43E7-840F-E518230E4CB0",

"deviceName" : "iPhone13,4"

}C?�8?�?�??мC?�8

Live Link: Warning: （01） PoseAI: failed to deserialize json object from 192.168.18.235:62108

I just sucessed once, then everytime connect it fails.

— Reply to this email directly, view it on GitHub https://github.com/PoseAI/PoseCameraAPI/issues/16, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOGO6DDFNQ635TVQEUKTTODVUODMBANCNFSM53ZDJ6PA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Pose AI - 'Understanding bodies in motion'

NOTE: These emails are for discussion purposes only. Unless and until we agree and sign a written License Agreement, our email exchanges do not constitute a binding contract, or a commitment by Pose AI Ltd or its employees to provide any software or services. We accept no liability, obligation or responsibility toward the recipient.

[lucasjinreal]

@PoseAI I think it's not iOS client issue, the python sever printed the hello messages normal. Sometimes I kill the app, restart the game, then connect again it can work, but sometimes also not.

I am also trying to debug in the livelink source code to get errror messages. currently no clue.

Is that possible because of the cached frames? ( now I am writting another python client to test the json send to PoseAI Livelink)

[lucasjinreal]

@PoseAI Hi, thanks for your reply, would like help me to debug this, I think I roughly find where goes wrong, but I don't know why:

You can reveal this sometims maybe (not every time which is the toughest one).

I using Python to send a handshake to UE5 livelink server, the server side log this:

{"version": "1.2.5", "userName": "iPhone2Python", "sessionUUID": "8EF5BA6A-A73E-419A-9BDC-7E18759C180C", "deviceName": "iPhone13,4"} , error: Unexpected additional input found. Line: 1 Ch: 132 
Live Link: Warning: （01） PoseAI: failed to deserialize json object from 127.0.0.1:54386
Live Link: Warning: （05） PoseAI: Incoming connection does not have a hello handshake

For better investigate I post a screenshot here:

I just using UE_LOG logged the recivmsg text content, you can see a block A bytes at the normal json str, Every single frame (except for handshake, the normal data), got same error:

error: Unexpected additional input found. 

The python I tested is very simple:

import socket
import json
from pprint import pprint
from llv.socketconn import SocketConn
import time
from alfred import logger

PORT_NUM = 8080

""" Prints your local IP address.  Configure this in the App.
  Make sure your router and firewall do not block the port """

def show_my_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(("10.255.255.255", 1))
        ip = s.getsockname()[0]
    except Exception:
        ip = "127.0.0.1"
    finally:
        s.close()
    print("Connect the app to IP:", ip, "Port:", PORT_NUM)

handshake = {
    "version": "1.2.5",
    "userName": "iPhone2Python",
    "sessionUUID": "8EF5BA6A-A73E-419A-9BDC-7E18759C180C",
    "deviceName": "iPhone13,4",
}

show_my_ip()

connections = {}

socket_conn = SocketConn("127.0.0.1", port=8080)
socket_conn.send_json_encoded(json.dumps(handshake).encode())

the Socket here is not as server but as client.

So here is my confusion:

• Why python same handshake decoded json error same as iOS client does? It should have something wrong cooperated with UE json reader;
• But this is happend only at 90% time, another 10% time the handshake is normal, but the frames data 100% decode error.

Hoping for your answer if you have time help me have a look.

[PoseAI]

hi, you changed the format of the handshake compared to our example, the handshake should be a nested dictionary with a field called "HANDSHAKE", i.e.

handshake = { 'HANDSHAKE': { ... } } The Unreal deserializer looks for the field 'HANDSHAKE'. Since it doesn't find it, it thinks the incoming connection is something else and doesn't establish the connection.

For the extra byte at the end of the packets (square with a in first, question mark in second), it is probably due to something in the send_json_encoded function which I can't see. Check the output in python just before sending to socket. it could be adding something at the end, expect a different format than dumps().encode() or perhaps sending the wrong length (i.e. +1 byte which would likely be read as a random byte)

[lucasjinreal]

@PoseAI Hi, thanks for the reply. I am not a skilled man in UE livelink, but the JSONReader deserialized error should not about HANDSHAKE field? I mean, if the json is normal, it should decode at least. Nevertheless the PoseAILivelink how to read the key in the json.

My json encode is very standard in python:

def send_json_encoded(self, data):
        self.s.sendall(data)

# call
socket_conn.send_json_encoded(json.dumps(handshake, ensure_ascii=False).encode('utf-8'))

As you can see, it was just dump the dict to json string, then encode as utf-8, then send the bytes to UDP connect.

Why just UE livelink sometimes get the extra bytes after? It shouldn't happen.

The thing that drive me crazy, is that this problem happened also on iOS client, and not everytime I can connected to the UE. Even as you said, I lack of HANDSHAKE field, it still can make connections sometimes..... I am totally missed here,.

[PoseAI]

We mentioned the Handshake because even if you resolve the extra symbol issue, you would still have a connection problem.

I can't explain why your code/install is sending the extra byte. Probably worth diagnosing outside of UE first and seeing if a different receiver also gets that extra byte. Can you confirm if our simple example, with no changes whatsoever, works as expected?

Re sometimes connecting, sometimes not, that extra byte is the most likely cause and should be resolved first. If it is happening with iOS then the most common causes are firewall/router issues or that you have duplicate LiveLInk sources open in the Unreal Engine (This can be checked by opening the LiveLink GUI from the Virtual Production submenu).

good luck!

[lucasjinreal]

@PoseAI Hi, I have to say, am exactly copy another blueprint to driven a metahuman character. Which means, I have 2 character using same blueprint in the scene. Except the thirdperson character using 8081 at runtime.

Could that be the problem? If so, why? (I thought the PoseAIComponent can setup 2 server at the same time, so that one client can driven them all), However, I am not sure if another metahuman blueprint runned, since it seems 2 server can not using same port.

[PoseAI]

Different sources on different ports is fine. We mentioned this as a possible issue because LiveLink sources persist through play in editor sessions, and some users have inadvertently had multiple sources running on the same port due to stopping and starting the play in editor (we have a close source node but it can get neglected). Worth opening the LiveLink GUI and checking that in your case you have the sources on the ports you expect

[lucasjinreal]

@PoseAI hi, the sources you mentioned here, is the client or the UE livelink server? You mean it could be the livelink server doesn't actually closed when I close and re-run in UE editor? If sometimes the server didn't closed, then start it again, and sending data, will it read wrong bytes?

I haven't get this issue though when using livelink arkit to drive face. even I stop and re-run for manytimes, it still can streaming data and decoded in right bytes.

[lucasjinreal]

@PoseAI Hi, sorry to bother you again.

I made these changes:

• delete another metahuman, using original single character level;
• add HANDSHAKE in python hello msg;

Now, here is what I got:

when I first run the editor, it connected:

as you can see, it says recived contact from iphone2python.

Then I stopped it, and run again in editor, now it shows nothing:

the livelink windows does not have a active source, and the warning message keep got format error. deserialized error.

Please help, my iOS device also have this issue, which I don't know why, I bought the app but I can not get it work fluently. Wanna learn something from it, but this problem drive me crazy, I really want figure it out, can u help me?

[PoseAI]

Hi, it seems you are claiming two issues:

1. Your Python script. It doesn’t happen on our system so it would be helpful if you did some more diagnostics outside of unreal. As we suggested before, inspect your json packet outside of unreal to see if the extra byte is occurring there
2. you say you are having an issue with iOS. Can you please disable your firewall, use packet format 1 in the handshake and see if that resolves?

[lucasjinreal]

@PoseAI Hi, I switched to JsonLivelink plugin, it now seems can receive any of my json msg now. But when I using this to send subjectName to Livelink, there is no subjectName shows in livelink window, can u kindly give me some hints?

if (PoseAIRig::IsFrameData(JsonObject))
        {
            UE_LOG(LogTemp, Warning, TEXT("[magic] this is a framedata, sending to subject target..."));
            // source_->UpdatePose(rig, jsonObject);
            FLiveLinkFrameDataStruct frameData(FLiveLinkAnimationFrameData::StaticStruct());
            FLiveLinkAnimationFrameData& data = *frameData.Cast<FLiveLinkAnimationFrameData>();
            data.Transforms.Reserve(100);
            if (rig->ProcessFrame(JsonObject, data))
            {
                Client->PushSubjectFrameData_AnyThread(subjectKey, MoveTemp(frameData));
                UE_LOG(LogTemp, Warning, TEXT("sent frame data to: %s %s"), *subjectKey.SubjectName.ToString(),
                       *subjectKey.Source.ToString());
            }
            // UPoseAIEventDispatcher::GetDispatcher()->BroadcastFrameReceived(source_->GetSubjectName());
        }

this function PushSubjectFrameData_AnyThread seems doesn't give me a SubjectName at window:

[PoseAI]

We are trying to get to the bottom of the deserialization error as that seems the root of the problem, and one other user has reported something similar. We haven't been able to replicate the error ourselves, but can you tell us if you built either UE5 or the plugin from source, or whether you were using the Epic Games installer / pre-built binaries?

[PoseAI]

@PoseAI Hi, I switched to JsonLivelink plugin, it now seems can receive any of my json msg now. But when I using this to send subjectName to Livelink, there is no subjectName shows in livelink window, can u kindly give me some hints?

if (PoseAIRig::IsFrameData(JsonObject))
      {
          UE_LOG(LogTemp, Warning, TEXT("[magic] this is a framedata, sending to subject target..."));
          // source_->UpdatePose(rig, jsonObject);
          FLiveLinkFrameDataStruct frameData(FLiveLinkAnimationFrameData::StaticStruct());
          FLiveLinkAnimationFrameData& data = *frameData.Cast<FLiveLinkAnimationFrameData>();
          data.Transforms.Reserve(100);
          if (rig->ProcessFrame(JsonObject, data))
          {
              Client->PushSubjectFrameData_AnyThread(subjectKey, MoveTemp(frameData));
              UE_LOG(LogTemp, Warning, TEXT("sent frame data to: %s %s"), *subjectKey.SubjectName.ToString(),
                     *subjectKey.Source.ToString());
          }
          // UPoseAIEventDispatcher::GetDispatcher()->BroadcastFrameReceived(source_->GetSubjectName());
      }

this function PushSubjectFrameData_AnyThread seems doesn't give me a SubjectName at window:

i would guess since you are using your own processor for the json packets, you did not set up the subject key. it is set up in the ReceiveClient method.

[lucasjinreal]

thanks sir, ReceivClient just assigned a SourceUID, which part is mainly for register such as subjectKey?

I am using UE5.0.3 prebuilt binary on macOS. the plugin is build from source myself.

[PoseAI]

Ok, MacOS was the missing piece of the puzzle! We were checking on Windows...

We can replicate the issue on MacOS and we just uploaded a fix here on github for UE5.0 (we haven't ported to 4.26/4.27 yet). Can you try using our plugin from source, don't use your own json code and see if this resolves the issue?

[lucasjinreal]

@PoseAI thanks for the fix sir. May I ask why does FString -> UTF8CHAR do the magic? that's a good catch.

what's more, how to explain that this error only happens sometimes? I am really still very confused.

[PoseAI]

The previous version relied on an Unreal Macro, so it wasn't clear what was happening under the hood, and it was combined with a rather hacky solution from our side to terminate the string.

While investigating the issue we discovered that the socket calls (Unreal implementation) were behaving differently in MacOS and Windows, with very different byte sizes returned for the same packet across the two platforms. MacOS was recording a larger size than it was actually reading. Generally Epic highlights that all their socket stuff is "temporary", but I may report the issue to Epic when I have the opportunity to delve further.

For the fix. we instead decided to stop using their macro, explicitly specify as UTFCHAR and explicitly set the size to the number of bytes actually returned by the recv call. Hopefully this fix is more robust.

We have had other insidious bugs in MacOS Unreal before with memory, including one which the Epic marketplace team tried to resolve but could never fix it themselves either. To answer the "sometimes" - my best guess is the socket reader was reserving a larger buffer based on a larger byte count, which got included in the macro conversion routine. If those extra bytes were in vacant memory, then this would be whitespace and no error would happen, but if they had problematic data written in those addresses then it would add extra unreadable characters which stumped the json reader. So sometimes you had vacant memory and sometimes not.

I will close this for now, and if I don't hear back from you soon I will assume this is resolved and will submit a patch to the marketplace. Thanks for raising the issue and your patience while we resolved.

[lucasjinreal]

@PoseAI thank u sir. For you notice, I just double tested new version can received data normally all the time on macOS!

[PoseAI]

Looks like you have an extra byte being added to the end of your packets… depending on the byte it would often confuse the deserializer. I suspect it is in your dumps + encode calls, you may be able to just send dumps output (or try turning the ascii only setting if necessary).
I don’t know the module you are using to manage the socket but I would check in Python just before it goes through to socket that you are not sending a packet with an extra character due to your encoding

On 17 Jul 2022, at 15:13, JinTian @.***> wrote:

 @PoseAI Hi, thanks for your reply, would like help me to debug this, I think I roughly find where goes wrong, but I don't know why:

You can reveal this sometims maybe (not every time which is the toughest one).

I using Python to send a handshake to UE5 livelink server, the server side log this:

{"version": "1.2.5", "userName": "iPhone2Python", "sessionUUID": "8EF5BA6A-A73E-419A-9BDC-7E18759C180C", "deviceName": "iPhone13,4"}� , error: Unexpected additional input found. Line: 1 Ch: 132 Live Link: Warning: （01） PoseAI: failed to deserialize json object from 127.0.0.1:54386 Live Link: Warning: （05） PoseAI: Incoming connection does not have a hello handshake For better investigate I post a screenshot here:

I just using UE_LOG logged the recivmsg text content, you can see a block A bytes at the normal json str, Every single frame (except for handshake, the normal data), got same error:

error: Unexpected additional input found. The python I tested is very simple:

import socket import json from pprint import pprint from llv.socketconn import SocketConn import time from alfred import logger

PORT_NUM = 8080

""" Prints your local IP address. Configure this in the App. Make sure your router and firewall do not block the port """

def show_my_ip(): s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) try: s.connect(("10.255.255.255", 1)) ip = s.getsockname()[0] except Exception: ip = "127.0.0.1" finally: s.close() print("Connect the app to IP:", ip, "Port:", PORT_NUM)

handshake = { "version": "1.2.5", "userName": "iPhone2Python", "sessionUUID": "8EF5BA6A-A73E-419A-9BDC-7E18759C180C", "deviceName": "iPhone13,4", }

show_my_ip()

connections = {}

socket_conn = SocketConn("127.0.0.1", port=8080) socket_conn.send_json_encoded(json.dumps(handshake).encode()) the Socket here is not as server but as client.

So here is my confusion:

Why python same handshake decoded json error same as iOS client does? It should have something wrong cooperated with UE json reader; But this is happend only at 90% time, another 10% time the handshake is normal, but the frames data 100% decode error. Hoping for your answer if you have time help me have a look.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.