Cursor received from listUsersStorageObjects appears to be garbled

[DeoVolenteGames]

To test I'm calling listUsersStorageObjects with a small limit and logging the cursor. My code looks something like:

auto SuccessCallback = [&, this](NAKAMA_NAMESPACE::NStorageObjectListPtr ObjectListPtr)
{
    UE_LOG(LogNakamaBPExtension, Log, TEXT("Nakama cursor received: %s"), UTF8_TO_TCHAR(ObjectListPtr->cursor.c_str()));
    // Other stuff
};

NakamaBP::Client->listUsersStorageObjects(
    NakamaBP::Session,
    std::string(TCHAR_TO_UTF8(*Collection)),
    std::string(TCHAR_TO_UTF8(*UserID)),
    Limit > 0 ? optional<int>(Limit) : nullopt, // Limit is set to 3 or 4 in tests
    Cursor.IsEmpty() ? nullopt : optional<std::string>(TCHAR_TO_UTF8(*Cursor)), // Cursor is an FString
    SuccessCallback,
    ErrorCallback
);

This the call fires correctly with an empty cursor. However, my logs return: LogNakamaBPExtension: Nakama cursor received: ??ȟ?ഠ綊ƨ or similar (it's never the same value). If I try and make another call with the cursor I receive, I get the error:

[Nakama::RestClient::reqError] NError: InvalidArgument
message: Malformed cursor was used.

I also haven't been able to find an example of what a cursor is supposed to look like, but I assume it's a UUID or something? If I knew what it was that might help me figure out what's going wrong. Thanks for any assistance!

[lugehorsam]

@DeoVolenteGames cursors are binary data in Golang's gob format that are base64 encoded when communicated to/from the client. So from the client's perspective, cursors are just ASCII strings.

It looks like you're trying to re-decode these ASCII strings to UTF-8 with your TCHAR_TO_UTF8 calls. What happens if you remove those?

[DeoVolenteGames]

That's useful to know, thanks! Text encoding is something I don't think about a lot admittedly. I'm converting these to FStrings so that they are accessible in blueprints, but if they're just base64 I guess can just ignore that. I could pretty easily wrap std::string in a struct and then blueprint users could pass those around instead. I'll try that a bit later.

The weird thing to me is that I feel like this should work. If I remove the TCHAR_TO_UTF8 macro then UE4 just assumes they are UTF-16 and mashes them together, so a 6 character std::string becomes ꢘꐺ翿 as an FString. I've tried to see what the actual values are (just in case, idk, UE_LOG is doing something funky) and when I try to just log each character as uint8 I'm not getting results in the base64 range of ASCII characters. However, I'm not much of a C++ coder so I'm likely making a very silly mistake somewhere in my limited understanding of C, C++, and UE4 string handling.

Example code:

// Control example
std::string test = "base64";
UE_LOG(LogNakamaBPExtension, Log, TEXT("Nakama cursor test to string: %s"), UTF8_TO_TCHAR(test.c_str()));
for (char& c : test) {
    UE_LOG(LogNakamaBPExtension, Log, TEXT("Nakama cursor test to uint8: %d"), static_cast<uint8>(c));
}

// In success callback
UE_LOG(LogNakamaBPExtension, Log, TEXT("Nakama cursor to string: %s"), UTF8_TO_TCHAR(ObjectListPtr->cursor.c_str()));
for (char& c : ObjectListPtr->cursor) {
    UE_LOG(LogNakamaBPExtension, Log, TEXT("Nakama cursor to uint8: %d"), static_cast<uint8>(c));
}

Example output:

LogNakamaBPExtension: Nakama cursor test to string: base64
LogNakamaBPExtension: Nakama cursor test to uint8: 98
LogNakamaBPExtension: Nakama cursor test to uint8: 97
LogNakamaBPExtension: Nakama cursor test to uint8: 115
LogNakamaBPExtension: Nakama cursor test to uint8: 101
LogNakamaBPExtension: Nakama cursor test to uint8: 54
LogNakamaBPExtension: Nakama cursor test to uint8: 52

LogNakamaBPExtension: Nakama cursor to string: ??:??쎜翿
LogNakamaBPExtension: Nakama cursor to uint8: 152
LogNakamaBPExtension: Nakama cursor to uint8: 168
LogNakamaBPExtension: Nakama cursor to uint8: 58
LogNakamaBPExtension: Nakama cursor to uint8: 164
LogNakamaBPExtension: Nakama cursor to uint8: 255
LogNakamaBPExtension: Nakama cursor to uint8: 127

[lugehorsam]

@DeoVolenteGames I think some of what you're seeing is an artifact of logging strings in either Windows or UE4. So I'd expect you to see some unexpected characters when logging them.

With cursors in Nakama, the only important thing is that it is sent back to the server in an untampered manner. So could you pass the std::string that you receive from the server to listUsersStorageObjects without any c_str() or UTF8_TO_TCHAR calls and see if you get a Malformed cursor was used. error? Of course, you'll also want to make sure you are actually receiving a cursor from the server (e.g., you are requesting a number of items storage objects that is greater than the limit value you provide.)

[DeoVolenteGames]

@lugehorsam Thanks for your reply. I've tried the simplest possible case: sending a listUsersStorageObjects request in the success callback of a listUsersStorageObjects request, directly using the ObjectListPtr->cursor that is returned. The error I get is:

[Nakama::RestClient::listUsersStorageObjects] exception: UTF-8 string character can never start with 10xxxxxx

It seems to simply fail immediately. I also checked my test server and it has 10 storage objects in a collection. During testing I ask for and successfully retrieve 4 items when calling listUsersStorageObjects the first time, so the cursor should be valid. When I use a cursor that is returned from calling listUsersStorageObjects with a limit that is too large then I get:

[Nakama::RestClient::listUsersStorageObjects] exception: UTF-8 continuation byte is missing leading bit mask

When I dug a little deeper I found that the first error seems to be called from createWebSocketCppRest(), just in case it's relevant. Also, my Nakama server version is 3.1.1 and my Nakama UE4 client version is 2.4.0.

[erlend-sh]

(Sidebar: @DeoVolenteGames I sent you an email via your contact form to learn more about the game you’re working on. If it didn’t arrive you can find my email in my GH profile.)

[lugehorsam]

@DeoVolenteGames it looks like your new encoding issue occurs in the cpprestsdk if you use a mismatched Debug/Release .dll: https://github.com/microsoft/cpprestsdk/issues/1142#issuecomment-636398350

Could you confirm that you are using the right .dll (debug or release, depending on your Visual Studio configuration) in the Additional Dependencies section referred to by our README?

[lugehorsam]

I've also added a passing cursor test to our general test suite for the cpp client:

https://github.com/heroiclabs/nakama-cpp/commit/e1206977cc42e4f8d17bd3a52d66a1f373e08f06

[DeoVolenteGames]

@lugehorsam Though I'm aware that the UE4 plugin is based off the Nakama general C++ client, I'm not sure how I can easily access the dlls that you're referring to. In the UE4 client release that I'm using nakama-cpp.lib and nakama-cpp.dll are precompiled and included in the archive. There are no references to other Debug or Release libs or dlls as far as I can see, except for the standard binaries UE4 creates for Debug/Development/Release. I don't think I can access what you're referring to without downloading from source and figuring out how all that works. I just want to check before I go down a rabbit hole, am I on the right track here? I feel like I'm missing something.

Also, I'm using Rider for UE4, but if needed I can go back to Visual Studio for this.

[marot]

@lugehorsam It looks like the C client implementation does not copy the cursor from the cpp type, see https://github.com/heroiclabs/nakama-cpp/blob/e4b41656e4185d4aa2d993c5a585ebbbbf5eade5/src/nakama-c/DataHelperC.cpp#L642

It is missing cObjList.cursor = objList.cursor.c_str();

[lugehorsam]

@DeoVolenteGames this should now be fixed on https://github.com/heroiclabs/nakama-cpp master and will be in the next unreal release.