Question regarding changes to the way `Quazal::String` types are encoded

[jonbarrow]

I'm the lead developer over at Pretendo. Right now @DaniElectra and I are currently in the process of rewriting all of our core NEX libraries, and one of our goals is to achieve compatibility with clients using the original Rendez-Vous library, that way developers like yourself no longer need to have the burden of maintaining your own forks

Right now the rewrite for nex-go is working with Rayman on the WiiU, which uses the original Rendez-Vous library instead of NEX. So we are fairly confident that we have achieved compatibility

I decided to check out your forks of our libraries to see if there were any major changes you made that we missed when adding in support for these clients, and noticed that you added stream methods for Quazal::String types that use a uint32 as the length rather than a uint16

Can you add some clarifications on this? I saw that this was used in your jsonservice implementation. Are you sure that these are actually Quazal::String types, and not just Quazal::Buffer types using JSON strings as the buffer data?

I know that GoCentral is designed for Rock Band 3, so I attempted to look into the game to see if I could verify these changes. Unfortunately, Rock Band 3 does not seem to have symbols and my lack of experience with Wii games became a roadblock

I did, however, find that Rock Band 2 ships with debug symbols, and when checking that game it does still use a uint16 for Quazal::String types. We find it odd that they would change the way these are encoded when making Rock Band 3

Dani and I are looking to confirm this change in encoding so that we can properly decide on if it's something we should handle in our rewrite. Any information or guidance from you would be appreciated on the subject, especially some pointers on where to look in Rock Band 3 to find the updated encoding

[ihatecompvir]

@jonbarrow thanks for reaching out.

I am pretty sure (like 99.9%) that the Quazal::String type now has a uint32 length as of RB3 and that it isn't just a Quazal::Buffer containing a string. My reasoning is because of a few things:

• The original ReadString function is essentially unused code in GoCentral - I constantly had to change the code to use the uint32 variant instead, you can see this in some of the non-JSON methods such as RegisterEx, Login, and some others, indicating that the core String type is different
• In RB2's Quazal::_Type_string::Add function, it gets the length of the string using Quazal::String::GetLength and then uses BinStream::Write functions to write it to memory, and they write it with a length of 2 bytes. In RB3, the same BinStream::Write call uses 4 bytes. The same occurs when strings are being read, you'll see it go from 2 bytes in RB2 to 4 bytes in RB3.

Because of that I can pretty safely say this is just how it is in RB3. It definitely seems strange that they would switch from uint16 to uint32 prefixed strings, and I still wonder why they changed it.

Also, you are correct in saying RB3 has no symbols. That being said, I've put in a lot of time and effort to try to match functions based on the Green Day: Rock Band and Rock Band 2 symbols and have labeled a ton of stuff now as a lot of functions did not change too much, however the lion's share of my RE research is for the 360 version of Rock Band 3 but I do have the above-mentioned functions labeled for the Wii version if that helps:

BinStream::Write(BinStream this, void data, int size); = 0x80342f48 Quazal::_Type_string::Add = 0x80054094 Quazal::String::GetLength = 0x80018650

Let me know if I can help with anything else and thank you all for taking this rewrite on. I greatly appreciate it and the day I can deprecate my now several years out-of-date nex-go fork will be a day for celebration.

[jonbarrow]

Thanks for the information! I'll definitely be sure to check out Rock Band 3 using those symbols. Not that I don't believe you, of course, just to have another set of eyes on it

Some Ubisoft games, for example, can change the size of packet checksums on the fly by changing the packets configured checksum size. I'm curious to see if something like that is also implemented here, or if they really did just change it globally

The only thing Dani and I can guess as to why a change like this would be made is if something in this new jsonservice used strings larger than what a uint16 would allow. Though I find it hard to believe that the game is sending over 65kb of JUST strings

Also do feel free to try out our rewrite branches if you'd like! We've done limited testing with Rendez-Vous clients, only really trying Rayman on the WiiU. More tests are also good

[jonbarrow]

I've gone ahead and updated our rewrite to account for this. They really did just update the size for some reason. Our new library now supports configuring the size of the length field, so it should be usable to you now

[ihatecompvir]

@jonbarrow awesome, I will definitely be trying this out to see how things are currently working.

I do have a question about the rewrite (and this is getting a bit off-topic for this issue, I know) but is there support for incoming fragmented packets? Almost every JSON message is large enough to be fragmented, this was an issue I ran into with the original nex-go and I had to come up with a rather buggy solution to put the fragments together before passing it to the handler, as it is critical that I handle all JSON messages.

[jonbarrow]

(and this is getting a bit off-topic for this issue, I know)

No worries, I don't have a problem with it. If you're worried about clogging up this issue though (which, is technically solved and can be closed), you can always create one on our repo or reach out on Discord

is there support for incoming fragmented packets?

Yes. The rewrite is a total rewrite from the ground up. Your forks are based on a version that started development before PRUDP was even finished being documented. So is was cobbled and hacked together over the years as we discovered new things, which led to it being a mess of a code base that was a nightmare to maintain

The new rewrite started from scratch with these things in mind. PRUDP is, as far as we know, fully documented at this time. So that's the new starting point the library has. We went into it with the goal of fixing those things, it was designed with stability and being feature complete from the start

[ihatecompvir]

Awesome, thanks for clarifying. Closing this out now, any future questions or etc. I'll just put in Discord