Not connecting to Pretendo

[Anonymous941]

When connecting to Pretendo (IP 116.202.98.9, port 60003), it gives this error during login:

RuntimeError: RMC connection is closed

[kinnay]

Could you post the access key and NEX version of that server? I would need a bit more information to debug this.

[Anonymous941]

According to the Pretendo source code, the access code is 9f2b4678 and I think the NEX version is 3.8.3 based on this line:

globals.AuthenticationServer.SetDefaultNEXVersion(nex.NewPatchedNEXVersion(3, 8, 3, "AMAJ"))

[kinnay]

The Pretendo server sends a DISCONNECT packet after NintendoClients sends the log in request:

DEBUG:nintendo.nex.rmc:Connecting RMC client to 116.202.98.9:60003:1
DEBUG:nintendo.nex.prudp:Connecting PRUDP transport to 116.202.98.9:60003
DEBUG:anynet.udp:Connecting UDP client to 116.202.98.9:60003
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_SYN flags=NEED_ACK seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_SYN flags=ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_CONNECT flags=NEED_ACK,RELIABLE,HAS_SIZE seq=1 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_CONNECT flags=ACK,HAS_SIZE seq=1 frag=0>
INFO:nintendo.nex.authentication:AuthenticationClient.login()
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_DATA flags=NEED_ACK,RELIABLE,HAS_SIZE seq=2 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DATA flags=MULTI_ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DISCONNECT flags= seq=0 frag=0>

Maybe @jonbarrow knows why this happens.

[jonbarrow]

The Pretendo server sends a DISCONNECT packet after NintendoClients sends the log in request:

DEBUG:nintendo.nex.rmc:Connecting RMC client to 116.202.98.9:60003:1
DEBUG:nintendo.nex.prudp:Connecting PRUDP transport to 116.202.98.9:60003
DEBUG:anynet.udp:Connecting UDP client to 116.202.98.9:60003
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_SYN flags=NEED_ACK seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_SYN flags=ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_CONNECT flags=NEED_ACK,RELIABLE,HAS_SIZE seq=1 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_CONNECT flags=ACK,HAS_SIZE seq=1 frag=0>
INFO:nintendo.nex.authentication:AuthenticationClient.login()
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_DATA flags=NEED_ACK,RELIABLE,HAS_SIZE seq=2 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DATA flags=MULTI_ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DISCONNECT flags= seq=0 frag=0>

Maybe @jonbarrow knows why this happens.

I'll have to take a deeper look into this. We just recently did a large overhaul of our libraries, which may have introduced this regression, but just taking a quick glance the only time we send a DISCONNECT packet is as an ACK to the one the client sends?

The Super Mario Maker server is running on an older version of the libraries, however, so I'll see what I can dig up in some older commits

[jonbarrow]

I'm gonna CC @DaniElectra as well on this as he was the one doing the overhaul with me

[Anonymous941]

This is interesting: https://github.com/PabloMK7/nex-go/blob/9d6e91cbde41c1eba5411cc208534971dd1b2603/server.go#L206-216

        // * PID should always be 0 when a fresh connection is made
        if client.PID() != 0 {
            // * Was connected before on the same device, using a different account
            server.Emit("Disconnect", packet) // * Disconnect the old connection
        }
        err := client.Reset()
        if err != nil {
            // TODO - Should this return the error too?
            logger.Error(err.Error())
            return nil
        }

[jonbarrow]

This is interesting: PabloMK7/nex-go@9d6e91c/server.go#L206-216

      // * PID should always be 0 when a fresh connection is made
      if client.PID() != 0 {
          // * Was connected before on the same device, using a different account
          server.Emit("Disconnect", packet) // * Disconnect the old connection
      }
      err := client.Reset()
      if err != nil {
          // TODO - Should this return the error too?
          logger.Error(err.Error())
          return nil
      }

The repository you're looking at is a fork of ours. It may not be representative of what our upstream repository looks like. These specific lines do exist in older commits, but you shouldn't use forks as a reference for how our production servers would operate. You should be looking upstream. https://github.com/PretendoNetwork/nex-go

Also this fork is extremely outdated

On top of that, these lines are irrelevant to the actual issue at hand. The Emit function simply sends out an event to any listeners of the event type. It doesn't do anything beyond sending that signal. How the event is handled is an implementation detail of the listener

[Anonymous941]

The repository you're looking at is a fork of ours. It may not be representative of what our upstream repository looks like. These specific lines do exist in older commits, but you shouldn't use forks as a reference for how our production servers would operate. You should be looking upstream.

Got it, I'm not sure why I was looking at that fork

[DaniElectra]

It looks like the server is timing out the connection, since the the DISCONNECT packet doesn't have FLAG_RELIABLE set. Not sure why it would be happening on a first glance though

BTW probably unrelated to the original issue, but we enforce the usage of LoginEx on the auth servers, with the only exception being the friends server for now

[kinnay]

I tested it with Login before, but the same thing happens if I use LoginEx:

DEBUG:nintendo.nex.rmc:Connecting RMC client to 116.202.98.9:60003:1
DEBUG:nintendo.nex.prudp:Connecting PRUDP transport to 116.202.98.9:60003
DEBUG:anynet.udp:Connecting UDP client to 116.202.98.9:60003
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_SYN flags=NEED_ACK seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_SYN flags=ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_CONNECT flags=NEED_ACK,RELIABLE,HAS_SIZE seq=1 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_CONNECT flags=ACK,HAS_SIZE seq=1 frag=0>
INFO:nintendo.nex.authentication:AuthenticationClient.login_ex()
DEBUG:nintendo.nex.prudp:[CLI] Sending packet: <PRUDPPacket type=TYPE_DATA flags=NEED_ACK,RELIABLE,HAS_SIZE seq=2 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DATA flags=MULTI_ACK,HAS_SIZE seq=0 frag=0>
DEBUG:nintendo.nex.prudp:[CLI] Received packet: <PRUDPPacket type=TYPE_DISCONNECT flags= seq=0 frag=0>

Here is the script that generates these logs with LoginEx:

from nintendo.nex import backend, settings, authentication
import anyio

import logging
logging.basicConfig(level=logging.DEBUG)

async def main():
    s = settings.default()
    s.configure("9f2b4678", 30803)
    auth_info = authentication.AuthenticationInfo()
    auth_info.token = "test"
    async with backend.connect(s, "116.202.98.9", 60003) as be:
        async with be.login("test", "test", auth_info):
            print(be)

anyio.run(main)

[Anonymous941]

Is this a bug with Super Mario Maker or the other games as well?

[DaniElectra]

I did some tests on my side and I have found the issue. You were trying to connect to the secure server directly, not the authentication server. The auth server is located on port 60002, not 60003. After changing that, everything works on my side

[Anonymous941]

@DaniElectra Thank you so much, it finally seems to be working! But I'm not sure how to dump my Pretendo NEX credentials on a 3DS - the server rejects the normal credentials obtained via get_3ds_pid_password, so I assume it's dumping from prod, not test

[jonbarrow]

That's probably pretty likely tbh. @DaniElectra maybe Nimbus could be updated to dump this?

[Anonymous941]

I was able to manually extract partitionA.bin from nandctr:/data/<id0>/sysdata/00010032/00000000, then at offset 0x10000 I found an account struct that had the test NASC environment.

After manually extracting it and decoding the password as UTF-16, and the PID as decimal, I got the same result as get_3ds_pid_password, but the password has an extra character.

So I entered these credentials into the SMM archival script, but it still fails to authenticate (Authentication::ValidationFailed (0x80680007)). I have no idea why. I tried every combination of entering my the PrincipalID and HMAC, the password with and without the extra character, encoding the HMAC as decimal, but nothing works. The PID matches exactly with my packet captures, and the password is never sent directly so I can't verify that that's exactly right

What is going on here? Why won't the Pretendo server accept my credentials?

[Preloading]

From my experience, get_3ds_pid_password seems to work fine for getting both nintendo, and pretendo keys

[Anonymous941]

@Preloading That's weird. It seems to be getting the right credentials, but the server just keeps sending Authentication::ValidationFailed (0x80680007)

[DaniElectra]

Are you passing the AuthenticationInfo to the login function? As I have said, our servers require LoginEx on all servers (except for the friends server, since the console uses Login instead)

[Anonymous941]

@DaniElectra This seems to be working! I'm getting another error, but I can't tell if that's because I passed an invalid token, as the value doesn't actually change anything. What's the correct value for it? The docs just name it...

The error I'm getting is nintendo.nex.common.RMCError: Core::NotImplemented (0x80010002), when running datastore_smm_client.search_object(param)

[jonbarrow]

As the error says, that RMC method is not implemented on the server

Unlike Nintendo's servers ours only implement what the games are known to use for simplicity

[Anonymous941]

I found PretendoNetwork/nex-viewer/src/protocols/patches/datastore_smm.js, and this seems to contain the information I need. I'll try to experiment with this

Can there just be some way to download an archive of the Pretendo SMM courses? I'm trying to allow searching and downloading with only the 3DS version of SMM, and the only method I can think of is to scrape them like this

Also, does the token actually matter? Because I'm getting the same results no matter what it's set to, and have no idea what it's supposed to be (packet dumps don't contain that as far as I can tell)

[Anonymous941]

Okay, this is strange, I'm sending custom_search_object (0x2f), and it's still returning nintendo.nex.common.RMCError: Core::NotImplemented (0x80010002) regardless of this being defined in the source code. What's going on?

[jonbarrow]

I found PretendoNetwork/nex-viewer/src/protocols/patches/datastore_smm.js, and this seems to contain the information I need

No, it doesn't. This is a viewer, as indicated both in the repository name (nex-viewer) and in the README of the repository:

Utility for parsing and (eventually) viewing NEX connections from PCAP(NG) network dumps

This has nothing to do with what is and isn't implemented on our servers, and a lot of what is implemented in the viewer was made with codegen

Can there just be some way to download an archive of the Pretendo SMM courses?

As stated you can only use the methods available on a per-server basis. Methods only get implemented in game servers if they're seen in use in the game, in order to simplify game server setups. Currently the methods you're trying to use are not implemented. So, no. At the moment there isn't

I'm trying to allow searching and downloading with only the 3DS version of SMM, and the only method I can think of is to scrape them like this

You're making 3DS game patches that can use Python scripts? It sounds like it would be easier to just patch the game itself than do this

Also, does the token actually matter?

Not all of our servers validate the token atm because we are changing how we handle token generation fairly quickly

Okay, this is strange, I'm sending custom_search_object (0x2f), and it's still returning nintendo.nex.common.RMCError: Core::NotImplemented (0x80010002) regardless of this being defined in the source code. What's going on?

No, it's not strange. Again. Only RMC methods fully implemented on our servers can be used. If you're getting Core::NotImplemented it means the server just simply doesn't support that method. That's the only time it ever returns that error

Please actually check what is and isn't implemented, you can find our Super Mario Maker specific methods in the Super Mario Maker server, and for common DataStore methods those are implemented in our common implementations library. Be aware that just because a method is implemented in the common library does not mean a game server uses it. If the game server is not configured to use those methods (by not providing the fields/helpers the method needs) then it is considered not implemented. In the case of DataStore::SearchObject that method relies on the GetObjectInfosByDataStoreSearchParam helper, which our Super Mario Maker server does not currently provide

[Anonymous941]

Thank you for your help so far

This has nothing to do with what is and isn't implemented on our servers, and a lot of what is implemented in the viewer was made with codegen

Got it, I misunderstood that file then. Which method is used by the Wii U when it just normally searches for files? If you don't know, how can I figure this out myself without a Wii U?

As stated you can only use the methods available on a per-server basis. Methods only get implemented in game servers if they're seen in use in the game, in order to simplify game server setups. Currently the methods you're trying to use are not implemented. So, no. At the moment there isn't

Could you consider just allowing you to download a .zip with the uploaded Pretendo courses somewhere (maybe an HTTP endpoint)? Although there's probably some reason I'm not thinking of that would make this difficult...

You're making 3DS game patches that can use Python scripts? It sounds like it would be easier to just patch the game itself than do this

I'm making a Python script that can search and download from Pretendo's servers, then edit the ExtData to inject those courses. It can currently convert a local Wii U save file, so I'm hoping I can just download something similar from Pretendo. But since I don't own a Wii U, just a 3DS, it's been extremely difficult

[jonbarrow]

Which method is used by the Wii U when it just normally searches for files?

This depends on the game and it's implementation. It's not a Wii U vs 3DS thing. I don't remember off the top of my head which method Super Mario Maker uses specifically, there's several of them and that game even implements custom search methods

Could you consider just allowing you to download a .zip with the uploaded Pretendo courses somewhere (maybe an HTTP endpoint)? Although there's probably some reason I'm not thinking of that would make this difficult...

That's not something we're likely going to be interested in any time soon. The server doesn't have the concept of a "course". It only knows about "objects", and sometimes those don't even have real files associated with them. Games just upload and download objects from the server, and handles them however they need to. As far as the server is concerned they're all pretty much the same thing, stored exactly the same way regardless of object type (more than just courses are objects here)

It's not as simple as just "put the courses in a zip". The server would need to process the data of every object in the database, determine if it is or isn't a course (which is not always so clear), collect it's metadata, pack it into something usable, and then add both the metadata and object data to the zip. This will take a good amount of time, not to mention likely provide you with a fairly large download. Our archive of Nintendo's data was over 500GB. While we aren't anywhere near that right now, it's not like this would be tiny. A rough estimate based on our current data would be \~200MB of just object data, which while not massive is large enough for us to think about sending over the network. And that doesn't include any of the object metadata. This will only keep growing larger and larger as time goes on