Support for encrypted clients

[joaoescribano]

I've found some information about encrypted clients....

http://necrotoolz.sourceforge.net/kairpacketguide/encryption.htm https://code.google.com/p/majesticuo/wiki/LoginEncryption

But only for the first connection communication... after that, we must decrypt messages before process packets using TWOFISH algorithm (probably with PHP mcrypt) as well encrypt before sending it to the client.

TODO LIST

• [x] First connection decrypt algorithm
• [x] Client key fully list
• [x] Client keys detecting
• [ ] Decrypt the packet 0xA0 and others after client select server and so...
• [ ] Second connection decrypt using TWOFISH algorithm (0%)

----------------------- 2019 update, 4 years later I've been working on this issue, but as far I can tell the entire socket server need to be rewritten from scratch.

The bizarre thing that I can't figure out is that UltimaPHP is able to decode the first packet 0x80 and send the server list to the client as we can see here:

But after that, the decrypt goes crazy and don't work at all:

To me, it seems that the keys or the seed of the socket changes at the second socket connection to the client and I don't find a way to figure it out.

[joaoescribano]

Started to work again on it :+1:

[MuhsinAlaca]

Hi Teens I've been dealing with this extraordinary work for a long time. but, from what I see, you seem to have taken a break? I hope you continue this work. As a matter of fact, Ultima Online is like a philosophy of life, as a 49-year-old Ultima Online regular, I still follow with curiosity, I think I'm ready if I have a position I can contribute to. I wish you continued work and wish you happiness. I've been thinking about UltimPHP for a long time, but I couldn't manage to connect, and I would appreciate it if you could help me with that, thanks ...

[joaoescribano]

@MuhsinAlaca thanks for the feedback. We had to take a break but we still willing to get back on rails as soon as we can, but life have been a little rush these last months.

About your issue, it seems that you are using an "encrypted version" of the client, and we didn't completed solved the encryption from Ultima Online yet, so now we still need to use UltimaPHP with an unencrypted client.

You can build and use a tool called SUMP - Simple Ultima Multi Patcher (or download it) and remove the client.exe encryption, you also can use assists to do it, like Razor or UOSteam.

After that, you can properly login the server.

[MuhsinAlaca]

Thank you for the return, my attempt to connect with both steam and razor failed, I'm still grateful for your ideas, I hope you continue your work as soon as possible ...

[joaoescribano]

Don't worry we're here to help :+1:

But did you checked the box that says "remove encryption"? I really think you should download SUMP and remove the client encryption entirely (just remember to back up it first), sump will create a new file called "client_patched.exe" in the Ultima Online folder and this will work for sure!

Let me know if it worked

[MuhsinAlaca]

I've tried many ways since my last message, but again the same problem, my dear brother Joaoescribano, I don't want to bother you much! I would be grateful if you have any other suggestions or would like to connect to my network.

[MuhsinAlaca]

Latest situation What do you think this way? First Screen Last Screen

[gokaygurcan]

I remember it was a b.tch to get that working. Here's an easy implementation in JavaScript to give you some ideas to get it working with encrypted clients: https://github.com/gokaygurcan/poc/blob/develop/index.js#L58-L77

[mbnunes]

You chance your Login.cfg?

[MuhsinAlaca]

Login.cfg şansınız var mı?

I remember it was a b.tch to get that working. Here's an easy implementation in JavaScript to give you some ideas to get it working with encrypted clients: https://github.com/gokaygurcan/poc/blob/develop/index.js#L58-L77

Brother, I've reviewed the link you suggested, but I haven't received any results yet.

[MuhsinAlaca]

You chance your Login.cfg?

Login.cfg as we know "LoginServer=127.0.0.1,2593" Always check the same fault connector 168. "if (is_object($packet)) {"

[joaoescribano]

Sorry, I've been all day busy...

@gokaygurcan thanks for sharing, I really understand better JS than C and C++, so it may really help!

Let me think a little bit about it, as far I can remember there is a difference between the windows socket libraries from Linux/Unix and since I only develop at Linux, I really need to think about what I can be.

@MuhsinAlaca can u please set UltimaPHP to debug=True at the config file and share the packet dump?

From the screenshot you shared, I can see everything is pretty good and well configured, so don't need to worry anymore about what can be wrong at your environment, it is good! This may be more related to the windows socket library than anything.

I'll download everything needed to run on windows and try to figure it out, I'll update this post as soon I figure it out.

[joaoescribano]

@MuhsinAlaca I've made it work on windows 10, but really need to try it.

Update your code and please try to run it again, it works please let me know.

I'll try this source now on Linux, to see if it keeps working. -------------------------------------------------------------------------------_ Yes, it is working on Linux as well:

[joaoescribano]

Getting back to the encryption issue, I've been trying to implement it since the beginning, the first login packet 0x80 I can decrypt but the next packets the keys or the seed seem to be changed somehow and I can't figure out why and how to properly do it.

Another issue is that the "first" (0xEF) packet (it's not actually a packet, but the seeds and so), have a bizarre behavior, sometimes it comes complete (21 bytes) sometimes it comes with 1 byte and next a 20 bytes packet arrives and sometimes it just doesn't send the packet command byte (0xEF):

many things happen here and I really think the entire socket server needs to be redone to fix it, as far I can see.

@gokaygurcan I've converted the calculateKeys from your repo to UltimaPHP, thanks!

[MuhsinAlaca]

I'm sorry, I've been having internet problems since yesterday, because there are studies in the area, I don't get very healthy data! During the last debugging I noticed that there were some problems. Debugging these errors in the scripts.

[MuhsinAlaca]

I also downloaded your last installed emulator, but do I still have a problem? My operating system is Windows 10 x64. First, when the server files are loaded, it sometimes reads Uop files, sometimes not. In short, I need to put the Ultima Online files back in the specified C: \ UoFiles \ directory! I couldn't figure out what caused this mistake! In short, I want you to know that I'm still not in, thanks ... First Last Result What do you think, masters?

[MuhsinAlaca]

And after waiting a few minutes, I get a result that way! Memory Problem

[MuhsinAlaca]

I think the basis of the problem Windows! However, the Windows user is too much, I hope your work will work well.

[MuhsinAlaca]

Without forgetting, I couldn't understand UltimaPhp.ini! I see it wrong, or is there such a Client Version? "client=7.0.80.0,67.0.74.0"

[MuhsinAlaca]

Hello again, I noticed a detail that caught my attention! The emulator sees the Uops files before converting the Uop extensions to Mul, but does not read the Uops after converting them!

[gokaygurcan]

@gokaygurcan I've converted the calculateKeys from your repo to UltimaPHP, thanks!

Glad that it helped.

Another issue is that the "first" (0xEF) packet (it's not actually a packet, but the seeds and so), have a bizarre behavior, sometimes it comes complete (21 bytes) sometimes it comes with 1 byte and next a 20 bytes packet arrives and sometimes it just doesn't send the packet command byte (0xEF):

Yes, it's a common problem. Almost in every 1 in 10 login requests, it behaves like that. I don't know the exact reason or logic behind it either. But sure it's interesting enough to check later ^_^ I should allocate some time in my weekly routine and continue working on this again. Maybe it can be a working project rather than a PoC :P

[joaoescribano]

@MuhsinAlaca I've been trying to understand all you have posted but I think I didn't get it all, so let me get to the points I've understood and you can tell me more soon:

1) Please, don't run the code at the browser, it's not made to run on it so use on cmd (as you run on windows) so you will be able to run better. 2) Post you ultimaphp.ini file so I can see your configuration, I've changed a few things at the server loading, so it needs to be properly configured and as far I can see at the pictures you load isn't going well as the packet class wasn't loaded. 3) The server still not working 100% with the UOP files, the loading is almost done but the map reading still needs work on. 4) The last client released from Electronic Arts to the game is 7.0.80.0, your configuration seems to correct to me. 5) Edit your ultimaphp.ini and change debug=0 to debug=1 and send me the server output of you login attempts using the unencrypted client_patched.exe 6) Please, post only 1 time and if need, edit the post to add more information so we can keep the issue clean.

@gokaygurcan it's an annoying problem, I really don't understand what the f*** is this behavior... I've tried many ways, believe me when I say many, to do it. But no success.

To everybody: As I mentioned, the "correct and better" solution (not the easy one) is to rewrite the entire socket server and I know it's difficult but it's needed... and before doing this, we need to really understand the start communication and write down that so we can understand why the client behaves like crazy.

I'll be working on it as soon as I can.

---

Update

@gokaygurcan @MuhsinAlaca please test the last commit please, I've been able to login 10x times sequentially with no issues, all worked well!

I've placed a bandage at the socket code to try to force the socket wait/compose the entire 0xEF packet, I also added some checks to prevent crashing and changed some variables that have been called statically to the local scope.

Thanks

[MuhsinAlaca]

First of all, I'm sorry for my bad English, it's normal that you didn't understand it because I wrote it using a translate.

Debug0 connection attempt

And Debug1 connection attempt

0xEF has improved, but this time we're facing a 0xD0 package problem!

[joaoescribano]

@MuhsinAlaca don't worry about the translation, at the end we will be able to fix it all 😆 Everything seems good so far, all config is good,

When you used the SUMP.exe file did you added them and -e parameters to it? example:

Copy client.exe to any folder in your desktop and place SUMP.exe there too, open cmd and navigate there and type: SUMP.exe client.exe -m -e (these parameters enables the multi-client opening and the encryption removal)

I say it because the packet after 0xEF should be 0x80, then 0xA8 as you can see here:

And as you can see, your packet has the same length as 0x80 but its all "masked".

Also, I just committed new changes that I think could make it work, can you please try again?

@gokaygurcan I think I may be fixed the issue to the 0xEF packet... I've fixed the site at https://github.com/UltimaPHP/UltimaPHP/blob/master/core/defs/packets.def.php#L240 so my code now can organize and "glue" the socket inputs and the validation now waits until the collect all 21 bytes and it only happens while the connection has no relay to the game server.

The message saying that the packet 0xEF came invalid with only 1 byte still appears, but it's just some creep cause of the changes, soon I figure out a fix.

After 2-3 hours of coding and restarting the server, I got zero errors 👍

[MuhsinAlaca]

Irmão João, tudo no seu computador parece bom, mas agora é 0x82. Eu acho que é um problema de conexão! Na verdade, tudo parece normal, eu não entendo! Espero que você não esteja com raiva. Acredite, ninguém mais pode resolver isso, obrigado.

[joaoescribano]

No @MuhsinAlaca you are almost there, packet 0x82 means that you have been disconnected by the server because your login has been denied by the server (reason 04).

That means that the account you are trying to log in wasn't correct, try login using this credentials:

Owner Account: Username: test Password: test

Player Account: Username: test2 Password: test

Also, check at your MongoDB if everything was fully imported (Example):

mongoimport.exe --db ultimaphp --collection accounts --type json --file c:/UltimaPHP/tools/Mongo_Database/accounts.json
mongoimport.exe --db ultimaphp --collection players --type json --file c:/UltimaPHP/tools/Mongo_Database/players.json
mongoimport.exe --db ultimaphp --collection server_starting_locations --type json --file c:/UltimaPHP/tools/Mongo_Database/server_starting_locations.json
mongoimport.exe --db ultimaphp --collection objects --type json --file c:/UltimaPHP/tools/Mongo_Database/objects.json

Updating

Useful stuff I've found around: https://github.com/Sphereserver/Source/blob/6c2c6152574b7876a076fdb05ff6a3e95f861951/src/common/CEncrypt.cpp#L324-L331 When the client switches between login and game server without opening a new connection, the first game packet is encrypted with both login and game server encryptions. This method attempts to initialize the game encryption from this initial game packet. If this wasn't inconvenient enough, it seems that not all client versions behave like this and some earlier ones do properly switch over to game encryption.

I've also checked the source below this quote, and it really seems that the server changes the seed somehow in here https://github.com/Sphereserver/Source/blob/6c2c6152574b7876a076fdb05ff6a3e95f861951/src/common/CEncrypt.cpp#L344-L348

After that, it calls decrypt() and it will end at the "two fish" decryption, which is not anymore available in PHP since mcrypt has been removed in PHP 7.2.

As I can see, there is 4 paths that we can go:

1) Only use unencrypted client 2) Fix the PHP version to 7.2 with mcrypt libraries on it and forget about updates 3) Develop our own extension to PHP using some known code that does encryption 4) Use some know shared library (.so / .dll) to make this (like Ultima Lib) and call it using the new API provided to PHP called FFI

I really don't know what to do, as I can search online.... there is no standalone implementation of TwoFish to PHP, I found many in C and C++.

What do you guys think?

[MuhsinAlaca]

Senhores, está tudo bem, finalmente estou aqui! Mas devo concluir as deficiências e, se houver algo que eu possa fazer, tentarei ajudar! Muitas deficiências Muitos detalhes a serem adicionados, por exemplo, cliloc, itens do npcs etc.

[joaoescribano]

That's awesome @MuhsinAlaca!!!! There is a LOT to do on the server, for sure.

But as you can log in now, I'm certain you will help us develop this systems :)

[youhide]

awesome

[MuhsinAlaca]

Espero que você melhore o mais rápido possível! Estou realmente interessado nisso, é uma linguagem extremamente rápida. Além disso, vou buscar o apoio do pages'll expressa em vários fóruns Ultima Online plataforma de peru confiança mim

[MuhsinAlaca]

Sinto muito, espero que você entenda.

[MuhsinAlaca]

Na verdade, acho que se você começar com a lógica da esfera, seguirá o caminho mais fácil.

[joaoescribano]

@MuhsinAlaca puedes escribir en español, entiendo menos pero creo que debería ser mejor.

Comenzaré a trabajar nuevamente en el servidor, pero realmente necesitamos más personas involucradas para crear los sistemas necesarios para la ejecución del servidor.

Las cosas más importantes que veo que se deben hacer por ahora:

1) Leyendo correctamente los archivos UOP 2) Interpretar correctamente el mapa y encontrar la Z correcta donde el jugador está en el mundo 3) Haga que la IA sea más inteligente (camine solo, camine por las escaleras, etc.)

Gracias por el apoyo y espero que pueda involucrar a más personas en UltimaPHP

[MuhsinAlaca]

Intentaré ayudar tanto como pueda, y debo mencionar que estoy más basado en Sphereserv, ¡también puedo ayudar con los diálogos! Sé que algunos amigos en Turquía que tengo un guión bueno, puedo pedir ayuda negociar con ellos ...

[joaoescribano]

Solo puedo agradecerles por esto, también tengo más conocimiento en la esfera que otros emuladores como RunUO y ServUO. Muchas cosas aquí están desarrolladas en base a Esfera pero desarrolladas para ser mejores :)

Gracias de nuevo

[gokaygurcan]

Hello Google Translate my old friend ^_^