search

OgarProject / Ogar

An open source Agar.io server implementation, written with Node.js.
Other
716 stars 823 forks source link

New Agar.io protocol #569

Open lukerayman opened 8 years ago

lukerayman commented 8 years ago

I work with Wireshark to figure out the new protocol for Agar.io servers. I know how to connect, read leaderboard, cells data. Player actions are not changed, you send the same data to join, spectate, move, split and eject as before. Here is what I already have.

Client -> Server

Initialize connection

The fist two messages initializes connection (the web socket is already open, you open it as before, no changes here).

Size Type Value Comment
1 byte 255 Opcode
4 uint32 6 Protocol version. Before the great protocol change it was 5.

The first message is enough to start getting world update and leaderboard messages. But if you want to play, you probably have to send the following message.

Size Type Value Comment
1 byte 254 Opcode
4 uint32 2106059819 This tricky. This value changes daily or even more often. If you send it wrong, your connection will be closed.

You are connected now. You can join, spectate and play. However, I discovered that your actions may be ignored until you send one more mysterious message.

Size Type Value Comment
1 byte 113 Opcode
20 byte[] 34:ac:3b:fd:3f:90:d5:88:fe:69:3c:9e:01:79:e6:e2:6f:f4:3c:c8 Some key that is different every time you connect to some server.

This key may be SHA-1 of a server token that comes in response after sending POST message to http://m.agar.io/findServer.

Spectate

Just send one byte 0x01.

Move

This message is not changed after the great protocol change.

Size Type Value Comment
1 byte 16 Opcode
4 int32 * Mouse X
4 int32 * Mouse Y
4 uint32 * Probably your cell ID

Other

Probably all the messages are the same as before. Only strings have a new format. It's probably UTF-8 but I'm not sure.

Server -> Client

Clear

This is the first message from the server. It's just a one byte 18 (0x12). It can be sent later again when the server is reset, so the client should remove all the cells from its memory and start a new fresh gameplay.

Leaderboard update

This message contains IDs and names of players in top 10.

Size Type Value Comment
1 byte 49 Opcode
4 uint32 0-10 Number of players on the leaderboard. Mostly 10.

Then you will get a list of players in following structures. Strings are sent in a new format.

Size Type Value Comment
4 uint32 0 Should be player ID, but the value is always 0. I guess it's not 0 only for us.
* string "Doge" Name of the player.

This is how you read strings now: each character is 1 byte long, no 2 as it was before (if it's UTF-8 then special characters can have more bytes). You read the string until you get byte 0. It means the end of the string and you can start reading ID of the next player.

World update

I still work on this but I already know how structures for new cells or updating existing ones look like. The message starts from opcode 255. Then you get the size of the rest of the message.

Size Type Value Comment
1 byte 255 Opcode
4 uint32 * The size of the rest of the message (in bytes). I counted it and the actual massage will have 2 more bytes than this value.
* * * Here's the main content. It starts from something and then you have a list of cells that are new, are just move.

Data structure for a new cell in our viewport looks like this:

Size Type Value Comment
4 uint32 * Cell ID
4 int32 * X position
4 int32 * Y position
2 uint16 0 - 65535 Size
1 byte 0 Flags. I discovered some examples: 0x03 (0011) - virus, 0x0E (1110) - normal cell that uses skin, 0x0A (1010) - normal cell that has no skin.
3 byte[] 0xFF6407 Cell color in RGB format. Colors from agar.io servers always have two bytes with values: 0xFF and 0x07 in random order. The third byte is random. Viruses always have color 0x33FF33.
* string "%snake" The name of the skin. This field is optional. You have to check flags to know whether to read this or skip.
* string "Doge" The name of the cell. For an unnamed cell you will just get byte 0. Then you can start reading ID of the next cell.

Strings are sent in the new format of course. If some call has skin and name, then you will find byte 0 in between these fields as a normal end of the skin string.

This is my guess about flags: 0x01 (0001) - virus 0x02 (0010) - i don't know, but all cells have this one 0x04 (0100) - cell with skin 0x08 (1000) - cell controlled by a user?

Structure for cell update is much simpler. Its size is always 15.

Size Type Value Comment
4 uint32 * Cell ID
4 int32 * X position
4 int32 * Y position
2 uint16 0 - 65535 Size
1 byte * Flags

Camera update

In spectator mode. This packet has not been changed after the great protocol change.

Size Type Value Comment
1 byte 17 Opcode
4 float * Camera X position
4 float * Camera Y position
4 float * Zoom

My cell

You get this message when you join the game.

Size Type Value Comment
1 byte 32 Opcode
4 uint32 * Your cell ID

That's all I know so far. The hardest thing for the client is to obtain this secret number for the second initial message. In the previous version of the protocol you could easily find it in the "main_out.js" file. Right now I have no idea where to get it. Another mysterious part is the message with opcode 113. I guess you can't start playing without sending this message. It worked yesterday, but today it doesn't: your messages to spectate or join the game are ignored.

lukerayman commented 8 years ago

I think I will be constantly updating this topic when I find something new about the protocol.

LostCodesy commented 8 years ago

@lukerayman too good! Good Works!

firelightning13 commented 8 years ago

@lukerayman Good find!

BaumanDev commented 8 years ago

@lukerayman If we try updating this, will it work?

Andrews54757 commented 8 years ago

@NatsuTheGreat It wont work well. We would have to update a "key" every day to join. this is because they made it so it sends a special key that changes every day and if it is the wrong key, the connection is closed

Andrews54757 commented 8 years ago

F*ck moneyclip

lukerayman commented 8 years ago

@NatsuTheGreat I think you can create a branch for a new Ogar implementation and try. I think it may work.

lukerayman commented 8 years ago

@Andrews54757 This problem is on the client side. I think Ogar servers don't have to care about this key. They can just accept connection and send packets using the new protocol. The first packet should be one byte 18 and then we can try updating Leaderboard. If agar.io will render it correctly, then we can try go further and send packets with cells data.

BaumanDev commented 8 years ago

So do I try it or not? xD

BaumanDev commented 8 years ago

@lukerayman Can you actually try all this out and tell me if it works?

Luka967 commented 8 years ago

I'll begin integrating v7 protocol to my c9 server w/ improved anti-teaming.

BaumanDev commented 8 years ago

Alright man, tell me how it goes!

BaumanDev commented 8 years ago

Can anybody tell me if they succeed, please?

Andrews54757 commented 8 years ago

I sent this mail to moneyclip:

Hello nuno (or is it someone else?),

I am a dev from ogar ul as you may already know. I want to first set things straight: Do you, or do you
 not want private servers? if you are okay, then we would work on adjusting the protocal to comply. 
Also, if you dont want them, can you please make a site such as agar.io/private/ which has the old 
client? I know you deleted the old one at agar.io/v72/. I just dont want to use an illegal client because 
of my morals, I even suspended the temporary client sites for ogarul. If all you want is money, you can 
still have ads there and to make more money, you could sell backgrounds as they are highly wanted. It 
would mean a lot to the community if you bring private servers back. It would mean winning over the 
tens of thousands of ex-supporters back. Thank you

andy s
BaumanDev commented 8 years ago

@Barbosik Any way you could try that and make some kind of instructions into placing the new protocls correctly for it to work? That was a lot of information through at me from lukarayman and you. It's kinda confusing.

Andrews54757 commented 8 years ago

They are probably going to change the protocal again after we figure it out

Barbosik commented 8 years ago

Andrews54757 I don't think so, because they already added protection from bots, so there is no reason to update protocol again.

BaumanDev commented 8 years ago

@Barbosik Alright, tell me if you get it done.

irmgrx commented 8 years ago

For the cell size, maybe 0xE010 read by 0x10E0 which is 4320 mass. I studied for virus cell, f17c4644 6bf2ffff 0c110000 6400 03 33ff33, 0x6400 is size, read by 0x0064 which is 100 mass.

Andrews54757 commented 8 years ago

@Barbosik Remember, we are dealing with MONEYclip. I think they disabled private servers because we cracked skins and they got a lot of money from skins

Barbosik commented 8 years ago

irmgrx no, the first byte is lower byte it's approved by other cells. If you want, I can send you full websocket communication session in txt file, it will be easier to research.

Aslo there is not so much updates for this strange cell with id 28 11 43 B2. First it appears in the message that I posted above. This cell Id appears in entire session just several times:

Message 1 (I posted it above):

Id          X           Y           Size  Fg Color
=========== =========== =========== ===== == ========
FF 96 03 00 00 F2 33 
10 00 00 
FE C9 43 B2 D0 FF FF FF 2C D8 FF FF 0C 00 02 6E FF 07 
74 2E 43 B2 75 01 00 00 DC D7 FF FF 0B 00 02 FF D9 07 
5A 15 43 B2 E6 FB FF FF BF D7 FF FF 0A 00 02 F0 07 FF 
28 11 43 B2 49 00 00 00 BA 12 00 F1 10 E2 FF 07 
A7 1C 43 B2 A5 FC FF FF 40 D8 FF FF 0D 00 02 85 FF 07 
69 19 43 B2 7A 00 00 00 45 D9 36 00 D1 07 AD FF 
94 1A 43 B2 19 FE FF FF 86 D9 5A 00 D2 07 FF 48 
BD 04 43 B2 5E 02 00 00 13 D8 24 00 B2 F3 FF 
36 07 43 B2 4C FB FF FF B2 7E 00 C2 51 FF 07 
41 06 43 B2 41 01 00 00 7C 36 00 C2 93 07 FF 
2D 02 43 B2 2D FE FF FF FB 5A 00 C2 FF 31 07 
ED 02 43 B2 F8 FE FF FF 5B 12 00 83 17 FF 07 
0B 0F 43 B2 E3 C6 00 F2 00 0A 00 02 3F FF 07 
66 0E 43 B2 84 FD FF FF FA B4 00 C2 88 07 FF 
9E 0E 43 B2 9F 00 00 00 46 7E 00 C2 FF E4 07 
CB 09 43 B2 20 FD FF FF 84 12 00 C2 07 FF 25 
65 0A 43 B2 17 FF FF FF F9 5A 00 D1 65 07 FF 
09 0A 43 B2 31 FB FF FF 04 D8 C6 00 C2 B0 07 FF 
65 75 43 B2 9D 01 00 00 7D 36 00 D1 FF A5 07 
F3 74 43 B2 2A FC FF FF 07 DA 24 00 C2 BC 07 FF 
FB 74 43 B2 6F FC FF FF CC 36 00 C2 FF 08 07 
9D 77 43 B2 79 FF FF FF C0 5A 00 B3 93 07 FF 
4F 71 43 B2 00 FE FF FF 48 00 C2 99 07 FF 
F2 72 43 B2 D9 FE FF FF 60 FC 00 C3 07 CA FF 
D1 72 43 B2 C0 00 00 00 EE 6C 00 B4 07 85 
CC 7D 43 B2 DD FF FF FF 16 12 00 A2 F3 
50 7C 43 B2 39 01 00 00 72 5A 00 C2 FF 07 62 
FC 7C 43 B2 48 FC FF FF B4 FC 00 C4 4E 07 FF 
ED 7E 43 B2 33 02 00 00 3B 36 00 72 79 
D9 7E 43 B2 56 FD 7A 01 F4 00 0A 00 02 07 FF F6 
2C 79 43 B2 4A FD FF FF D8 24 00 A2 FE 
B3 79 43 B2 33 01 00 00 3E 5A 00 C2 07 FF 8E 
ED 78 43 B2 25 FE FF FF DB 12 00 D2 FF 07 BF 
61 7B 43 B2 0E FC FF FF 0D DA 48 00 B2 62 FF 
AB 7B 43 B2 1C FC FF FF 73 48 00 C2 07 AD FF 
DF 64 43 B2 5F FC FF FF 62 12 00 C2 D6 FF 07 
84 64 43 B2 70 FC FF FF F4 A2 00 C2 FF 07 7F 
85 64 43 B2 5F FF FF FF 4A 24 00 C2 FF 07 AD 
F5 67 43 B2 48 FE FF FF A3 6C 00 C3 07 FF 88 
80 67 43 B2 DB FF FF FF 0F 6C 00 85 AD FF 
7E 66 43 B2 0B FD EA 00 C3 62 FF 07 
45 66 43 B2 65 FF FF FF 24 36 00 85 51 FF 
BB 60 43 B2 E4 FC EA 00 C2 5F 07 FF 
BF 60 43 B2 83 00 00 00 BD 7E 00 C2 A5 07 FF 
CA 63 43 B2 8A FE FF FF F1 12 00 C4 FF 07 A8 
CE 63 43 B2 D1 FB FF FF 01 90 00 A2 7F 
6A 6C 43 B2 E9 FE FF FF B8 12 00 C2 34 FF 07 
76 6C 43 B2 79 02 00 00 92 12 00 C3 07 FF F0 
C6 6C 43 B2 F8 FC FF FF 45 7E 00 F0 14 FF 2B 
44 6F 43 B2 F9 FE FF FF BE D7 FF FF 20 00 0A FF A5 07 54 45 53 54 58 58 58 58 00 
00 00 00 00 00 00

Message 2:

FF 25 00 00 00 F0 16 
10 00 00 
44 6F 43 B2 49 FC FF FF E0 D7 FF FF 29 00 08 54 45 53 54 58 58 58 58 00 
00 00 00 00 01 00 
28 11 43 B2

Message 3:

FF 51 00 00 00 F0 42 
10 00 00 
28 11 43 B2 49 00 00 00 BA D7 FF FF 0A 00 02 E2 FF 07 
00 11 43 B2 B8 FD FF FF 68 DB FF FF BA 00 00 
44 6F 43 B2 52 FC FF FF 7B D8 FF FF 2B 00 08 54 45 53 54 58 58 58 58 00 
A5 6E 43 B2 09 FC FF FF 72 DB FF FF BA 00 00 
00 00 00 00 00 00

Message 4:

FF 25 00 00 00 F0 16 
10 00 00 
44 6F 43 B2 41 FC FF FF FA D7 FF FF 30 00 08 54 45 53 54 58 58 58 58 00 
00 00 00 00 01 00 
28 11 43 B2

The goal is to find the way on how to split these records cell by cell. I split it in the quote by the assumption that each new cell is started with cell ID which is easy to identify, because high bytes of ID usually the same for all cells. But there is need to understand how to prepare message for the client. There is need to understand how to mark the size of data block for each cell.

Barbosik commented 8 years ago

Andrews54757: I think they don't think to disable ogar server, but they also don't wan't to support it. So, it's the reason why they make changes in the protocol without worry about ogar compatibility. They fighting with bots and alternative clients, because it reduces advertisement statistics. But they don't need to worry about ogar, because it doesn't affect their advertisement.

F0RIS commented 8 years ago

@ItzLevvie don't want to waste one hour for that. Can you tell long story short what useful he said there?

Andrews54757 commented 8 years ago

@Barbosik I think they want to disable it. One reason is because earlier there was agar.io/v72 that had the old client, but as soon as the community pointed that out, they deleted it for no reason. I mean, even if they didn't delete it, bots or things can't connect to the official servers by that. so I'm assuming that they don't like ogar

Barbosik commented 8 years ago

Some new findings:

Meaning of some bits in Flags byte:

Bit Hex Mask Description
0 0x01 isVirus
1 0x02 isColorPresent (3 bytes with R,G,B components included into update)
2 0x04 isSkinPresent (zero terminated string with skin name included into update)
3 0x08 isNamePresent (zero terminated string with skin name included into update, UTF8 encoding)
4 0x10 isAgitated (increase wave amplitude on the cell outline)
Barbosik commented 8 years ago

Some new finding about eat notification.

Eat message:

Offset Value Type Description
0 0xFF byte Update Message
1 xx int32 size of rest message in bytes
5 0xF1 byte ? some strange code works with 0xF0...0xFF
6 xx byte ? size of rest message - 15 (here is something strange with message > 255 bytes
7 0x10 byte Update Opcode
8 xx byte ? some kind of flag, it is not zero when message contains eat entry
9 0x00 byte ? may be high byte of Eat Cell Count?
10 xx byte array Cell Eat notification list

Cell Eat Notification entry is very easy:

Offset Type Description
0 uint32 Hunter Cell Id (who)
4 uint32 Prey Cell Id (whom)

Message example (tested and it works good) with two notifications - 0x988E7150 eats 0x988E7151 and 0x988E7150 eats 0x988E7154:

FF 19 00 00 00 F2 0A 
10 02 00 
50 71 8E 98 51 71 8E 98
50 71 8E 98 54 71 8E 98
00 00 00 00 00 00

There is also need to understand how to send remove notification

Andrews54757 commented 8 years ago

@ItzLevvie could you share? im trying to see how the custom skins worked before the update

Barbosik commented 8 years ago

Good news, I uncovered part of the new protocol which is needed for Ogar server! :)

It seems like message with compression type==0xF0 is not compressed! :) I'm testing it and at a glance it seems like it works well! :) I can process cell update, cell eat and cell remove notifications with compression type==0xF0 :)

Message Update structure (for compression type=0xF0):

Offset Value Type Description
0 0xFF byte Message Update ID
1 xx uint32 ? Usually contains count of bytes from field at offset 7 (from Update Type till the end). I didn't checked it more deep.
5 xx byte Compression Type, 0xF0 means that there is no compression
6 xx byte* ?Contains count of bytes after this field - 0x0F. If this byte is 0xFF then there is need to read next byte and make sum.
7 xx byte Update Type: 0x40 - border update; 0x10 - cell update
8 xx byte array Data Chunk 
FF 3A 00 00 00 F0 0C 
10 
00 00 
C8 70 52 6B 37 FF FF FF A9 11 00 00 DE 00 00 
F1 6C 52 6B D0 FE FF FF AD 1B 00 F0 0C 00 00 
8E 6D 52 6B 5B 00 00 00 3F 10 00 00 0E 01 00 
00 00 00 00 
01 00 
9C 0C 52 6B

Structure of Data Chunk for border update:

Offset Value Type Description
0 xx float64 Left
8 xx float64 Top
16 xx float64 Right
24 xx float64 Bottom
32 xx int32 (optional) Server type (0=FFA; 1=Teams; 4=Experimental; 8=Party)
36 xx string (optional) Zero-terminated string with server version

Structure of Data Chunk for cell update:

Offset Value Type Description
0 xx uint16 Eat Record Count
2 xx byte array Eat Records
xx xx byte array Cell Update Records. Ends with cell ID == 0x00000000
xx xx uint16 Remove Record Count
xx xx byte array Remove Records

Structure of Eat Record:

Offset Type Description
0 uint32 Hunter Cell Id (who eat)
4 uint32 Prey Cell Id (whom eaten)

Structure of Remove Record:

Offset Type Description
0 uint32 Cell ID (which should be removed)

Structure of Cell Update Record:

Offset Value Type Description
0 xx uint32 Cell ID, 0x00000000 is used as terminator
4 xx int32 Coordinate X
8 xx int32 Coordinate Y
12 xx int16 Cell size
14 xx byte Flags
15 xx 3 byte (optional) Color (r,g,b components)
xx xx string (optional) Skin name (zero-terminated string)
xx xx string (optional) Cell name (zero-terminated string, UTF8 encoding)

The flags:

Bit Hex Mask Description
0 0x01 isVirus
1 0x02 isColorPresent (3 bytes with R,G,B components included into update)
2 0x04 isSkinPresent (zero terminated string with skin name included into update)
3 0x08 isNamePresent (zero terminated string with skin name included into update, UTF8 encoding)
4 0x10 isAgitated (increase wave amplitude on the cell outline)

Message example:

FF 5D 00 00 00 F0 4E                               // header
10                                                 // Update Type = cell update
02 00                                              // 2 eat records
B5 A9 55 CA E7 B1 55 CA                            // eat record #1
51 4F 55 CA 44 5D 55 CA                            // eat record #2
38 4C 55 CA 59 0B 00 00 7B 00 00 00 CF 00 00       // cell update record #1
51 4F 55 CA 1D 0E 00 00 F0 00 00 00 C5 00 00       // cell update record #2
CC A1 55 CA D3 12 00 00 C8 FE FF FF E4 00 00       // cell update record #3
B5 A9 55 CA D8 0C 00 00 94 FE FF FF 9E 01 00       // cell update record #4
00 00 00 00                                        // cell update record terminator
02 00                                              // 2 remove records
44 5D 55 CA                                        // remove record #1
E7 B1 55 CA                                        // remove record #2

Large message example:

FF 17 01 00 00 F0 FF 09 // note that we have two byte size: FF 09 = FF+09 = 0x0108
10 
00 00 
93 CA FE 55 87 04 00 00 CD 0B 00 00 32 01 0E 07 FF 88 25 77 61 73 70 00 64 6F 67 65 00 
57 BD FE 55 8D FE FF FF 07 0C 00 00 58 01 0A 8B FF 07 4E 6F 6B 69 61 6E 00 
71 7A FD 55 0F 02 00 00 8C 09 00 00 CC 00 0E 07 ED FF 25 73 6E 61 6B 65 00 D1 81 D0 B2 D0 B5 D1 82 D0 BA D0 B0 20 D0 B4 D0 BE D0 B1 D1 80 D0 B0 D1 8F 20 00 
98 67 FD 55 E5 06 00 00 E9 0D 00 00 A7 00 0A 07 FF C8 D0 81 20 D0 BC D0 BE D1 91 00 
FD 13 FD 55 89 FF FF FF E5 0E 00 00 1D 01 0E 07 4B FF 25 75 72 61 6E 75 73 00 D0 90 D0 99 D0 A2 D0 95 D0 9D 20 D0 90 D0 97 D0 95 D0 A0 D0 98 00 
92 04 FD 55 11 02 00 00 08 0D 00 00 41 01 0E 07 FF D3 25 62 61 74 00 D0 B8 D1 89 D1 83 20 D0 BB D1 83 D1 87 D0 B0 D0 B2 D0 B0 20 D0 B4 D1 80 D1 83 D0 B3 00 
42 0D FD 55 A4 03 00 00 DB 0D 00 00 0A 00 02 31 07 FF 
56 33 FD 55 2E 06 00 00 18 0B 00 00 0A 00 02 51 FF 07 
00 00 00 00 
00 00

Basically this information is enough to fix ogar server :) There is not so clear on how to handle large messages (> 255 bytes), but we can split large update into several short messages.

Here is still open question on how to process extra large messages (with length > 255 bytes) and how to decompress messages with compression type 0xF1, 0xF2, 0xF4. If you have any ideas or any kind of information, please let me know

lukerayman commented 8 years ago

@Barbosik this is awesome! :D

Luka967 commented 8 years ago

Damn so much info! I'll try to implement this right now! :+1: edit: Apparently there is a problem with websocket for me. Server doesn't respond to GET message.

firelightning13 commented 8 years ago

@Barbosik awesome!

BiliBiliLzy commented 8 years ago

@Barbosik how can we do? wait a new version?

BiliBiliLzy commented 8 years ago

@Barbosik Which should I update Ogar2 or Ogar Node.js Version?

Barbosik commented 8 years ago

I've found that compression type is not limited to values 0xF0..0xFF. Agario server uses different values, I catch at least the following: 0x21, 0x80, 0x90, 0xB1, 0xD0, 0xF0, 0xF1, 0xF2, 0xF3, 0xF4, 0xF5, 0xF6, 0xF7, 0xFA, 0xFD, 0xFF.

It seems like some compression type is used for specific update only.

For example 0x90 is used for empty update:

FF 09 00 00 00 90     // note that there is no length field for 0x90
10 
00 00 
00 00 00 00 
00 00

0xD0 is used for remove only update:

FF 0D 00 00 00 D0   // note that there is no length field for 0xD0
10 
00 00 
00 00 00 00 
01 00 
A7 4A 41 94

The most often used compression type is 0xF1, 0xF2, 0xF4.

Also, I found that compression starts from cell with size >= 0x0800 (it's mass = 41943). So, such size is impossible for the new protocol, at least in plain form. It should not be used for plain update with no compression.

PS: added some fix into Data Chunk for cell update descriptions (Remove Cell Count is not optional and always exists)

Barbosik commented 8 years ago

Some info for implementation in Ogar server.

Any session should be started from Clear Message (just single byte 0x12). After this server should send border update (through message 0xFF) and then cell updates (through message 0xFF).

After join command (0x00), server should send Clear Message (0x12) and border update. Then server should send Owned Message (0x20 with cell id). And then server should send cell updates.

I tested it and it works.

BaumanDev commented 8 years ago

So any tutorials? :P That is 6 tons of info right there xD

Barbosik commented 8 years ago

I wrote all what is needed for Ogar server in this post: https://github.com/OgarProject/Ogar/issues/569#issuecomment-219332968 Also, general info on how to deal with new agario client: https://github.com/OgarProject/Ogar/issues/569#issuecomment-219547592

Currently I'm working on compression research for the new protocol. I have a little news about it. It seems that I found the meaning of the message 0xFF header :)

What we have:

Offset Type Field Name Value Description
0 byte Message Code 0xFF Update Message Code
1 uint32 Uncompressed Length xx Length of the Message Content (after decompression)
5 byte Compression Type xx ?Determine how to decompress the message. For example 0xF0
6 byte Compression Offset xx (optional) Pointer to the first compressed byte - 15 (within message content, also this byte is missing for some compression types, such as 0xD0, 0x90, probably it means that compression is not applied for such messages)
x byte Extended Compression Offset xx (optional) This byte is present when Compression Offset==0xFF
x byte array Message Content xx Message Data (may be compressed)

For example, the full message:

FF 27 00 00 00 F0 0B 
10 00 00
00 B6 4C 77 F9 00 00 00 C9 00 00 00 6E 00 00
02 9E 4C 77 7E 01 00 00 1A 00 90 71 00 01            // <= this part is compressed
00 00 00 00 
00 00

We have Compression Offset = 0x0B, it means that compression starts at offset 0x0B + 0x0F = 0x1A within Message Content block.

Here is compressed Message Content:

10 00 00
00 B6 4C 77 F9 00 00 00 C9 00 00 00 6E 00 00
02 9E 4C 77 7E 01 00 00 1A 00 90 71 00 01            // <= this part is compressed
00 00 00 00 
00 00

It means that this part of the Message Content is not compressed:

10 00 00
00 B6 4C 77 F9 00 00 00 C9 00 00 00 6E 00 00
02 9E 4C 77 7E 01 00 00

This is compressed part:

1A 00 90

which should be decompressed into 4 bytes and after this we have data which is not compressed:

71 00 01
00 00 00 00 
00 00

So, the Message Content should be decompressed into the following data:

10 00 00
00 B6 4C 77 F9 00 00 00 C9 00 00 00 6E 00 00
02 9E 4C 77 7E 01 00 00 xx xx xx xx 71 00 01
00 00 00 00 
00 00

It's not so clear what is xx xx xx xx exactly. But I found that in this case the value xx xx xx xx doesn't depend on the value Y for the first cell. I modified the message a for little and tested it on my test server. I get both cells on a single line with this message:

FF 27 00 00 00 F0 0B 
10 00 00 
00 B6 4C 77 71 01 00 00 10 00 00 00 01 00 00 
02 9E 4C 77 7E 01 00 00 1A 00 90 01 00 00 
00 00 00 00 
00 00

So, in such way I found that 1A 00 90 is decompressed into 10 00 00 00.

And decompressed Message Content is the following:

10 00 00
00 B6 4C 77 F9 00 00 00 C9 00 00 00 6E 00 00
02 9E 4C 77 7E 01 00 00 10 00 00 00 71 00 01
00 00 00 00 
00 00

Now I need to understand how to decompress 1A 00 90 to 10 00 00 00.

Here is another message with compressed data block:

FF 3A 00 00 00 F0 0C 
10 
00 00 
C8 70 52 6B 37 FF FF FF A9 11 00 00 DE 00 00 
F1 6C 52 6B D0 FE FF FF AD 1B 00 F0 0C 00 00   // <= compression applied here
8E 6D 52 6B 5B 00 00 00 3F 10 00 00 0E 01 00 
00 00 00 00 
01 00 
9C 0C 52 6B

In this case we have compressed block 1B 00 F0 0C which should be decompressed into 4 bytes, probably it will be 10 00 00 C8 (I tested it a little but still not sure). I found that byte 1B in this case means backward offset for the pattern (it points to the begin of Message Content).

I modified this message for test purposes in the following way:

FF 3A 00 00 00 F0 0C 
10 
00 00 
0A B6 4C 77 37 FF FF FF AD 10 00 00 0A 00 00 
01 B6 4C 77 D0 FE FF FF AD 1B 00 F0 0C 00 00 
02 B6 4C 77 5B 00 00 00 AD 10 00 00 0A 00 00 
00 00 00 00 
01 00 
03 B6 4C 77

And I get all 3 cells on single horizontal line with the same size (0x000A).

I tried to modify first bytes of Message Content:

10 
00 00 
20 B6 4C 77

and it works - the size of second cell is changed to 0x0020! :)

So, 0x1B in this case is definitely pattern offset. The same case as in the previous message (where 0x1A also points to 10 00 00). I have no idea what is the meaning of another compressed bytes (00 F0 0C for this message), but it looks very similar to the compression type and 00 looks like high byte of pattern offset ;) At a glance it should works, because this approved on several messages. Need to check more deep. Stay tuned :)

Actually this info is not needed for Ogar server, because we can just use non-compressed messages. Just set Compression Offset field to the end of Message Content and there is no need to deal with compression.

Barbosik commented 8 years ago

My assumptions about compression approved on a large amount of messages :)

So, the server sends compressed message:

Offset Type Field Name Value Description
0 byte Message Code 0xFF Update Message Code
1 uint32 Uncompressed Length xx Length of the Message after decompression
5 byte Compression Type xx Compression type for the first compressed block
6 byte Compression Offset xx (optional) Pointer to the first compressed byte within message (there is need to add 15 to get actual position)
x byte Extended Compression Offset xx (optional) This byte is present when Compression Offset==0xFF.
x byte array Message xx Compressed Message

If you don't want to use compression, you can use compression type 0x90. In this case Compression Offset field is missing and message will not be compressed.

The question is how to process different compression types. I found that compression types 0xF0-0xFF include Compression Offset. Also I found that the following compression types don't include Compression Offset field: 0x21, 0x70, 0x83, 0x90, 0x92, 0xA1, 0xA2, 0xB1, 0xB3, 0xC2, 0xD0, 0xD1, 0xD2.

At a glance, lower 4 bits of Compression Type for 0xF0-0xFF means length of the pattern which is used for replacement. For 0xF0, compressed block has the following structure:

Offset Type Description
0 int16 Offset to the pattern which should be inserted (in backward direction)
2 byte Compression Type for the next compressed block
3 byte (optional) Compression Offset for the next compression block

Pattern length for 0xF0 is 4 bytes

Barbosik commented 8 years ago

I got it :)

Now I can decompress blocks with compression type 0xF0..0xFF! Works great :)

Pattern length can be calculated with following formula: patternLength = (type & 0x0F) + 4 where type is Compression Type (in range 0xF0...0xFF)

BiliBiliLzy commented 8 years ago

@Barbosik now how can i do , wait you fix the server done or change some config?``

irmgrx commented 8 years ago

This is awesome! How do you figure out the formula? Did you dig into the source code?

EnTerr commented 8 years ago

@Barbosik - so it's a LZ77 with variation on length/offset pairs?

Barbosik commented 8 years ago

irmgrx: no, I just analyzed data which I captured from browser network session :)

EnTerr: may be you're right, need to test it.

I have some news, now I understand what means high 4 bits of Compression Type! :) High 4 bits is Compression Offset ;) If it's 0xF0, then additional byte is present. So, it's the reason why there is need to add 0x0F for compression type 0xF0.

Now I can decompress almost all packets. Here is a little problem with compression type 0x0F.

Message example:

96 01 00 00 F2 82 
10 
00 00 
E6 DE 1B 76 F1 F4 FF FF B6 EF FF FF 64 00 03 33 FF 33 
60 00 1A 76 66 F5 FF FF DB F2 FF FF 0B 00 02 07 FF E4 
9A 02 1A 76 14 F6 FF FF CF EF FF FF 0A 00 02 0B FF 07 
04 07 1A 76 DF F1 FF FF 07 F2 FF FF 48 01 0A 07 FF 14 D0 9F D0 BE D0 BC D0 BE D0 B3 D1 83 20 D1 87 D0 B5 D0 BC D0 A1 D0 BC D0 BE D0 B3 D1 83 00 
D9 19 1A 76 28 F2 FF FF A7 EF FF FF B8 01 0E 65 07 FF 25 75 66 6F 00 4B 6F 6C 69 62 72 69 00 
F0 1E 1A 76 30 F6 FF FF C0 61 00 D2 FF 07 4B 
03 10 1A 76 1D F6 FF FF C1 F0 12 00 C1 6B 07 
71 10 1A 76 59 F6 FF FF E1 F1 12 00 FF 02 07 34 FF 
16 11 1A 76 72 F3 FF FF 5E F2 FF FF 53 00 85 00 0F DF 
17 11 1A 76 61 F0 FF FF 3A F2 FF FF 3A 30 00 10 AF 
0C 11 1A 76 0A F3 FF FF 13 F1 30 00 13 DF 
02 11 1A 76 47 F3 FF FF 78 F1 FF FF 3B 60 00 10 A1 
26 12 1A 76 F6 F4 FF FF 7C F2 D2 00 90 88 07 FF 
00 00 00 00 
00 00

With using information that I understand, I can decompress this part:

10 
00 00 
E6 DE 1B 76 F1 F4 FF FF B6 EF FF FF 64 00 03 33 FF 33 
60 00 1A 76 66 F5 FF FF DB F2 FF FF 0B 00 02 07 FF E4 
9A 02 1A 76 14 F6 FF FF CF EF FF FF 0A 00 02 0B FF 07 
04 07 1A 76 DF F1 FF FF 07 F2 FF FF 48 01 0A 07 FF 14 D0 9F D0 BE D0 BC D0 BE D0 B3 D1 83 20 D1 87 D0 B5 D0 BC D0 A1 D0 BC D0 BE D0 B3 D1 83 00 
D9 19 1A 76 28 F2 FF FF A7 EF FF FF B8 01 0E 65 07 FF 25 75 66 6F 00 4B 6F 6C 69 62 72 69 00 
F0 1E 1A 76 30 F6 FF FF C0 EF FF FF 0A 00 02 FF 07 4B 
03 10 1A 76 1D F6 FF FF C1 F0 FF FF 0A 00 02 FF 6B 07 
71 10 1A 76 59 F6 FF FF E1 F1 FF FF 0A 00 02 07 34 FF 
16 11 1A 76 72 F3 FF FF 5E F2 FF FF 53 00 0A 07 FF 14 D0 9F D0 BE D0 BC D0 BE D0 B3 D1 83 20 D1 87

After this, here is something strange. We have compressed block with following data:

12 00 FF 02

it is decompressed into this data, it's ok:

FF FF 0A 00 02

Next we have compressed block with following data

85 00 0F DF

My method is able to decompress it into this:

0A 07 FF 14 D0 9F D0 BE D0 BC D0 BE D0 B3 D1 83 20 D1 87

And next, something is going wrong. I see that decompressed data should be:

0A 07 FF 14 D0 9F D0 BE D0 BC D0 BE D0 B3 D1 83 20 D1 87 D0 B5 D0 BC D0 A1 D0 BC D0 BE D0 B3 D1 83 00

So, there is need to copy additional 15 bytes but next compression type is 0x0F (strange compression offset = 0 and .strange pattern length with maximum value)

EnTerr commented 8 years ago

As a compression this is REALLY bad. E.g. the best case i saw was squeeze 1020 bytes down to 871, that's only 15% less, not worth it. Royal pain in the ass, for no practical benefit.

BaumanDev commented 8 years ago

LOOL Enterr you mad?

"PS: Go finger yourslef Enterr"

Barbosik commented 8 years ago

I found another interesting message:

DC 00 00 00 F1 0A 
10 
01 00 
03 FF FA 0E 6E 43 FA 0E 
8B 7E F9 0E E7 0C 00 00 AB F0 FF FF 84 00 17 00 F1 6F 46 0A 00 00 1B F0 FF FF 75 00 00  // <= compression applied here 17 00 F1 6F
32 ED FA 0E 45 0A 00 00 E0 F2 FF FF E3 00 00 
10 37 FA 0E AF 0A 00 00 CF ED FF FF 6E 00 00 
EE 5B FA 0E 70 0C 00 00 B0 F2 FF FF 4C 00 00 
6C 55 FA 0E 7D 09 00 00 69 EF FF FF 96 00 00 
66 43 FA 0E 8A 09 00 00 A2 F0 FF FF 38 00 00 
65 43 FA 0E 99 0A 00 00 75 F0 FF FF 38 00 00 
64 43 FA 0E D9 0A 00 00 A0 F0 FF FF 39 00 00 
62 43 FA 0E E1 0B 00 00 6D F1 1E 00 92 
60 43 FA 0E 73 09 00 00 94 3C 00 A1        // <= compression applied here 3C 00 A1  
6F 43 FA 0E 3D 08 00 00 DB F2 1E 00 22     // <= compression applied here 1E 00 22 
6B 43 87 00 12 77 1E 00 E0                 // <= compression applied here 87 00 12 and 1E 00 E0
00 00 00 00 
02 00 
29 21 FA 0E 
6E 43 FA 0E

Something is going wrong on this compressed sequence:

12 77 1E 00 E0

So, something is not so clear for compression type=0x0x and 0x1x. Probably it should be decompressed into this sequence, but I'm not sure:

77 F0 FF FF 38 00 00
EnTerr commented 8 years ago

@Barbosik - last packet is great because i think it shows 0xFF is just optional "compression" applied on the 0x10 packet in the new format - cue how the first copy takes "00: 03:FF:FA:0E:" from the "eats" section.

I bet that means an Ogar server wouldn't have to use that "compression" with Agar.io client - it can just send 0x10 in the new format (considering color is now optional and flags goes first), without wrapping it in 0xFF

EnTerr commented 8 years ago

@Barbosik - here is your last package unpacked:

FF
DC:00:00:00:      len(unpacked) = 220                           [F1:0A:]=skip 25, copy 5
10:
01:00:  03:FF:FA:0E:    6E:43:FA:0E:
8B:7E:F9:0E:    E7:0C:00:00:    AB:F0:FF:FF:    84:00:          [17 00]=back 23, get the 5
                                                       (00:  
03:FF:FA:0E:)                                                   [F1 6F]=skip 126, copy 5
                46:0A:00:00:    1B:F0:FF:FF:    75:00:  00:
32:ED:FA:0E:    45:0A:00:00:    E0:F2:FF:FF:    E3:00:  00:
10:37:FA:0E:    AF:0A:00:00:    CF:ED:FF:FF:    6E:00:  00:
EE:5B:FA:0E:    70:0C:00:00:    B0:F2:FF:FF:    4C:00:  00:
6C:55:FA:0E:    7D:09:00:00:    69:EF:FF:FF:    96:00:  00:
66:43:FA:0E:    8A:09:00:00:    A2:F0:FF:FF:    38:00:  00:
65:43:FA:0E:    99:0A:00:00:    75:F0:FF:FF:    38:00:  00:
64:43:FA:0E:    D9:0A:00:00:    A0:F0:FF:FF:    39:00:  00:
62:43:FA:0E:    E1:0B:00:00:    6D:F1:                          [1E 00]=back 30, copy the 5         
                                     (FF:FF:    38:00:  00:)    [92]=skip 9, copy 6
60:43:FA:0E:    73:09:00:00:    94:                             [3C 00]=back 60, copy the 6   
                                  (F2:FF:FF:    4C:00:  00:)    [A1]=skip 10, copy 5
6F:43:FA:0E:    3D:08:00:00:    DB:F2:                          [1E 00]=back 30, copy the 5
                                     (FF:FF:    38:00:  00:)    [22]=skip 2, copy 6
6B:43:                                                          [87 00]=back 135, copy the 6
     (FA:0E:    AF:0A:00:00:)                                   [12]=skip 1, copy 6
77:                                                             [1E 00]=back 30, copy the 6
  (43:FA:0E:    3D:08:00:)                                      [E0]=skip 14, copy 0
00:00:00:00:
02:00:  29:21:FA:0E:    6E:43:FA:0E                             2 deletions

Decodes beautifully, no surprises. Maybe should explain my notation if not obvious - in square brackets are the removed [compression codes], in parenthesis are the inserted instead (copied codes). I verified the my unpacked length is 220, so yeah.

EnTerr commented 8 years ago

@Barbosik - here is your 2nd msg from https://github.com/OgarProject/Ogar/issues/569#issuecomment-219899289 that you said "incorrect uncompressed length". Doing it my way, i get uncompressed length right:

FF:96:00:00:00: total len 0x96 (150)                            [FF 4E]=skip 93, copy 15+4 next
10:
00:00:  
5F:F6:B2:0D:    20:F4:FF:FF:    0B:08:00:00:    3E:01:  00:
83:00:BD:0D:    DA:F2:FF:FF:    2B:0B:00:00:    CB:01:  00:
17:12:BD:0D:    B8:F6:FF:FF:    7A:0B:00:00:    4B:02:  00:
B9:21:BD:0D:    51:F4:FF:FF:    5F:08:00:00:    E1:00:  0A: FF:99:07:   2D:2D:4B:75:72:64:69:73:74:61:6E:2D:2D:00:
BD:21:BD:0D:    28:F4:FF:FF:    86:08:00:00:    38:             [20 00]=back 32, copy the 19                     
                                                  (00:  0A: FF:99:07:   2D:2D:4B:75:72:64:69:73:74:61:6E:2D:2D:00:)
                                                                [00]=WTF?! (No-op, copy 0 more from the same place?)
                                                                [9F]=skip 9, copy 19
B4:21:BD:0D:    15:F4:FF:FF:    83:                             [20 00]=back 32, copy the 19
                                  (08:00:00:    38:00:  0A: FF:99:07:   2D:2D:4B:75:72:64:69:73:74:61:)
                                                                [04]=copy 4 more from the same place
                                                                                                     (6E:2D:2D:00:)
                                                                [60]=skip 6, copy XXX
00:00:00:00:
00:00:
EnTerr commented 8 years ago

@Barbosik - here is decoded the 1st msg from https://github.com/OgarProject/Ogar/issues/569#issuecomment-219899289 : 


FF:96:01:00:00:     len(unpack)=406                                     [F2:82:]=skip 145, copy 6
10:
00:00:
E6:DE:1B:76:    F1:F4:FF:FF:    B6:EF:FF:FF:    64:00:  03: 33:FF:33:
60:00:1A:76:    66:F5:FF:FF:    DB:F2:FF:FF:    0B:00:  02: 07:FF:E4:
9A:02:1A:76:    14:F6:FF:FF:    CF:EF:FF:FF:    0A:00:  02: 0B:FF:07:
04:07:1A:76:    DF:F1:FF:FF:    07:F2:FF:FF:    48:01:  0A: 07:FF:14:   D0:9F:D0:BE:D0:BC:D0:BE:D0:B3:D1:83:20:D1:87:D0:B5:D0:BC:D0:A1:D0:BC:D0:BE:D0:B3:D1:83:00:
D9:19:1A:76:    28:F2:FF:FF:    A7:EF:FF:FF:    B8:01:  0E: 65:07:FF:   25:75:66:6F:00:     4B:6F:6C:69:62:72:69:00:
F0:1E:1A:76:    30:F6:FF:FF:    C0:                                     [61 00]=back 97, copy the 6
                                  (EF:FF:FF:    0A:00:  02:)            [D2]=skip 13, copy 6
                                                            FF:07:4B:
03:10:1A:76:    1D:F6:FF:FF:    C1:F0:                                  [12 00]=back 18, copy the 6
                                     (FF:FF:    0A:00:  02: FF:)        [C1 ]=skip 12, copy 5
                                                               6B:07:
71:10:1A:76:    59:F6:FF:FF:    E1:F1:                                  [12 00]=go back 18, copy the 5
                                     (FF:FF:    0A:00:  02:)            [FF 02]=skip 17, copy 19
                                                            07:34:FF:
16:11:1A:76:    72:F3:FF:FF:    5E:F2:FF:FF:    53:00:                  [85 00]=go back 133, copy the 19
                                                       (0A: 07:FF:14:   D0:9F:D0:BE:D0:BC:D0:BE:D0:B3:D1:83:20:D1:87:)
                                                                        [0F]=copy 15 more
                                                                                                                   (D0:B5:D0:BC:D0:A1:D0:BC:D0:BE:D0:B3:D1:83:00:)
                                                                        [DF]=skip 13, copy 19
17:11:1A:76:    61:F0:FF:FF:    3A:F2:FF:FF:    3A:                     [30 00]=back 48, copy the 19
                                                  (00:  0A: 07:FF:14:   D0:9F:D0:BE:D0:BC:D0:BE:D0:B3:D1:83:20:D1:)
                                                                        [10]=copy 16 more
                                                                                                                 (87:D0:B5:D0:BC:D0:A1:D0:BC:D0:BE:D0:B3:D1:83:00:)
                                                                        [AF]=skip 10, copy 19
0C:11:1A:76:    0A:F3:FF:FF:    13:F1:                                  [30 00]=go back 48, get the 19
                                     (FF:FF:    3A:00:  0A: 07:FF:14:   D0:9F:D0:BE:D0:BC:D0:BE:D0:B3:D1:)
                                                                        [13]=copy 19 more
                                                                                                        (83:20:D1:87:D0:B5:D0:BC:D0:A1:D0:BC:D0:BE:D0:B3:D1:83:00:)
                                                                        [DF]=skip 13, copy 19
02:11:1A:76:    47:F3:FF:FF:    78:F1:FF:FF:    3B:                     [60 00]=go back 96, copy the 19
                                                  (00:  0A: 07:FF:14:   D0:9F:D0:BE:D0:BC:D0:BE:D0:B3:D1:83:20:D1:)
                                                                        [10]=copy 16 more
                                                                                                                 (87:D0:B5:D0:BC:D0:A1:D0:BC:D0:BE:D0:B3:D1:83:00:)
                                                                        [A1]=skip 10, copy 5
26:12:1A:76:    F6:F4:FF:FF:    7C:F2:                                  [D2 00]=go back 210, copy the 5
                                     (FF:FF:    0A:00:  02:)            [90]=skip 9, copy XXX
                                                            88:07:FF:
00:00:00:00:
00:00:

Unpacked length fits, everything fits! The good news is, it's simpler than you imagined - in particular 0x0Z and 0x1Z just mean "copy that many additional bytes from where we were".

The bad news is, i am exhausted having spent all my day on this puzzle. Also, did i mention it's not a great compression? :)