LibCompress doesn't work in Roblox

[LoganDark]

I'm planning on using LuaSerializer in Roblox for transfer of data structures between games.

It works if I don't use compression:

(this is using serialize_nocompress and unserialize_nocompress)

However, it doesn't work if I do use compression:

What's going on? I haven't modified the code.

[Rochet2]

You need to check what all characters you can send/store between two games. Most likely you cannot use at least the \0 character. The LibCompress uses all character codes between 0 and 255. Often data sent through web is base64 encoded for the reason that the 64 characters used in the encoding are considered safe to be sent over basically anything without problems.

I hit a similar problem when I was sending data in World of warcraft through chat messages. The reason was that the compression output sometimes uses the character \0 which is used to end a string in C and thus the strings were cut off at those points. I tried encoding the \0 characters into something else, but the end result was almost as big as the uncompressed text.

For this reason I developed lzw compression algorithm in lua and coded it not to use \0 character. You can find the compression algorithm in the zeros branch here: https://github.com/Rochet2/lualzw/tree/zeros

I developed LuaSerializer because I did not find any similar alternative at the time. However I then later found https://github.com/gvx/Smallfolk which was a lot faster and more human readable, so I changed to use that. You can still use the lzw compression with Smallfolk.

[LoganDark]

I was planning on encoding the compressed version as base64. However, I created this issue because compressing and decompressing in the same script does not even seem to work. If I take the output of serialize and feed it immediately into deserialize, it gives me garbled output.

---

After failing to figure out how to make this work, I switched to BitBuffer. That is infinitely better than this is for compactness. Since I was actually planning on creating a string that a user can copy and paste, the ability to make extremely compact and specialized formats is a life saver.

For example, here is a model of an exclamation mark:

QCfLncZ5CfTn+72QAAAAYA////AAAAAC0GqUjE+WO5yyZW2MdIAAAAMg///fAAAAABaDVqRCfLncZ5DDDn6b4QAAa7OgmZmZBAAAAC0GqUjE+WO5yylm+Ma6QAAAAYA////AAAAAC0GqUjE+WO5yyHGGu00nRTHCfPb3B0MzMLAAAAQg2Qla01ysJXmM13IT7+cZIAAAAMg///fAAAAABaDVqRCfLncZ5DDDnZZz0hw3z2dANzMzCAAAAEoNUpGZMNb2CfTn+72QgZmZSgzMzMAYmZmAU9NDjMmmN7hldTvscIYmZGIwZmZGAMzMTAq+mhRGTzmN13whQzMzQAnZmZAwMzMBo6bGGVE/Y2hbHme63uhAuMzcANzMDBAAAAEoNUpGBts80vJLM88yQYUmZWglZm5AAAAAC0GqUjgWWe63kp+GOEwlZmDQZmZWAAAAgAthK1Iolln+NZhyTT+cZppPjmOEkLfyFoZmZWAAAAgAthK1Iolln+NZhyTT+cZmlNTHCylP5C0MzMLAAAAQg2QlaIkpdfuMEeAAkxgYAAoxgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyMLAAAYoDAAAgRAAAAAAAAAAgDAgMGEDAgUGEAAAAAAAAAAAAAAAAAAAAAAAAAAQAOmZBAAAXfEAAA1BAAA8//fAAAcAAkxgYAAoygAAAAAAAAAAAAAAAAAAAAAAAAAAAC4xMHIAAgekAAAwQAAAA///DAAAqAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYxCQQwpmZDAAAAAAAAAqqqCAAAVAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAgsYBIAAgzsBAAAAAAAAAqqqCAAgEAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAB///AE4Z7KAAAAAAAAAAVVVBAAQEACAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//fAg8sdFAAAAAAAAAYVVVAAAICgBAgKDCAAAAAAAAAAAAAAAAAAAAAAAAAAAY0MTAA0nkFA0mZGAAAAoqqKAAAaAABAAAAAAAAAAAAAAAAAAAAAAAAAAAEWxCQAAgmZDAAAAAAAAAVVVFAAANAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAArYBIAAQzsBAAAAAAAAAqqqCAAgOAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAhVsAEEsrZ2AAAAAAAAAwqqqAAAQHAEAAAAAAAAAAAAAAAAAAAAAAAAAAAg4KWACAAwMbAAAAAAAAAoqqqAAAIBABAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAgAqe2uCAAAAAAAAAoqqKAAAkCgEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYZmZAAAAAAAAAAAAAAAAAaBQCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgZm5BAAAAAAAAAAAAAAAAlAABAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAGBVmZGAAAAAAAAAAAAAAAAgaAwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYmZeAAAAAAAAAAAAAAAAQOAYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwA+ECpaiTAAAAAAAAAAfjWAAAoHAMAAAAAAAAAAAAAAAAAAAAAAAAAAAQAYAfChUNxJAAAAAAAAAkvRLAAAEDAEAAAAAAAAAAAAAAAAAAAAAAAAAAAIAAAYkw3MzEAAAAAAAAAAAAAAAACBQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYmZGAAAAAAAAAAAAAAAAAZAMAA0YQAEAAAQAAAAAAAAAAAAAAAAAAAAAAWI1AEoRScAgyMzAAAAwqqqAAAA

Except you can't use it, because only my scripts support it. But only my scripts need to use it, so that's fine!

It's even immune to most hex editors, because none of them think to support formats that aren't byte aligned. It's amazing.

[Rochet2]

Nice to hear you found a working alternative - closing. Consider leaving a link to BitBuffer if its available.

[LoganDark]

Original: https://devforum.roblox.com/t/bitbuffer-module-for-binary-data-manipulation/8315 Repost: https://devforum.roblox.com/t/stravants-bitbuffer-module-compact-storage-of-data-tutorial/20189 The module itself: https://www.roblox.com/library/174612085/BitBuffer-Module

It basically allows you to manually push values to a buffer, or manually consume values from one. You're in complete control of how many bits you use. It's pretty cool, consider checking it out.