Transmitting signals can take an absurd amount of bandwidth

[TheAIBot]

As described in #23 Transmitting 262 signals for a single tick takes up at least 5.8kb and sending 10 ticks takes 58kb(all signal counts = 5).

Here is some data which describes what uses space. The data below will consider 60 ticks worth of signals where each signals count is set to 1.000.000.000

Content	Size	%
All	489kb	100%
Text describing the signal type and signal name	283kb	58%
Delimiter \0	32kb	6.5%
Delimiter ;	16kb	3.2%
Numbers	158kb	32%

Numbers from this table is taken from the following signal file. Spaces in that file are supposed to be \0 characters.

We need some way to compress this a lot which is what this issue will be discussing. The goal is to end up with a size of ~10kb(probably not possible) in the above scenario. That a 98% compression. Obviously we can't compromise too much on performance as that would defeat the purpose.

[justarandomgeek]

If we can treat "known" signals in a given set (say, vanilla signals using Feathernet or similar numbering - it would be nice if they're at least similar for low values) as a giant array of int32's (edit: as in, giant binary blob of them) then it comes out only slightly more than 1KB for that portion. Could make an extended "universal numbering" that covers other common mods too.

Such a system would require a fall back to full string-specified signals though, or a system for them to be assigned numbers by the cluster to be used across all worlds.

Edit: alternately, clusterio could get a int->signal mapping for each world at startup and use those indexes for that world, and do the translations between as required.

More edits: Could use a pre-determined id list and a packet formed of a list of 16:firstid|16:runcount|32:signal|32:signal|32:signal|... to specify a series of signal groups within the full map when sparse. 16 bits is plenty for ID because that's the size they are internally in Factorio.

And for moving large binary blobs over rcon: unicode has lots of printable characters that won't require any escapes, surely at least base256 is possible to move an array of bytes without too much overhead for character encoding.

[TheAIBot]

So if we implement the id list idea then we could reduce the text space usage with 83% taking it from 283kb to 46.6kb. In total that's a 48% compression. That's a really good start.

We can definetly convert numbers to bytes instead of strings which should also compress quite nicely. I will have to test if this is performance intensive to do in lua.

[justarandomgeek]

Yeah, IDs alone should be a solid improvement just by having gotten rid of most/all the strings, depending on how extreme.

Unsure if binary-blob is worth it, I was just kinda throwing all the ideas all the way to the extreme out to see what would stick!

Also a further note, I've looked again at Factorio's internals and it's actually a 16bit ID for each of Item, Fluid, Virtual, so if we do go for a binary format, we should probably use at least 18 bits there (2 for type, 16 for id). Steal them from runcount I guess.

Another crazy idea: if everything along the way will pass illegal characters without complaint, we can just print out bytes or encode the numbers as a character's codepoint directly. I suspect we'll run into problems with something that evil though... Has anyone done testing on how binary-friendly the rcon connection actually is?

[justarandomgeek]

For a partial answer to the last question: The RCON Spec says body is a null terminated ascii string, but Factorio makes it a std::string (it's unclear from quick reading what character encoding, but i'd guess UTF-8, since it surely isn't plain ascii!), and loads it from the message by declared size. We might get away with some wacky stuff, but probably best to stick to text-like encodings. I'm also not sure how picky the JS lib is on the other side for such hacks.

[TheAIBot]

I will try these things out, just don't have the time right now. From the proposed ideas we could convert everything into bytes and then go from there. So a normal signal could be 1 byte to define the signal type, 2 bytes to define the signal name and 4 bytes to define the signal count. So each signal would be 7 bytes int total. That would reduce the total size to 110kb. We could probably assume that the max amount of signals would be < 16384 and then use the last 2 bits in the signal name bytes to define the type. That would reduce the size to 94kb which is an 81% compression.

As you said if it's important that the string is valid utf8 then we will have to convert the bytes to utf8 which should be less efficient but hopefully not a big problem.

The main thing that i am concered with right now is whether rcon.print actually can handle \0 characters inside a string before the end of the string.

Another issue is that mods between multiple worlds need to implement the exact same signals. So it's possible for different world to have different mods as long as they all implement the same signals. Not sure if they is a desired restriction but i am sure it's possible to work around.

[justarandomgeek]

I will try these things out, just don't have the time right now.

I'm having similar problems getting all the things i want going ;)

so a normal signal could be 1 byte to define the signal type, 2 bytes to define the signal name and 4 bytes to define the signal count. So each signal would be 7 bytes int total.

This gets much denser for non-sparse frames if you use the startid+count and then a series of sequential signals as i described above. The type/id and count could be combined together in four bytes (16 bits for ID, 2 for type, and 14 for count) and then a series of count 4-byte values follow it with no further tags until the next gap (followed then by another tag and value series) - this is slightly less dense than tagging every signal for sparse frames, but the denser the data is, the denser it packs! We can probably assume <16k signals total for now without issue, but facorio does actually allow the full 64k per type.

As you said if it's important that the string is valid utf8 then we will have to convert the bytes to utf8 which should be less efficient but hopefully not a big problem.

So we pick 256 (or maybe 65536?) "safe" characters and break the data in that size chunk and index them into the mapping table to get a character. As you say, less efficient, but much more stable!

The main thing that i am concered with right now is whether rcon.print actually can handle \0 characters inside a string before the end of the string.

Factorio's rcon implementation itself should (admittedly, i haven't tested - just read over it) handle it, and it looks like the lua bits will all do reasonably well with it - though if the JS lib took a more literal interpretation of the spec (loading payload by first-null instead of declared size) then it's no good anyway, since internal nulls are technically against spec. I'm starting to think we should just avoid actual binary data on this very texty protocol.

Another issue is that mods between multiple worlds need to implement the exact same signals

I suggested further up-thread that the clusterio client might pull a mapping table from a world after starting it, perhaps at the same time it injects the WorldID (and before feeding any inbound frames). Clusterio can then do the cross-mappings easily enough in JS outside factorio's main loop. Being a one time thing, it's okay that this table will be a bit large (all those strings...).

edit: Alternatively, instead of pulling a map from each world, we could just pull a signal list from each, construct a combined map and push that back everywhere - individual maps seem simpler though.

[justarandomgeek]

Map building lifted from one of my older computer building projects (for a flat linear space):

local map={}

for _,v in pairs(game.virtual_signal_prototypes) do table.insert(map, {id=#map+1, name=v.name, type="virtual"}) end
for _,f in pairs(game.fluid_prototypes) do table.insert(map,{id=#map+1, name=f.name, type="fluid"}) end
for _,i in pairs(game.item_prototypes)  do table.insert(map,{id=#map+1, name=i.name, type="item"}) end

and split by type:

local map={virtual={},fluid={},item={}}

for _,v in pairs(game.virtual_signal_prototypes) do table.insert(map.virtual, {id=#map.virtual+1, name=v.name}) end
for _,f in pairs(game.fluid_prototypes) do table.insert(map.fluid,{id=#map.fluid+1, name=f.name}) end
for _,i in pairs(game.item_prototypes)  do table.insert(map.item,{id=#map.item+1, name=i.name}) end

[justarandomgeek]

Some test results:

/c rcon.print("foo\0bar")

Causes factorio to emit an rcon response as if the string had ended at the embedded null. I haven't followed this all the way out to see where it gets trimmed, it may be fixable with a patch to factorio.

/c ch={} for i=1,255 do ch[i]=string.char(i) end rcon.print(table.concat(ch))

Causes factorio to emit an rcon response containing a series of bytes counting 0x01 to 0xff, all intact. string.char() fails after 255.

Assuming the JS library allows access to the raw bytes (i haven't looked at it, i'm using a c# rcon tool, which helpfully mangles things further up, but i can see the bytes as they come in...), we can transit whatever we want, as long as it doesn't contain nulls.

I'm gonna need to hack up my tools a bit to test what goes through the other way.

[Danielv123]

@TheAIBot do you still have the test sets used for the breakdowns you made?

[justarandomgeek]

More testing: I swapped out the encoding in my rcon tool, and confirmed that factorio is infact using utf8, with the following test (From my c# rcon tool again, this time with fixed encoding):

>/c rcon.print("☃")
☃

Internal nulls on inbound commands truncate at the null. /c rcon.print("fooXbar"), where X is replaced after encoding with a null, produces an error that the command is incomplete (truncated at the null). Again I haven't followed it all the way through to see where it gets truncated, but it should be at least after reading from the TCP socket (which is done by byte count)

If i get some time later this evening i'm gonna have a go at a decently efficient binary-ish format (based on all the discussion above and this new testing) to send/receive combinator signals and see how fast i can generate/parse this data in lua.

[justarandomgeek]

So i spent some time last night/this morning playing with this, and have a branch on my fork that transits packets as essentially a not-quite-utf8 packed series of ints. I chose this encoding becuase it was relatively straightforward and never contains embedded nulls for non-zero values. Zero-valued signals are simply ommitted entirely, and signal IDs are all non-zero. The source and destination world IDs are pulled out to a header to make unicast easier on the JS side, and are written as -1 if not specified (for broadcast/unknown-source). The packing might could be improved slightly with one additional bit per follow-on byte, but the prefix 1 bit is required to prevent internal nulls in larger values with many low 0 bits. In theory the current system preserves utf8's self synchronizing properties, which this change would lose - we probably don't need it here, we could drop an entire message if malformed. Either way, this encoding is different enough from utf8 (5 and 6 byte encodings, with 6 having an extra bit) that we'll need to parse raw bytes into values ourselves - but that's just hacking on rcon libraries to get the bytes. I also considered a variant where shorter encodings are sign-extended instead of zero-extended, to allow compact representation of small negative values, at the cost of large positive values.

After encoding, values are handled in lua as strings (which lua seems to define as "sequence of non-zero bytes", with no notion of multibyte encoding - factorio handles as utf8 though if you print them anywhere), and stored in a buffer (lifted from #23) for retrieval by rcon. I also updated the receive side to take these data strings and convert them back into frames suitable for application to a CC's params. The maps used to do this are updated in Reset() and available for retrieval in json form by rcon, to allow translation between differing maps.

I was testing in a debug factorio build (with a slight hack for single-player + rcon), so it's not exactly comparable timings, but the mod was taking about 2.2ms when transmitting a decent sized lognet inventory every frame, vs 0.05 when idle. I'll do some more scientific testing soon with a normal Factorio build and more controlled data (timing/size with all=5, all=id, all=rand(), random subsets, etc).

[Danielv123]

Circuit networks are removed to be replaced with whatever @justarandomgeek comes up with for his mod.