Suggestion / Comment: Pre-process files rather than compressing on the fly

[kellybyrd]

While reading the description of this plugin, one thing that occurred to me is that this plugin is directly in the path for every serial send (unless I misunderstood). While the compression itself may take very little CPU, having something else "in the way" could be the source of problems (see DisplayLayerProgress's github for issues about layers shift as an example of something that gets called often potentially being a problem). Some folks have a lot of things running on their RPi, and it's not exactly a real time OS.

Instead of processing the g-code stream live, you could do it beforehand, similar to how ArcWelder does it. With ArcWelder, the output file can overwrite the original, or just create a new file.

[scottmudge]

There is no downside to having an interface layer between OctoPrint and the firmware. I mean, the plugin is just replacing the existing Serial interface with an expanded version... so in reality the vanilla serial interface is already something "in the way". In that regards, this is not an additional layer, it's a replacement layer.

The only feature which might interfere with custom commands is the whitespace removal feature, but that is only enabled and used for "G" commands, which according to RS-274 standards should behave exactly without whitespace as with. So this is not a problem.

Making it pre-process files would be a complete departure from the existing architecture and design, and would go against the original intention. And I'm not sure what the benefit would be, it would only present downsides. You would not have benefit of compressing the various control commands that are sent routinely.

And finally, with OctoPrint anyway, preprocessing would not work at all with firmwares which use checksum validation for printing (all that I am aware of). The checksum byte and line number are added to the g-code on the fly by OctoPrint, so you would be mixing uncompressed and compressed data, which doesn't make sense. Especially considering that uncompressed data consumes 50% more data while MeatPack is enabled.

The plugin is meant to compress serial data that is primarily g-code. Compressing the g-code before it even gets to the serial port presents too many complications, and you end up losing all of the benefit.

Let me know if you find a conflict with this and and any existing serial commands or data, but I have not found any, and I do not anticipate there being any. I have used many different plugins and custom/arbitrary data transfers, and it works just fine. So long as the characters being sent are within the standard ASCII range (or below 0xF0), it will send just as it normally does.

[kellybyrd]

I was speculating based on limited info, not pointed out a concrete problem. My thinking was that if OctoPrint has to call out to your plugin and your plugin has to do some work and return, that's one more set of function call than are there right now.

I had not considered your points about the various routine commands sent as part of the stream (like all the other things OctoPrint plugins can add like OctoLapse or display updates, temp readings, etc). Also, I had no idea there was a checksum feature!

I don't think it's fair to say you lose "all of the benefit", but I now understand what you intended to do, you really do want to be in a position to translate the entire serial stream in order to get max compression!

[scottmudge]

Perhaps that was unfair, there would still be some benefit, just considerably less as the firmware-side code would become much more complicated, as it would need to decode intermingled compressed and raw data without a structured stream. I did originally compress the g-code files at first to test the approach initially (when printing off of SD cards), but when I was implementing the same approach in OctoPrint, it became clear it wasn't going to work.

Fortunately this plugin is not an additional layer that OctoPrint has to additionally interface with. As far as OctoPrint is concerned, it's doing nothing differently. It's just sending serial data to an interface object. Except instead of a vanilla serial.Serial() instance receiving it, it's a custom wrapper which packs the data before passing it to the base class. Using c-type lookup arrays, the packing process is incredibly efficient, so there shouldn't really be much of any computational overhead.

[scottmudge]

I did want to thank you for the suggestion, however. I apologize for the litany of reasons why not. There's almost always something useful to think about with any idea, so I appreciate you taking the time to contribute. 👍

[kellybyrd]

No apology needed! I shot off an unsolicited not-very informed idea, your reasons taught me more about how your plugin and OctoPrint handles serial coms (or at least the hook it offers for this)!