[FR] G code command to report LCD framebuffer

[logiclrd]

Description

I operate a printer remotely, and wish that I could see what's on the 12864 LCD remotely as well. I already have a camera pointing at the print area, I'd currently have to point a second camera at the screen as well in order to stream this data.

I'd like a G code command that I can use to request that the current LCD framebuffer be sent back across the serial link, so that I can view it without having to be physically present. I believe there are details shown on the LCD status screen that aren't available for retrieval as raw data with existing G code.

The raw data for the 12864 screen is only 1 KB of data (128 x 64 = 8192 pixels, and each pixel is 1 bit of data). This, of course, would be unsuitable for the serial link because it contains control characters -- the simplest would be a hexadecimal encoding, resulting in 2KB of hex characters for the full framebuffer.

Feature Workflow

Suppose the new command is called M251 (because M250 is another LCD-related command). Then, any time M251 is submitted over the serial link, Marlin would respond with 2048 hexadecimal characters representing the contents of the 12864 screen's framebuffer at that time.

Sample:

> M251
< 071C7000000000000000071C0000000008A2890000000000000008A24000000000829A8000000000000008A6A03FFFF00104A900000000000000072A403800700208C80000000000000008B2003186300410880000000000000008A2002387100FBE7000000000000000071C00278790000000000000000000000000002F87D0007F80000000000000000410402F03D000FFC00000000000000002082020301000FFC00000000000000002082020781000FFC000000000000000041040207810007F8000000000000000082080203010007F80000000000000001041002F03D000FFC0000000000000001041002F87D000FFC00000000000000008208027879000FFC000000000000000041040238710003F0000000000000000000000318630001E00000000000000001FFFF8380070000C00000000000000001FFFF83FFFF000000000000000000000000000000000071CF8000000000000000F9C00E7DF6408A2090000000000000000A241141064008212800000000000000122A0179E0801043100000000000000011E40204110020808000000000000000202004041200410880000000000000002040084514C0FBE7000000000000000021801F38E4C00000000000000000000000000000000FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF80000000000000000000000000000001910010E7C08801CF8407C0047C07C00191003110408802208C00400C040400018A001010805000211400800408078001840010218020004324010004180040018A00104040200080BE0200040400400191001084402001088404000444C44001910039F3802003E70407C00E38C3800180000000000000000000000000000001FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF000000000000000000000000000000000000000000000000000708008100000000000000003FC000000898018300000000000000003FE00000098860850000001C021CFB203FF000000A8860890000001186220B203FF000000C88008F8000001D422610403FF000000888608100000011822A10803FF00000071C61C100000011423221003FF000000000000000000001422222603FF3FFFFFFFFFFFFFFFFFF00071C22602013FFFFF000000000000100000000002013FFFFF000000000000100000000003FF3FFFFFFFFFFFFFFFFFF0000000000000000000000000000000000000000000000000000000000000000F0020020800000000000000000000000880000200000000000000000000000008AC62CF98B1C00000000000000000000F32232208CA2000000000000000000008202222088A2000000000000000000008202222088A20000000000000000000082072219C89E000000000000000000000000000000020000000000000000000000000000001C00000000000000000000

Additional Information

Some form of packing, or perhaps run-length encoding, could reduce the response size considerably, but would also greatly complicate the feature and increase the likelihood of bugs in the code, not to mention the size of the code.

I don't know if there is any potential issue with sending a 2KB response over the serial link on a regular basis -- could this interfere with timing and disrupt prints, or is the serial I/O well-segregated from other concerns?

If a caller decided to send M251 four times a second, the responses would end up consuming a bit over half of the nominal capacity of a 115200 baud link, or about a quarter of a 250000 baud link.

[AnHardt]

That might be a bit more difficult than you expect. The complete framebuffer is not available on the processor Marlin runs on - only on the display - the processor buffers only slices of the complete screen. Most displays have only a one data direction connection. They can receive date but not send (missing MISO). It would need a serial-device for u8g-lib or a modified u8g-driver for one of the usual displays.

For the already mostly overloaded AVRs sending two data-streams will be tooo much. Sending only one for the ST7920s in a akzeptable quality is already difficult enough.

[logiclrd]

Huh, okay :-)

I'm running a BTT SKR 1.4 personally, I think that's got plenty of additional memory and CPU cycles for what's going on, compared to cheaper boards? I guess I'll just need to experiment with things in my own personal build, on my hardware I could probably easily afford to have a clone of the framebuffer in system memory ready to send, but I can totally see how that would not be a desirable solution for many target boards.

[logiclrd]

I just looked it up, and the SKR 1.4's processor is a Cortex M3 32-bit ARM chip running at 100MHz and 64KB of SRAM.

I wonder how I can tell how much SRAM is currently in use. If there's 1KB to spare then this approach could work :-)

[logiclrd]

Looks like I'd have to rebuild with #define M100_FREE_MEMORY_WATCHER.

[logiclrd]

Or would it work to just make a static char[1024] and see if it builds? :-)

[logiclrd]

Alternately, I could have it cache all the parameters from which the status screen is rendered, and report those on-demand, and port the rendering itself to another platform to consume the data (actually I already did this last night for the fun of it :-P that's where the sample hex data in the Feature Workflow section came from). But, this is a much less maintainable option, as it would be easy for it to fall out of sync with future changes to the status screen.

[rhapsodyv]

Why do you want see the screen? Isn’t enough a gcode to the status of the printing? Is there any information on screen that you can’t get using gcodes?

[logiclrd]

I want to be able to overlay the actual screen on a webcam feed when monitoring it remotely.

[rhapsodyv]

You can overlay the text info returned by g codes that marlin already support. It is a lot simpler. And those g code are reusable.

A frame buffer report may work only on very specific tiny screens. A 3.5" TFT have 300Kb of data. No board have such memory for that. Beside that, it would involve a lot of work.

By now, I will close this as it is too specific for a use case, and could be a lot of work to transmit information that are already available in Marlin.

Closing this don't mean it cannot be done. If you want, you can submit a PR for that.

Thanks for your help!

[logiclrd]

Just to be clear, I'm not talking about the graphical mode, which as I understand it Marlin doesn't even have visibility into because it's running on the screen's firmware. I'm talking about Marlin's 12864 framebuffer:

This screen has 8,192 1-bit pixels. No matter what hardware it's running on, it's always 1KB of data.

[logiclrd]

Also, I'm pretty sure there are details on this screen that aren't available for querying via G code. I mentioned this in the initial comment when opening the issue. For instance, what command retrieves the current LCD message (M117)?

[logiclrd]

I'm a little bit disappointed, because I answered all these questions in the initial feature request description. Yes, there is information that can't be retrieved with G-code, no, the amount of data doesn't vary. I even analyzed the quantity of data it would be transferring. And then you closed the issue saying, "Just use existing G-code, and plus some screens have very large framebuffers." Sigh. :-(

[sjasonsmith]

Unfortunately with over 400 feature requests, very few of them can be implemented. We have to look for those which provide the most value to the maximum number of users.

In this case the preferred solution would be to obtain the needed information through G-Code, rather than relate it to the configured display in any way. Aside from normal G-Code commands there is also an ExtUI interface which allows external serial devices to interact with Marlin for UI purposes. That would probably be the proper mechanism for adding the capabilities you need, rather than relating it to the configured display in any way.

[logiclrd]

That's fine and makes sense, and thanks for the tips. My feeling of disappointment mostly centred around the fact that the reply felt dismissive because of the evidence that the things I had written hadn't even been read. The rationale that was expressed for closing the ticket wasn't actually valid. That's not to say that there isn't a rationale for closing it (you explained one), just that the message given right before it was closed revealed that the action was taken having read only the title of the issue and making broad and incorrect assumptions.

[rhapsodyv]

What part felt “dismissive” or “making broad and incorrect assumptions”?

I don’t want to start a discussion or anything else. Don’t get me wrong. I just want to know where I could be said something wrong.

I’m not a native English speaker. Could be that. But I took a look in everything wrote here, and I have been working with Marlin UI/LCD/TFT code for about 6 months.

I tried to give you better ways (in my experience) to solves your problem, and tried explain in a example that you proposal is not generic enough. And, the last part about submit a PR was not joking, it’s real: because seems (to me) that you have some coding skills.

Really: I don’t want to start any fight. I just want to understand what I did wrong to communicate better next time.

Thanks.

[logiclrd]

I understand what you're saying. Specifically, it was that I created an issue saying, "I want to capture the 12864 monochrome display image, this will always be 1KB of data but I am worried about whether sending even that much data over the serial line could be disruptive. Here are some numbers for baud rates on the higher end. I'm pretty sure there are pieces of info on the screen that can't be retrieved with G code." When I read the reply, though, it basically came across as, "I'm just going to go ahead and close this, because there are other types of display and VGA frames are 300KB and I'm pretty sure you can query the data with G code anyway." I felt unheard. But I do recognize that it is a nontrivial feature that only a few people would use.

In fact, I have already had a stab at implementing it but I don't know what the limitations of the hardware are, or how to know if I am approaching or exceeding them, or even how properly to wire up a new G code command type -- my new build doesn't recognize my new command. :-P I must have missed something somewhere, but I haven't had a chance to take another look at it yet, and I decided to bring it here to find out what more seasoned developers would think and how they might approach it. I described the feature but not the things I have tried, and maybe that was a mistake seeing as the discussion hasn't gone terribly far at this point. :-P

[rhapsodyv]

Ok, I got what you mean now. I really wasn't trying to make less of your PR with that. I was just trying to give an example (now I see that it was badly chosen) that your FR would not be too generic, compared with the work that it would need. Sorry for that. Next time I will try to elaborate a little more my response.

Thanks for the sincere feedback!

[thinkyhead]

It's been quite busy around here, so it often takes many weeks before I get around to reading a new feature request. As for this feature request — or feature concept — I concur with other commenters that a "remote display" is better handled in a manner similar to the PanelDue, requesting the current state in JSON format and then displaying that data in its own layout. That work is already underway, and hosts can of course use the same information and show a "simulated LCD" if they want.

[logiclrd]

This sounds promising. This JSON data will include literally everything the 12864 display is showing? If the 12864 code were refactored so that the data were aggregated in an intermediate structure and then that structure were serialized in response to some G code command, it'd ensure that all the information needed was in the structure. :-)

[logiclrd]

10610

[thinkyhead]

Nerp. There is no "frame buffer" that we could "just send" to "some client" because the only buffer for HD44780 exists on the device itself. We send command strings over SPI to update bits of the display as-needed and the HD44780 handles the details and keeps its own buffer. There is no intermediate layer (in Marlin) where we could maintain an 80-byte buffer, filtering SPI communication and reproducing what is seen on the display, nor would it be a trivial task. The LiquidCrystal library currently manages all these details.

[logiclrd]

In my own experimentation, I played with updating u8glib to maintain a framebuffer and expose it as a static symbol. Every drawing primitive just also includes an update of that buffer. This builds, but I must be missing something for the definition of the new G code command I invented because it doesn't accept/recognize the command.

[thinkyhead]

This commit demonstrates all the elements required for a new G-code: b6a32500c4

[sjasonsmith]

Why are we pursuing this, rather than ExtUI or the JSON option you mentioned? I don't like the idea of another special-case feature in Marlin that will only ever be used by one person, and isn't portable to printers using different screens and configurations. It seems like more strange code to maintain with little actual benefit.

[sjasonsmith]

Another benefit of an ExtUI/JSON solution is that it does not tie the user interface to the functionality of the client. Relying on the screen buffer is the equivalent to "screen scraping" a web page. It all works fine until the web page changes, then it all breaks. It would be much more robust to expose the needed information in a functional way, rather than relying on assumptions.

For example, what happens when somebody switches to use the Prusa-style layout? Or just enters a menu? Your client code would have to figure out which menu is being used, to determine whether the requested data is even present.

[logiclrd]

I actually want to remotely monitor the screen. It's not about extracting information programmatically. I want the functional equivalent of pointing a webcam at the screen, but without needing the hardware.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.