PlummersSoftwareLLC / NightDriverStrip

NightDriver client for ESP32
https://plummerssoftwarellc.github.io/NightDriverStrip/
GNU General Public License v3.0
1.32k stars 213 forks source link

Web UI: Mesmerizer Monitor #302

Closed davepl closed 1 year ago

davepl commented 1 year ago

I've added support for pulling the state of the current display via a socket, and it would be nice to display the result in the UI on a preview page.

To fetch a packet, open a socket to port 12000. You can start reading packets out of the socket which have this form:

typedef struct { uint32_t header;
uint32_t count;
CRGB colors[NUM_LEDS]; // Array of LED_COUNT CRGB values } ColorDataPacket;

The header will always be 0x434C5244 The count is the number of LEDs, and it should match the width*height

You would then draw a matrix in the HTML with dots representing each of the pixels in a 64x32 matrix populated with the colors. You can simply continue pulling packets as long as they are available, as one will be sent every new frame.

davepl commented 1 year ago

To the extent its helpful, here's a Unity client I wrote to do this, but I'd rather have it in a normal web page than as a Unity app!

public class MatrixStreamer : LEDControllerChannel { public DotMatrix dotMatrix; private ColorDataPacket packet = new ColorDataPacket(); private readonly object packetLock = new object();

protected override void DrawLoop()
{
    for (; ; )
    {
        try
        {
            TcpClient client = new TcpClient("192.168.8.104", 12000);
            using (NetworkStream stream = client.GetStream())
            using (BinaryReader reader = new BinaryReader(stream))
            {
                while (true)  // Continually read from the socket
                {
                    uint header = reader.ReadUInt32();
                    if (header != 0x434C5244)
                    {
                        // Not our header byte, so continue scanning until we run out of data
                        // or hit the header byte
                        continue;
                    }
                    else
                    {
                        uint count = reader.ReadUInt32();
                        if (count != Width * Height)
                        {
                            Debug.Log("Size mismatch between data and matrix");
                        }
                        else
                        {
                            CRGB[] colors = new CRGB[Width * Height];

                            lock (packetLock)
                            {
                                packet.Header = header;
                                packet.Count = count;
                                packet.Colors = colors;

                                for (int i = 0; i < Width * Height; i++)
                                    packet.Colors[i] = new CRGB(reader.ReadByte(), reader.ReadByte(), reader.ReadByte());

                            }

                            //Debug.Log("Packet Received!");
                        }
                    }
                }
            }
        }
        catch (SocketException ex)
        {
            Debug.Log("Exception reading from socket for " + HostName + ": " + ex.Message);
        }
        catch (Exception e)
        {
            Debug.Log("Unhandled Exception reading from socket for " + HostName + ": " + e.Message);
        }
    }
}

// Update is called once per frame
public override void Update()
{
    var displayModel = dotMatrix.GetDisplayModel();
    Debug.Assert(displayModel != null);

    lock (packetLock)
    {
        if (packet.Colors != null)
        {
            for (int x = 0; x < Width; x++)
            {
                for (int y = 0; y < Height; y++)
                {
                    CRGB color = packet.Colors[y * Width + x];
                    int content = (int)color.GetCode();
                    displayModel.SetDot(x, y, content);
                }
            }
        }
    }
}
zephyrr commented 1 year ago

How is Height and Width determined by the socket reader? Does that have to be manually configured?

davepl commented 1 year ago

Yes, in my example there are three things that can be set: IP, Width, and Height

On Jun 1, 2023, at 9:54 PM, zephyrr @.***> wrote:

How is Height and Width determined by the socket reader? Does that have to be manually configured?

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/302#issuecomment-1573142519, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4HCF54L5JAQ5VRIJNVZ2TXJFWYVANCNFSM6AAAAAAYXSKCAE. You are receiving this because you authored the thread.

zephyrr commented 1 year ago

Would there be value in retrieving the current width and height via port 12000?

typedef struct { uint32_t header; uint32_t width; uint32_t height; // use 1 for linear strip CRGB colors[NUM_LEDS]; // Array of LED_COUNT CRGB values, == width * height } ColorDataPacket;

(Or one could make width and height uint16_t)

In this way the caller could automatically adjust to the current width and height.

Or if you REALLY want to configure the caller locally, one could replace

if (count != Width * Height)

with

if(width != config_width || height != config_height)

I'm thinking that such a change would be easiest now, before there is much code written assuming the other.

davepl commented 1 year ago

I could add that, sure. BTW the server side of this code is checked into the LEDViewer branch of davepl/NightDriverStrip and hasn’t been migrated to the main branch yet. I will do that shortly.

If you’d like this fields add, open an Issue so I don’t forget, and I’ll add them promptly!

On Jun 2, 2023, at 4:37 PM, zephyrr @.***> wrote:

Would there be value in retrieving the current width and height via port 12000?

typedef struct { uint32_t header; uint32_t width; uint32_t height; // use 1 for linear strip CRGB colors[NUM_LEDS]; // Array of LED_COUNT CRGB values, == width * height } ColorDataPacket;

(Or one could make width and height uint16_t)

In this way the caller could automatically adjust to the current width and height.

Or if you REALLY want to configure the caller locally, one could replace

if (count != Width * Height) with

if(width != config_width || height != config_height) I'm thinking that such a change would be easiest now, before there is much code written assuming the other.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/302#issuecomment-1574438101, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4HCF5PZRAPFSRGF3CAAY3XJJ2L5ANCNFSM6AAAAAAYXSKCAE. You are receiving this because you authored the thread.

Louis-Riel commented 1 year ago

@davepl, what are your thoughts on adding support for websockets for these comms, in addition to this pure socket implementation. It would make it possible to do all that in the browser, and allow for cool tricks, like have the audio sensing kit send the pulse event on the websocket, and have visual feedback of that. Not yet sure if that comm feature is actually useful/worth the effort, or if it goes against the logical architecture of the solution.

davepl commented 1 year ago

I actually started with web sockets, and at least at the time, the implementation on the ESP32 seemed buggy and slow. That was a few years ago, it could have improved, but since the perf with raw sockets is excellent, I’d be leery of anything that increased overhead.

Is there a concrete scenario that’s enabled by web sockets? Or are they easier to program from a web page than a raw socket? I guess I’ve never done a web page that did that!

On Jun 3, 2023, at 9:59 AM, Louis Sebastien Riel @.***> wrote:

@davepl https://github.com/davepl, what are your thoughts on adding support for websockets for these comms, in addition to this pure socket implementation. It would make it possible to do all that in the browser, and allow for cool tricks, like have the audio sensing kit send the pulse event on the websocket, and have visual feedback of that. Not yet sure if that comm feature is actually useful/worth the effort, or if it goes against the logical architecture of the solution.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/302#issuecomment-1575069541, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4HCF6LDUHZ6SCF5SATR2LXJNUO7ANCNFSM6AAAAAAYXSKCAE. You are receiving this because you were mentioned.

Louis-Riel commented 1 year ago

It's just sugar on top. Maybe a future thing. It's not possible to use raw sockets in a web page, web sockets are the only option.

It would be useful for the UI to have a web socket to handle system events. For example, if the effect change could be implemented over websocket, the UI would no longer need to query the chip periodically to confirm the effect change, it would react to websocket messages. This one would be useful and optimize the web app in this respect. As mentioned in my message, sound analysis could send messages for peaks, and that could be reflected in the UI. This, so for, is just bells and whistles, not yet a true step up, from a functionality perspective, it's more from an efficiency perspective.

As far as I know, it is not possible to interact with raw sockets in the browser. So websockets would allow for that type of comms in the browser. Websockets typically send jsons, but they do support streams as well, if the chunck of data to send is big enough, or if the nature of the data is an actual stream. If we have access to these socket functions, we could maybe build a web effect designer, and use the websocket to send the bytearrays to see them go.

One feature that websockets would make a good difference is if we want to have the effect triggerable ad-hoc from the Web UI. If we use POST requests for that, there is a slight delay. That is greatly reduced for web sockets. So if we ever wanted to do some sort of visual loop station, websockets would be required to make it work well enough to be usable.

From a buggy and slow perspective, I can't speak to that, I've only done it with ESP-IDF, and it's very stable. The esp thing I wrote is used for home automation, and that is handled by Node-Red, a workflow management system. Node-red opens a websocket on the different esps I got, and when ever the epss publish events (pins triggered, IR code received), node red figures out the happenings that need to happen from the event, and sends a POST message to trigger pins resulting from the happening. From a stability perspective, I've had chips stay online, with the websocket connected, for over 30 days without any disconnects.

rbergen commented 1 year ago

I've actually been casually thinking about the web socket thing, and for now I would only see a case for scenarios where something changes at the device side that is not triggered by the web UI. That would currently basically only be the interval-based effect change. Which is predictable to an extent, so a case but not a decisively strong case either. Of course, this is based on the premise/assumption that a user will typically use one web UI instance (read: browser) to control their device. The nuisance of the delay caused by POST requests is a matter of personal perception, but personally I think it's acceptable to the point it doesn't warrant the web socket overhead - yet.

davepl commented 1 year ago

OK… assuming it’s not a crap-ton of overhead or code, I’m OK with adding a web socket for at least the color data server and for anything else the web client might need like it.

Is that something you want to add to the ColorData server’s current socket impl, or would you like me to do it? If so, please raise an issue and assign it to me!

Thanks Dave

On Jun 4, 2023, at 1:05 AM, Louis Sebastien Riel @.***> wrote:

It's just sugar on top. Maybe a future thing. It's not possible to use raw sockets in a web page, web sockets are the only option.

It would be useful for the UI to have a web socket to handle system events. For example, if the effect change could be implemented over websocket, the UI would no longer need to query the chip periodically to confirm the effect change, it would react to websocket messages. This one would be useful and optimize the web app in this respect. As mentioned in my message, sound analysis could send messages for peaks, and that could be reflected in the UI. This, so for, is just bells and whistles, not yet a true step up, from a functionality perspective, it's more from an efficiency perspective.

As far as I know, it is not possible to interact with raw sockets in the browser. So websockets would allow for that type of comms in the browser. Websockets typically send jsons, but they do support streams as well, if the chunck of data to send is big enough, or if the nature of the data is an actual stream. If we have access to these socket functions, we could maybe build a web effect designer, and use the websocket to send the bytearrays to see them go.

One feature that websockets would make a good difference is if we want to have the effect triggerable ad-hoc from the Web UI. If we use POST requests for that, there is a slight delay. That is greatly reduced for web sockets. So if we ever wanted to do some sort of visual loop station, websockets would be required to make it work well enough to be usable.

From a buggy and slow perspective, I can't speak to that, I've only done it with ESP-IDF, and it's very stable. The esp thing I wrote is used for home automation, and that is handled by Node-Red, a workflow management system. Node-red opens a websocket on the different esps I got, and when ever the epss publish events (pins triggered, IR code received), node red figures out the happenings that need to happen from the event, and sends a POST message to trigger pins resulting from the happening. From a stability perspective, I've had chips stay online, with the websocket connected, for over 30 days without any disconnects.

— Reply to this email directly, view it on GitHub https://github.com/PlummersSoftwareLLC/NightDriverStrip/issues/302#issuecomment-1575456700, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4HCF2H6WWHBVQVI7WA3GLXJQ6TDANCNFSM6AAAAAAYXSKCAE. You are receiving this because you were mentioned.

rbergen commented 1 year ago

Superseded by #356.