Improving the web interface and API

[T-vK]

I'm currently trying to basically rewrite the whole web interface thing because the way it is currently written it is completely unmaintainable imo.

For example big chunks of html code within the javascript code, jquery is available but not used consistently, everything is written in functions - no classes etc, lots of global variables, a unconventional request/response system is forced onto websockets, very unclear api (e.g. sending the letter "I" will respond with the current system tick of the esp etc), .........

If you ask me websockets should only be used for data that the server is either continuously sending to the client or for data that the server should send to the client as soon as it changes. (E.g. the system clock or a change on a GPIO pin). Everything else (e.g. saving settings or uploading files) should use a traditional http api.

I would really like to do all this and I'm willing to write every piece of javascript code that needs to be written. But I really need help with the backend.

So if someone who knows c a bit better than me is willing to do this with me that would be awesome.

[cnlohr]

Oh boy do we disagree!!! I spent a great deal of time trying different things with websockets, and parsing data finding the fastest way. Turns out using strings is faster than JSON which confuses me! So, I am going to stick to my guns on the frontend <- websockets -> backend communication, i.e. "I" etc.

Considering the bulk of development was done on a livestream, there is a lot that could be done much better!!! It is unusual that all websockets is a challenge-response format. In other non-ESP implementations I have broken from that. But, by having the challenge/resposne format, you can handle the buffers more easily. You won't ever have to send data while there's data waiting in the send buffer.

TL;DR: Events vs Challenge/Response: There is push and shove. Challenge/response can do everything Event-based can do. I chose it.

Avoid traditional HTTP API. Will be bulky.

Rewrite frontend: would be great!

Could add to the protocol another layer to cleanup its comms.

[leopck]

@T-vK yeah, I had planned to fix up the website later too, it's great that you're gonna tidy it up. I can assist in the front end and back end part. What's there you want to change in the backend part?

Would be great if we can work together on this, since I'm thinking of creating another page on 'Developer Guide' on the wiki. I'll be able to insert the website's API on there as well.

Btw, I've been actually working on another front end for the espusb HID example, but the front-end has a slightly different objective from this so I think it won't be an issue? Just informing.

[T-vK]

@cnlohr Well, it makes perfect sense that strings are faster than json if you take the parsing time into consideration. But I'm not saying we have to use json. The way the code is written is just very unfortunate. I guess it would also work to define a bunch of constants like const SYSTEM_TICK = 'I' and would become much more readable. But the speed doesn't even really matter because I would like to change the websocket API in such a way that you just tell the server to e.g. "start continuously sending the system clock" or "stop continuously sending the system clock". So you just have to send data very rarely anyways.
I also don't think that an HTTP API would add much bulk to the application. I mean the HTTP server is already in the code otherwise we couldn't request the files for the web interface. So it would just be a matter of defining additional routes. And overhead really doesn't matter as I would only use HTTP for things like saving settings.

All in all the web interface would gain some performance and overhead would actually be reduced significantly in places where it matters.

@leopck Let's wait for cnlohr to get back on this first.
I also had some ideas for the HID, like an option to make it fullscreen on mobile devices and support for some gestures.

[T-vK]

Well, I just wrote a a little library to get a proper interface for the request/response websocket thing: https://github.com/T-vK/Responding-Websockets/blob/master/responding-ws.js
I couldn't really get it to work yet though, as my esp seems to crash or something as soon as I use the class... Not sure what's happening there, but in theory the library would allow us to make the frontend code way more readable without changing the interface on the backend for now.

[leopck]

@T-vK Done. https://github.com/T-vK/Responding-Websockets/pull/1

[cnlohr]

@T-vK I tried explaining before... You are going to run into the buffering problem. If it is push based instead of poll based, data will start to buffer at the socket. You might then run into a challenge-response, but the buffer is already full (or occupied). So you would need to actually store your own buffer instead of using the lwip buffer. I admit that it is easier to do this way (as why in commercial projects that use this websockets interface I do exactly that! but they are running on Linux boxes)...

If we do enable write buffering (definitely possible) with the ESP, it may work out, but it still makes me rather uncomfortable. In writing this, I might be willing to budge on it and let the architecture change. I just don't have enough time now to do it.

P.S. the way I did it in the commercial implementations was instead of responding to packets with 'I' -> returns an 'I' any push data comes in, prefixed with '_' -- that way it isn't confused with the response codes.

Additionally, I don't want to drop the challenge/response pattern completely because there are so many situations where it makes a lot of sense! I.e. "what is this value, when you get it, please call this function" and then to be able to get callbacks in Javascript is really convenient.

I will dig into this more...

I also regret that I'm so distracted at the moment with the vive stuff :(. I feel rather bad discouraging you, but, I am more than ready for you to prove me wrong!!!

[T-vK]

@cnlohr I don't really understand your concern about buffering. But that might be because I don't know how you implemented the backend.
In theory any buffering should be reduced if we'd change the interface in the way that I was thinking.

I also fully agree that things like gpio pin states should be sent to the client using websockets. But I disagree with the current approach. At the moment you have to send a message to the backend every time you want an update on the gpio pins. Instead of doing this, I would suggest to let the backend watch changes to gpio pins and send them to the client automatically.

Oh nice you have a vive? I have an oculus, but feel pretty bad that I barely ever use it. :p

[cnlohr]

@T-vK The fundamental problem is sockets on the 8266 are not set up in such a way where you can say "send this to the client" ooh "send this, too" oh my gosh "send this as well!" like BSD sockets. That feature can kind of be turned on in a really bizarre way.

For more info, see esp82xx/http.c

    //http://bbs.espressif.com/viewtopic.php?f=21&t=320
    espconn_set_opt(pespconn, 0x04); // enable write buffer
    //Doing this allows us to queue writes in the socket, which
    //SIGNIFICANTLY speeds up transfer in Windows.

It might be all we need to enable pushing? (That was not set until recently) It would require a lot of testing.

If you are willing to do the testing, and it works out, I would be happy to accept it in, since you are correct that push events are very powerful.

[cnlohr]

tl;dr my fears may have been obsolete, I encourage you to pursue your goal of push messages. If you're up for it, please use _ to indicate a push (unassociated) message so I can maintain my closed-source stack in parallel.

[T-vK]

@cnlohr Now I get it. Yes you would obviously need some kind of buffer. But it could be pretty small as long as you ensure that you don't queue multiple packets of the same kind (like system clock). Anyways...

my fears may have been obsolete

So it might actually be fairly easy to implement on the backend?

@leopck If you could look into this that would be awesome. In the meantime I might actually try to get it working with the current interface:
https://github.com/T-vK/espusb-api
But it really would simplify things immensely if we'd change the backend interface.

[leopck]

@T-vK I'm sorry probably I don't really get it. But change the backend interface? Change it to what? From websocket to ?(push events?, REST?)

[T-vK]

Well, yes. For instance the status of the gpio pins should automatically be sent when they change. For the system clock it would probably make sense to just send it every 500ms.

I'd also be interested in knowing what the http server on the esp is capable of. For instance I think it would make a lot of sense to upload the images for the flashing via http.

In my opinion it would also make sense to use http (rest) for: wifi scanning, settings changing/reverting and rebooting.
The overhead doesn't really matter because you only send the packets once. And by providing standardized interfaces people can actually use curl or anything to interface the esp.

While using websockets for that would reduce overhead for these requests, it also requires a lot of custom coding on the client and server side and since it is non-standard you are pretty much bound to use the web interface. No simple curl scripts or thelike to flash a file.

[cnlohr]

It sounds like the only real benefit to switching things to an HTTP interface over a websockets interface would be interoperability with things like curl/wget. Thankfully, the system already supports this!

The very first interface was actually over HTTP and I never removed the interface. Check out: esp82xx/fwsrc/http_custom.c - and look at the issue() function.

This function lets you use the same command interface, except with an HTTP call instead of connecting to the websocket. Another benefit of the unusually lettered protocol is the exact same interface is also available in UDP :-D.

For the http interface, you can issue:

http://192.168.4.1/d/issue?ehello

to which (I haven't tested) it should respond:

hello

The existing interface would not be able to upload firmware or anything. I've geared it mostly around a GET interface, and a fair bit of work would need to be done to enable chunked, or similar requests.

Needless to say, it would be more awkward to use the http interface if the websockets interface is already established.

I hope this clarifies what is available.

[T-vK]

You can see an http upload coming by looking at the request header. It would just be a matter of writing the file to the right place. (RAM would probably be a bad idea.) No need for chunking etc.

And I would say that things like 'less code', 'more readable code', 'easier to understand code' are pretty "real" benefits. ;)

[T-vK]

@cnlohr I have a some questions regarding the codes that are sent for the keyboard input.
Do the codes for the keyboard input follow any kind of convention? They don't seem to be ascii or scan codes.
Are the codes specific to your keyboard layout?
Is there a way to release keys individually? (It seems like there only is CK0\t0 which releases all keys. Or am I mistaken?) I mean I guess you can release all modifiers or all other keys. But it doesn't seem to be specific to individual keys.

Another question I have is regarding the end of your onMessage function. Basically every time you get a response from the server, you send another message (wx) to the server (which causes another response) from what I understand. Inspecting the websocket traffic seems to confirm my suspicion. Hundreds of packets wx are being sent within seconds. And that is without any user input on the web interface.
So I tried to just leave that part out. Now the weird thing is, when I leave it out my message are still sent and I still get responses, but the sockets constantly disconnect causing all my messages to get buffered and being sent at once, once every ~15 seconds.

[cnlohr]

Hey! Sorry, I was in Virginia. I'm back now. Oh boy, I really don't remember how I coded any of that.

It looks like I only support one letter/button at a time. see user_main.c:100. It looks like it passes the scancodes, and the javascript creates the scancodes.

The idea was I just always cycle messages, if there is a message that needs to be sent and responded to, it handles that (if it's on the queue) if nothing is happening, it sends wx. That's how the Hz counter is updated.

[leopck]

Hi guys! Sorry for coming back to you guys so late. My laptop's HDD was broken and I lost all my written programs haiz.... My bad habit of not keeping backups. I'll have to take a few more days before coming back to this project, (restore Linux and re-write what I was previously writing...) keep me posted :)

[cnlohr]

Bummer... Github has definitely changed me as far as keeping everything committed, all the time. Allow it to course through your veins.

[SinanAkkoyun]

Hello guys, I am sorry to message you all, but what happened to the HTTP request approach?

[cnlohr]

basically nada

[SinanAkkoyun]

Thanks for the awnser! Well then I have to connect to about 50 websockets in parallel on an ESP32, is thaz possible? :)

[timg11]

Hi all - I built this up because I need to cause a USB keyboard keypress from a script running on another computer. I've compiled, built, and flashed the code. I fixed the inability to save the Wi-Fi setup by compiling the station mode into the source with Wi-Fi credentials. I am able to send keypresses to the host system from a browser using the webgui. Now I just need to be able to script it. I tried the curl http://192.168.4.1/d/issue?ehello, which works, but it echoes hello to the computer running curl, not as keyboard input to the target computer connected by USB. I looked at http_custom.c, but I'm not seeing any methods that would cause keys or scancodes. Can anyone point me in the right direction?

[timg11]

With further examination, I found CustomCommand in custom_commands.c, which has case 'K': case 'k': //Keyboard.

I tried curl http://192.168.27.214/d/issue?CKz expecting a "z" character to be sent over USB, but nothing was sent. I thought maybe it was expecting scan codes after CK, so I tried CK2DAD for "X" make and break. I tried CK0x2D0xAD, which didn't work either. What is the expected format for the custom Keyboard command in a URL?

[idesignstuff]

Never used this, but often commands need an input variable in (parentheses) after the command. Sometimes, it may also need ("quotes in the parentheses")

Christian

On Sat, Feb 12, 2022, 5:17 PM Tim Godfrey @.***> wrote:

With further examination, I found CustomCommand in custom_commands.c, which has case 'K': case 'k': //Keyboard.

I tried curl http://192.168.27.214/d/issue?CKz expecting a "z" character to be sent over USB, but nothing was sent. I thought maybe it was expecting scan codes after CK, so I tried CK2DAD for "X" make and break. I tried CK0x2D0xAD, which didn't work either. What is the expected format for the custom Keyboard command in a URL?

— Reply to this email directly, view it on GitHub https://github.com/cnlohr/espusb/issues/23#issuecomment-1037564862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAGGGISFAUA5ZNTQABPKHLU23S67ANCNFSM4CYNWHIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

[timg11]

@idesignstuff , thanks for the comment. I've been searching the source, and I think the protocol is fairly simple, but I'm not able to make it work by sending commands with CURL. In main.js (which contains the browser code), I find this code:

function Keypress(id, type, shift) {
    console.log("188 Keypress ID (HID): " + id + " (" + id.toString(16) +  ")")
    var mods = KeyboardModifiers;
    if (shift) {
        mods |= 2;
    } else {
        mods &=~2;
    }

    if (type == 1) keyops.push("CK" + mods + "\t" + id); // Need a sanity check;
    if (type == 0) keyops.push("CK" + KeyboardModifiers + "\t0");
}
function KeypressMulti(str) {
    [].forEach.call(str, function(char) {
        //console.log(char);
        var modifier = KeyboardModifiers;
        if (allUpper.indexOf(char) >= 0) modifier |= 2;
        let id = UsbLookup(char.charCodeAt(0));
        if (id === undefined) return;
        $("#typed").text(($("#typed").text() + char).slice(-50));
        $("#codes").text(($("#codes").text() + " " + ("0"+id.toString(16)).slice(-2)).slice(-50))
        keyops.push("CK" + modifier + "\t" + id);
        keyops.push("CK" + KeyboardModifiers + "\t0");
    });
}

It seems to be building a string starting with CK, as I tried above, but then adds modifiers and codes that appear to be binary values. I'm unfamiliar with keyboard operation at this low level, so it is unclear what is needed. UsbLookup seems to map to a code, so the character "a" maps to 0x04. With no modifiers, then the final string would be CK\x0\t\x4 to send the key "a".

I looked at the browser console log, and saw this logged when I sent "abc" using the web gui:

148 Keypress Code (JS): 65 (41) main.js:249:11
188 Keypress ID (HID): 4 (4) main.js:290:10
148 Keypress Code (JS): 65 (41) main.js:249:11
188 Keypress ID (HID): 4 (4) main.js:290:10
148 Keypress Code (JS): 66 (42) main.js:249:11
188 Keypress ID (HID): 5 (5) main.js:290:10
148 Keypress Code (JS): 66 (42) main.js:249:11
188 Keypress ID (HID): 5 (5) main.js:290:10
148 Keypress Code (JS): 67 (43) main.js:249:11
188 Keypress ID (HID): 6 (6) main.js:290:10
148 Keypress Code (JS): 67 (43) main.js:249:11
188 Keypress ID (HID): 6 (6) main.js:290:10

The log seems to confirm the HID code for "a" is 0x04.

I'm studying how to send such a string as part of the URL in CURL. I tried the standard URL encoding of special characters as %xx where xx is the hex value:

curl http://192.168.4.1/d/issue?CK%00%09%04

That did not work - nothing was sent as a USB keypress.

Just in case the "\t" was taken literally and not interpreted as 0x09, I tried this form too:

curl http://192.168.4.1/d/issue?CK%00%5Ct%04

It did not work either.

[timg11]

Seeing the comment from @cnlohr above: "Needless to say, it would be more awkward to use the http interface if the websockets interface is already established." I made sure the tab with the web gui was closed, and verified no IP traffic to the espusb device with Wireshark. Then I tried the various CURL commands again. No response.

Then I thought the implementation on the ESP8266 might not be able to decode URLs encoded with the % notation. I created a quick python script to send UDP to port 80 (the only one open on the espusb) as a raw binary message.

import socket
UDP_IP = "192.168.4.1"
UDP_PORT = 80      
# message = b"CK\x00\x09\x04CK\x00\x09\x00"
message = b"CK\x00\x09\x04"
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.sendto(message, (UDP_IP, UDP_PORT))

I tried the long form message = b"CK\x00\x09\x04CK\x00\x09\x00" and the first key only form message = b"CK\x00\x09\x04" based on the way the keyops.push function in the webgui (above) seems to operate.

I verified the bytes were sent as expected in Wireshark:

0020 1b d6 cd a1 00 50 00 0d b8 af 43 4b 00 09 04 .....P....CK...

Unfortunately, these do not result in a key being generated by the espusb.

[timg11]

OK, my bad for jumping on an old project. Kudos to @cnlohr for sharing his expertise with the ESP8266 and the ESP SDK. It is knowledge I don't have. I will be able to achieve my goals faster using other approaches, compared to trying to come up to speed on this environment sufficiently to debug the code.

Bottom line - espusb works fine from a browser with the websockets API. The HTTP API described by cnlohr above on Dec 7, 2016 will echo back a string as shown in the post. However, I have been unable to use it to generate keys over the USB. It may indeed work, and I didn't find the right mode or syntax, or it may be broken.