usage question

[carterw]

I am new to micropython on the 8266, just beginning to experiment. One requirement I have is the ability to run a websocket client on the device (and I don't understand why it isn't everyone's requirement). For my purposes, I want it to interact with a Node.js websocket server.

There are several websocket server implementations for Node. One is based on socket.io and requires a special client, the others are more standard (I am not clear on the specific protocol details). I see wording in the Readme here that seems to imply that this one is the socket.io variety?

Also, will this implementation work with the vanilla most-recent firmware? Or do I have to build a special version that has some kind of logging?

[danni]

Hi,

So this is a work in progress. Expect there to be bees.

So the websockets module should talk to any websockets server. Although I've only tested it with two. The socketio module will talk the socket.io app layer on top of websockets (including the upgrading your session from HTTP to websockets, etc.). It's been tested against the Flask SocketIO server, but not the node.js one. It probably works, but I'm not 100% sure. Unlike the websockets protocol, there's no RFC for this.

As for firmware, you do require a custom firmware, including "freezing" these modules into it, as they're too big to compile at runtime. For example, like this: https://github.com/danni/micropython/tree/uwebsockets/esp8266/modules

Adafruit have a pretty useful Vagrant image for building firmware: https://learn.adafruit.com/building-and-running-micropython-on-the-esp8266/build-firmware

[carterw]

Thanks for the info. I know the runtime program memory on the 8266 will be very limited, and I am wondering how large a typical program can be. Is there an API function that returns available memory?

I have been using a JavaScript implementation for the 8266 with a fair amount of success, but support for it is evaporating; https://mongoose-iot.com/docs/#/javascript-api/

In that case I had to resort to building special firmware that contained my precompiled code similar to what you are describing, a hassle. Also I was finding that messages incoming to the websocket client could easily overflow the heap.

[danni]

I believe there is. Python will also return MemoryError if the malloc fails. It handles this to tell the server the packets are too large (NB if the server then tries fragmentation it will assert, I've not implemented any of the fragmentation support). I haven't tested it with a super chatty websocket server but it's processing the TCP buffer synchronously so it shouldn't run out of memory and cause the TCP stack to window.

As I said. There might be bugs, this particular yak is quite hairy (you might also benefit from trying my uselect module). But happy to help you get it working.

[danni]

As for code. While I've had to compile in this module. My app code, i.e. main.py has been able to just be uploaded. Also so far this module runs on the UNIX port with one patch (settimeout), which does make bug fixing protocol details easier.

[carterw]

I've finally had time to come back around to this. I got the firmware toolchain compiled and can build, compile, and flash the stock version micropython on a ESP8266. Now I am looking at how to add your code. While I have quite a bit of general programming experience I am new to this particular toolset and have only a brief exposure to Python.

I want to experiment with your vanilla websocket client (not socket.io at present). From what I can understand, I would add your client.py and protocol.py files to the micropython/esp8266/modules directory? Or, in your example I am seeing "import uwebsockets.client", so they should go into a uwebsockets subdirectory?

You mention baking in 'logging' as well, and I see the code "import logging". How to include the logging module so that it will be found and imported? Is the code for it here? https://pypi.python.org/pypi/micropython-logging/0.1.2

Would I simply copy the logging.py file into the modules directory?

[danni]

Yeah, add logging.py to the modules and this code to modules/uwebsockets directory to freeze them in.

[carterw]

Got it working this evening! Your websocket client connects appropriately to a websocket server on my PC written in JavaScript using Node.js! Here's the received data on the server side.

{ type: 'utf8', utf8Data: 'esp8266 2.0.0(5a875ba) v1.8.6-9-g30cfdc2 on 2016-11-14 ESP module with ESP8266' }

... and the client printed a response sent by the server. Very cool.

Websocket servers and clients are typically event-driven as you probably know. For example on my server side there is code like this; wss.on('connect', function(connection) { connection.on('message', function(dataString) { console.log(dataString); connection.sendUTF("This is the response!"); }); }); Can there be a way to implement this in your micropython version?

[danni]

Yeah, that's absolutely doable. I don't have time to do it right now, but I'd accept a PR. Have a look at the socketsio client module, which implements a main loop to consume messages, etc.

[carterw]

As I said, I have only brief exposure to Python, some code I wrote for an online class a couple of years ago. If you are uninterested in moving this forward that's ok but it does look like you have invested some time into it.

I see two directories, usocketio and uwebsockets. Are you talking about transport.py? Appears to be a semblance of event handlers in there. Why would you pursue the socket.io stack?

[danni]

Not that I'm uninterested, just currently time poor.

As for socket.io, I was using that because that's what I wanted to connect to. What I'm saying is that you can replicate the basis of an eventloop using what's in transport.py, you can just scrap the existing _handle_badger and replace it with your own event handler type things.

Does that make sense?

[carterw]

I understand, I have a day job too. Unfortunately my Python skills are very limited at this point, all I've ever done is a little text and file manipulation. I'd have to invest time learning the language a lot better in order to be able to implement the features.

[carterw]

I did some Python language study this weekend and am thinking that maybe I could add event handlers. But now I am concerned about the development cycle on this. You had mentioned that the websocket code won't fit into RAM and has to be baked in to the firmware. How are you debugging it? Does every iteration of the code need debug print statements, and then firmware rebuilt and reflashed?

[danni]

socket.io stack comes from needing to integrate with a socket.io server.

I am talking about transport.py. Yeah, you do need to rebuild to test it on the device. I do this by symlinking my code from my checkout into the micropython build tree, so it's always building the latest version of the code (sorry, it's annoying, welcome to embedded development). There's no debugger for uPython, so yeah, you're limited to print-debugging.

You can install micropython (UNIX port) locally for development. You will either need to patch your local uPython with support for setsockettimeout() (https://github.com/micropython/micropython/pull/2381) or patch your ESP8266 port with support for select (https://github.com/micropython/micropython/pull/2411) (better IMO) if you want to do anything other than receive websocket events (i.e. handle interrupts, do other processing, etc.)

[carterw]

Thanks for the tips, I'll try to install the unix port of micropython.

[carterw]

Apparently the unix port of micropython doesn't include the modules? Or I didn't build it correctly, at any rate I can't import the micropython-specific modules.

I've got a socket.io server running on my PC and I can connect to it from a browser client, and send messages back and forth. However the one you wrote doesn't appear to be able to connect, so I am back to the vanilla websocket client which does connect.

I'm trying to understand the event handling system you implemented in transport.py. I've tried copying several of the relevant methods over into the Websocket class of protocol.py, but I can't get anything to fire.

I can invoke; websocket.on('closed', closedHandler)

... and it appears to 'register' in the on() method, but I don't understand what the inner(func) is supposed to do there.

Also wondering how I turn on debug statements. What action causes debug to be true?

[danni]

You can install modules for the unix port using upip. What socket.io server are you using? There's probably a bug, as I had to reverse most of the socket.io protocol on the wire due to vagueness of the docs.

websocket.on is meant to be used as a decorator. Something like:

websocket.on('closed')
def on_closed(...):
    ...

I don't think I added a non-decorator form, if you need that it could be added.

[carterw]

I use node.js on the server side, the socket.io implementation is (somewhat obscurely) documented here; http://socket.io/docs/server-api/#

But no problem, I prefer the far less complicated ordinary websocket assuming the minimal event handling works. I did some looking around and found an implementation that I got working so far as to report close events, so others types should now be possible; https://stackoverflow.com/questions/1092531/event-system-in-python

I used the one with the comment "Here is a minimal design that should work fine", and it did work fine. :)

Can you please explain what action turns on the debug statements? Is it a compile-time flag?

Also, is it the case that websocket.recv() needs to be called in a loop?

[danni]

Try calling logging.basicConfig().

Yes websocket.recv should be called over and over. Like how it is for the socket.io case.

[carterw]

Any suggestions on what to do when the connection to the server gets dropped? There seems to be at least two scenarios. The server went down, or connection to it was broken somehow over the internet. This results in a error code of 104. If the wifi connection is lost the error code is 103.

Ideally you would want to try to reconnect and keep going. How to safely unwind the existing client code and reattempt on a timer?

[danni]

Mmmm, this is one of the things that seemed good in Socket.IO, you acknowledge messages so you can keep state of where the client is up to, if the user lost connectivity to the web etc. You could reimplement the same concept, add sequence numbers to messages and when you connect say "this is the last message I saw", where if you don't do that, it assumes you're new. Alternatively make your messages completely stateless or add key frame messages from time to time. Depends on the problem you're trying to solve.

[carterw]

I'm currently just trying to attempt reconnects to Wifi and/or the web socket server without crashing the 8266. A typical test case would be to shut down the websocket server, wait a few seconds, and bring it back up. Ideally the 8266 would reconnect and be able to continue to exchange messages. Or cycle Wifi down and up and be able to recover from that. It may be that just invoking a hard reset on the 8266 is the best solution, but I was trying to avoid it.

What I'm seeing at present is that after a successful reconnect it is crashing in the send code. I had refactored all of my websocket client code as a class, and currently was trying to delete the class instance, reinstantiate it, and periodically attempt a connect. But I'm not a Python expert, perhaps that isn't working and there are vestiges of the previous class instance still hanging around?

[carterw]

Getting a little farther on this.

I'm finding that memory is so constrained that I will get "maximum recursion depth exceeded" exceptions if I have more than a small number of chained function calls. For example if I receive a message from the server that requires a reply, I will get this exception before making it all the way through to the send code. The exception appears to be fatal? So what I do is queue the reply and return, and look for queued replies to send on a timer so that I have a smaller call chain.

https://docs.micropython.org/en/latest/esp8266/esp8266/quickref.html#timers

self.tim = machine.Timer(-1) self.tim.init(period=200, mode=machine.Timer.PERIODIC, callback=self.runTimerCallback)

Apparently I have to kill that timer on a disconnect or it will continue to run even if the containing class is deleted. I can now reconnect successfully when the server comes back up, but unfortunately I get the "depth exceeded" exception shortly thereafter in spite of what seems like adequate memory.

stack: 6864 out of 8192 GC: total: 36288, used: 23040, free: 13248 No. of 1-blocks: 358, 2-blocks: 44, max blk sz: 264, max free sz: 704 sendData exception maximum recursion depth exceeded

[danni]

It sounds like you might be calling into yourself instead of into your parent maybe?

[Hermann-SW]

Thanks for this great websockets implementation. According https://en.wikipedia.org/wiki/ESP8266 all ESP8266 have the same 80KB on RAM. I did never run into memory issues yet, and I did not bake uwebsockets into firmware. I used uwebsockets.client sofar only, by just importing it. Then I made a websocket connection to another ESP01s MicroPython WebREPL as websocket server and executed commands there remotely. It is all so surprisingly simple! https://forum.micropython.org/viewtopic.php?f=2&t=5351&p=30829#p30829

[carterw]

Just coming back around to this as I am seeing new work being reported. The memory problem I mentioned earlier was due to the fact that a number of methods were being called as a result of an incoming message, and then a response to the message was generated. For example; a request for the status of a pin comes in, gets sent to a handler, the pin state is examined, a response message is created, and that gets sent back to the server. This all happens in the same call stack, and the maximum function call depth in Micropython is ~9 from what I am told.

To work around this I had to put the return messages in a queue in order to let the call stack unwind, then send queued messages later on a timer thread.