WebIDE Integration

[FWeinb]

As discussed in Moddable-OpenSource/moddable#132 I am opening an issue here.

I just had a minute to test the integration of the new WebSocket-Debugger into the WebIDE. Looking well so far I started by just piping in the device log into the log area:

What would be great is a way to detect when the mod was uploaded and run. I am currently querying /mod/install but it returns as soon as the upload was completed.

Another problem is the mixed-content problem regarding https/http on resources hosted on the device itself. Chrome is blocking these requests and you need to manual allow loading the execution. This is problematic for uploading and connecting to the unsecured websocket connection.

[phoddie]

@FWeinb - Great start. I'm glad it worked for you too.

What would be great is a way to detect when the mod was uploaded and run.

Yes, I expected this would be something we need to work on. It it is a little tricky. Let me explain. The upload of the mod is separate from installation of the mod. The install can only happen on a clean boot of a virtual machine (because the mod's symbols need to be remapped). That's why the virtual machine always restarts after installing a mod. Restarting reboot to the device, which drops all network connections as Wi-Fi is restarted too. The mod is then run after restart. In theory, we could avoid dropping the network connection by not restarting the microcontroller and only restarting the virtual machine. But, that is complicated and unlikely to be stable (all running services need to be cleanly exited, all memory freed), so I don't particularly want to explore that route.

Instead, I propose we rearrange the flow. Currently runmod executes the mod at start-up. That is not necessary. It is just one useful behavior. Another useful behavior could be to wait for a debugger connection. Once that is established, then run the mod. That way, the debugger connection is in place from the start of the mod's excecution. The choice between these behaviors could be made by another HTTP request from the WebIDE.

What do you think?

Another problem is the mixed-content problem regarding https/http on resources hosted on the device itself. Chrome is blocking these requests and you need to manual allow loading the execution. This is problematic for uploading and connecting to the unsecured websocket connection

I'm not sure how to solve this. Chrome's move to treating http like a pariah on the web has very unfortunate consequences for IoT. We are bumping into one here. As I understand it, we cannot use public certificates here because ".local" addresses are not owned by a single entity. We could use a private certificate (with a great deal of work on the device side...) but that requires the user to install the private certificate into the browser (or machine running the browser) which works against the goal of having the WebIDE "just work" with arbitrary devices. Ideas?

[FWeinb]

@phoddie Modifying the startup behaviour of runmod based on a HTTP request seams like the best way to go forward.

The mixed-content policy is very hard to work around. It is very strange that there isn't a way to ask the user for permission to establish an unsecured connection. For now it would be possible to host the WebIDE on plain http but that is currently not supported with netlify and other services. The Web is moving to https only and IoT devices will need to catch up with that but there is no good way to generate a certificate that will be trusted.

[phoddie]

I'll make the changes to runmod so we try that out.

I am sympathetic to the challenges of browser security policies. I'm optimistic that we'll figure something out. That said, web security is security for the web, not the internet. While I am glad the browser vendors are working to improve the security of the web, IoT should not be obliged to support every decision made for the web. That would lead to a significant increase in complexity and cost for IoT.

[FWeinb]

@phoddie I had a litte bit of time and implemented a basic instruments view:

To implement the stepping debugger I need to refactor the WebIDE a lot. The file handling isn't optimal and I need to think about how to switch between the debugging and editor view.

I don't have that much time to work on it though.

@lprader Are you still evaluating the WebIDE for the use in a workshop? Would be great to get some input on features you want to see included.

[phoddie]

@FWeinb - Nice! Instrumentation is one of my favorite features. It is essential for embedded development. Happy to see it arrive so early here. ;)

I've committed changes to support running the mod when the debugger connects, as we discussed above. Let me know if it works for you. The docs are up-to-date with the details.

Regarding the workshop, our Mozilla friends invited us to help out with two workshops they are holding at JSConf EU. We would introduce the Moddable SDK as it relates to the Mozilla Web of Things initiative. That is in early June. It makes an interesting (if ambitious) goal, and perhaps in your part of the world?

[lprader]

@FWeinb yes, definitely still interested in the potential for this in workshops! The most time consuming part of every workshop is getting all the necessary drivers and libraries installed and Moddable SDK tools built on everyone's laptop, so the WebIDE is really great since it cuts that part out of the process. The ability to load gists is perfect for that setting too.

There are a few features I can think of off the top of my head that would be useful:

• Ability to add files manually
• Ability to open multiple gists
• Ability to specify which files to upload

Like Peter said, there's JSConf EU in June. I'll also be running a workshop in late May, which would be an even more ambitious goal than that :)

[FWeinb]

@phoddie Great will have a look at it as soon as possible.

JS Conf EU sounds exciting, I am living in Germany so definitely in my part of the world but I am currently still a student (last semester) and do not know if I could find the time and money to go there.

@lprader Thanks for the feedback. I will need to refactor most of the WebIDE going forward. So having these requirements is really valuable. What would be the most valuable features of the debugger? Currently we have Instrumentation and logging, which is a good start. Is a stepping debugger required for the workshops? I am just asking to evaluate on what has the highest priority.

Building the stepping debugger is a huge task with a lot of ui design involved and would definitely take up the majority of my time.

Late May early June is definitely ambitious but I think if we cut back on the debugger and focus on the features you mentioned we could have a useable WebIDE for the workshops.

[FWeinb]

@phoddie
I tried to integrate your latest changes and run into a different problem. The debugger is connected so fast that the ESP did not restart yet. I solved it by not relying on the Timer to restart but rather block the HTTP request to mod/install until before the ESP is restarted.

main.js

@@ -97,20 +97,20 @@ class ModServer extends Server {
            case 6:                                                 // request body received
                if (undefined !== this.position) {
                    trace("installed mod\n");
-                   this.server.restart();
                }
                break;
-
            case 8:
                return {headers: ['Access-Control-Allow-Origin', '*', 'Access-Control-Allow-Methods', 'GET, PUT'],  body: "done\n"};
+           case 10:
+               if (undefined !== this.position) {
+                   this.server.restart();
+               }
+               break;
        }
    }
    restart() {
-       Timer.set(() => {
-           trace("restarting\n");
-           this.close();
-           restart();
-       }, 5000);
+       trace("restarting\n");
+       restart();
    }
 }

It is working but the ESP is crashing instead of restarting cleanly:

Exception (3):
epc1=0x40201af1 epc2=0x00000000 epc3=0x00000000 excvaddr=0x400391a8 depc=0x00000000
ctx: cont
sp: 3ffe98e0 end: 3ffe9b80 offset: 01a0
>>>stack>>>
3ffe9a80:  3fff7d24 00000000 00000000 40207d10
3ffe9a90:  3ffefd0c 0000125d 0000125d 40201be0
3ffe9aa0:  00000018 3ffea248 3fff90f4 40202058
3ffe9ab0:  3fff7f8c 3ffea248 3fff8b54 40107054
3ffe9ac0:  402817bc 3fff8b54 000014d0 40100f7e
3ffe9ad0:  40280d89 3fff934c 00000018 40100ff8
3ffe9ae0:  402491ee 3fff8e94 3fff8e9c 40247101
3ffe9af0:  3fff7f8c 00000004 3fff8e94 40249f42
3ffe9b00:  fffffffe 3fff8e9c 3fff8e94 402486dd
3ffe9b10:  3ffefd0c 00001231 00001231 3fff8bec
3ffe9b20:  00000040 00000060 3fff8e94 40248799
3ffe9b30:  00000000 3ffefd2c 3fff8bec 402046d4
3ffe9b40:  40201462 3ffefd98 3fff8f5c 3ffe9bac
3ffe9b50:  3fffdad0 00000000 3ffe9ba4 40252262
3ffe9b60:  3fffdad0 00000000 3ffe9ba4 40252048
3ffe9b70:  feefeffe feefeffe 3ffe8b60 40100120
<<<stack<<<

# EXCEPTION DESCRIPTION
# Exception 3 LoadStoreError: Processor internal physical address or data error during load or store

# EXCEPTION LOCATION
# 0x40201af1: umm_assimilate_up at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/umm_malloc/umm_malloc.c:1163

# CALLS
# 0x40207d10: fxCall at /Users/FWeinb/GitHub/moddable/xs/sources/xsAPI.c:1713
# 0x40201be0: _umm_free at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/umm_malloc/umm_malloc.c:1287
# 0x40202058: free at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/umm_malloc/umm_malloc.c:1733
# 0x40107054: vPortFree at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/heap.c:18
# 0x402491ee: xs_socket_destructor at /Users/fweinb/GitHub/moddable/modules/network/socket/lwip/modSocket.c:1102
# 0x40247101: xs_timer_callback at /Users/fweinb/GitHub/moddable/modules/base/timer/modTimer.c:76
# 0x40249f42: xs_listener_destructor at /Users/fweinb/GitHub/moddable/modules/network/socket/lwip/modSocket.c:1530 (discriminator 2)
# 0x402486dd: socketDownUseCount at /Users/fweinb/GitHub/moddable/modules/network/socket/lwip/modSocket.c:1102
# 0x40248799: socketClearPending at /Users/fweinb/GitHub/moddable/modules/network/socket/lwip/modSocket.c:1102
# 0x402046d4: modMessageService at /Users/FWeinb/GitHub/moddable/xs/platforms/esp/xsHost.c:1072
# 0x40201462: delay at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/core_esp8266_wiring.c:53
# 0x40252262: loop at /Users/FWeinb/GitHub/moddable/build/devices/esp/main.cpp:87
# 0x40252048: loop_wrapper() at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/core_esp8266_main.cpp:56
# 0x40100120: cont_norm at /Users/fweinb/esp/esp8266-2.3.0/cores/esp8266/cont.S:109

[phoddie]

Yikes. I'll take a look over the weekend. FWIW - usually some time is needed before restarting to allow the HTTP request to finish. Without that, the installer (IDE) cannot reliably receive the HTTP response code that the upload succeeded.

[FWeinb]

@phoddie Thanks. I investigated a little more and noticed that the main problem is that the XsbugServer isn't properly closed before restart which will never close the detached socket and the browser has no idea that the server closed the socket at all. Closing that socket would allow the browser to try to reconnect to the server.

[phoddie]

Thanks. I was thinking along the same lines on my drive to the office. I will try closing the listeners immediately but waiting to restart until the HTTP install connection is complete.

[FWeinb]

I did a rework of the ui and added the toolbar from vscode to separat the editing and debugging. I toke the moddable logo and did a little modification and integrated it, if that is something I should not have done I will remove it:

The debugging button in the toolbar will reflect the connection state by displaying a little coloured dot in the corner. Green means connected, blue connecting and red a connection error. The instruments data is still just dumped into the UI, I will create something similar to what is found in xbug:

The next step will be to update the file explorer to support basic file operations. That will take a little more time because I have to improvement the internal state structure for that.

[phoddie]

@FWeinb - Nice progress. The logo is fun, but we probably should not break the Moddable logo into component parts.

I have a bunch of changes nearly ready for you to try (I'll post again when they are live). I believe they give the behavior needed for the WebIDE -- the WebSocket receives a close message when restarting and no new incoming connections are accepted once an install, or any other operation that requires a restart, finishes. Further, the restart only waits 5 seconds when the network is slow. It usually is able to cleanly restart almost immediately (once any active HTTP requests are complete). In addition, the changes fix crashes on ESP8266 and ESP32 (very different lwip in the host platforms requires subtle changes in the Moddable client code...).

Still, a deploy & debugging flow is a little tricky over the network, all the more so to work on a microcontroller. I imagine we will refine it further in the future.

[FWeinb]

@phoddie I though so too regarding the logo, I will revert back to just having WebIDE in the top bar. Getting the deploy & debugging flow right is tricky but essential. It is awesome to see that you could build something usable this fast. Great to see this much progress in this short of time.

I am very excited to work on the WebIDE more.

[phoddie]

Changes are pushed:

• Update this repository
• Update Moddable SDK
• Reapply the xsPlatform.c/h patches

I hope the crashes have been eliminated, though they were always sporadic for me so it is difficult to be sure.

[FWeinb]

@phoddie Great work, got it working in the WebIDE.

Also got some improvements into the debugging, very crude UI but it is getting there:

On thing I have noticed it that if the debugging is stopped for to long the ESP8266 is crashing.

Also added some very basic file adding/removing:

These icons are only shown on hover, to not distract too much from the editor.

[phoddie]

Great to see the pieces working together. I'll look into the crash. In my test, I never stop long at a breakpoint. It may just be the watchdog timer.

[phoddie]

I can't seem to get a crash when stopped at a breakpoint. I modified xsbug.js to wait two minutes at a break and continue. That works:

xsb.onBreak = function(msg) {
//  this.doGo();
    console.log("BREAK!")
    setTimeout(() => {
        console.log(" Continue")
        this.doGo()
    }, 1000 * 120);
}

How can I reproduce the problem?

Note that while stopped in xsbug, the http server is blocked since that is hosted by the virtual machine being debugged. Consequently, while stopped at a breakpoint none of the commands to install, restart, etc, will respond.

[FWeinb]

@phoddie I just tested it again. It was not the long break but the unexpected closure of the WebSocket connection from the browser (when closing the WebIDE while connected to the device)

You can find a crash dump here

[phoddie]

Thanks. Indeed, closing the browser while at a break caused problems. Fixes forthcoming - will let you know when everything is in place to try.

[FWeinb]

@phoddie Added file drag-and-drop support to the file explorer

I don't now if this is necessary but folder support would be nice. A way to download the code in a zip with all the files to compile it locally would be nice to.

[phoddie]

The upload works. While it seems to build the uploaded file, I couldn't seem to get it to execute -- the changes I made to mod.js to invoke it didn't seem to get deployed. I'll have to experiment with that further.

[phoddie]

There are some changes to help with the debugging connection reliability you reported. You'll need to update the Moddable SDK and then apply the latest xsPlatform.c/h patches from the runmod repository.

• Disconnect while in the debug loop should no longer trigger the watchdog timer. Instead execution should continue.
• Incoming connection requests are ignored while stoped at a breakpoint (incoming data was already deferred)
• Disconnect while transmitting data should no longer trigger infinite loops (but I couldn't seem to trigger this condition)

[FWeinb]

@phoddie Pushed a quick fix for the not updated deployment. All files are saved before the compilation starts now.

I need to completely rewrite the file handling because it is currently very messy, that part was written as a first draft.

Will try your changes tomorrow (it's getting very late/early here) Was curious and did test it. It is working really great now! The connection will stay open until just before the device is restarted and you get all the logs in the WebIDE about the upload.

[phoddie]

Very happy to hear the debugger connection behavior is trending in the right direction. I committed a small change to the xsPlatform.c patch to fix a memory leak when the network debugging connection is dropped. In practice that shouldn't matter here, but it was a problem for a scenario where the debugger connects and disconnects multiple times without an intervening reboot.

I was able to drag & drop a file and have it work as expected. Edits redeployed too. Very nice.

I did run into one hiccup worth noting. The file I uploaded first was named main.js.That didn't work because there is already a main module in the host and it is preloaded. An installed mod cannot override a preloaded module, so that silently failed. Not perfect, but given the way JavaScript and XS are defined, I'm not sure how to allow it. The IDE could flag the conflict (e.g. compare the list of modules returned by the debugger to those modules it intends to deploy). It isn't a priority though. Just noting it here so no one else loses time tracking it down.

[FWeinb]

@phoddie Had some time to work on this a little more. I build a proper storage solution and have a complete file manager now.

The latest changes to xsPlatform.c are very unstable for me. I get multiple crashes and can't redeploy at all. I need to rest the ESP to get it to work again.

The compiler is working with the full file tree but I think that runmod can't handle accessing modules in sub path. That needs to be updated.

I also modified the gist loading to just import the files into the current workspace (warning before overwriting files).

[phoddie]

The latest changes to xsPlatform.c are very unstable for me

That's troubling. I assume you are referring to the changes of April 10. Those are relatively monitor. I recall you said the changes of April 9 were working well for you. I'll take a look.

The compiler is working with the full file tree but I think that runmod can't handle accessing modules in sub path. That needs to be updated.

I'm not sure I understand. If you would provide instructions for how to see the problem, I can take a look.

[FWeinb]

@phoddie Yes. The changes made on April 10 are breaking the deploy workflow.

I am currently importing „ping“ as if it where in the root directory but in the file tree you can see it is located in the „src“ directory. If you change it to „src/ping“ runmod is complaining that it can’t find the module.

[phoddie]

runmod does support paths such as src/ping. The build needs to be modified to put the correct path in the binary. I've updated the ping example and build instructions in the documentation to show how to do that. With the changes, the ping module is now loaded fromnetwork/ping instead of ping.

[phoddie]

Regarding instabilities, I don't see any crashes. I do see the WebIDE waiting. I was able to resolve that by being less aggressive about restarting the microcontroller to allow the TCP socket to close cleanly. Here's a quick hack to try to see if that also addresses the problems you observe. Change line 122 of main.js from

restart();

to

Timer.set(restart, 1000);

[FWeinb]

@phoddie Will look into it tomorrow. If i understand it correctly I have to mirror the structure of the build sources and then link them in that structure too.

I don’t know if the building could be simplified some more because currently everything his hardcoded inside ofxscl might be good to look into it some more because I just did a very naive copy-and-past there e.g. currently it is not possible to create a release build.

[phoddie]

If i understand it correctly I have to mirror the structure of the build sources and then link them in that structure too.

More or less, yes. Currently, the archive has the ping as the module path when you intend src/ping. Changes to the build are needed to resolve that.

[phoddie]

I don’t know if the building could be simplified some more because currently everything his hardcoded inside of xscl might be good to look into it some more because I just did a very naive copy-and-past there e.g. currently it is not possible to create a release build.

Yes, we do intend to revisit that. What you did is working well for the moment, so it seems reasonable to keep going for a while to see what issues come up. Then we can take those into account when we do the revision of xscl.

[FWeinb]

@phoddie I did test adding Timer.set(restart, 1000); but that did not help. The main problem seams to be that I can't deploy a new bundle while the debugger is connected. That worked before. I am calling socket.close() before trying to upload a new bundle, but the ESP isn't closing the connection to the web browser. It looks like the route http://runmod.local/mod/config/when/debug isn't responding after the first deploy. (Edit: Just some more testing. This only happens if I connect to the debugging websocket, reverting back to April 9 everything is working fine)

Regarding the crash, if the ESP is running for a little longer connected to the debugger this is the crash dump i am getting. This seams to happen if I initiate a second deploy that is stuck in limbo because the ESP isn't responding to the upload request.

So sorry. I am dense... I did not reapply the patch correctly... everything is working fine. Really smooth experience now.

To support nested paths we need to update xscl because it currently just compiles everything to /build without any configuration possible.

I even got the WebIDE to run on an iPad: (Icons are broken because I am using some unicode symbols as placeholders and these look different in every browser)

[phoddie]

Ha! Glad to hear that it is working. I will try the iPad (!) later.

Just a reminder that when stopped at a breakpoint, the entire JavaScript virtual machine is blocked. Any new incoming TCP connections are refused. Any data sent over an open connection is deferred. That makes things a little tricky for the IDE.

To support nested paths we need to update xscl because it currently just compiles everything to /build without any configuration possible

Yes, that makes sense. If you can do that, it would be the most expedient path forward. My team will do some work on tools in wasm to help bring all the features of xsc & xsl to wasm but that will not be in the immediately future.

[FWeinb]

I will implement the changes into xscl. Looking forward seeing improvements in the wasm build.

The last big hurdle is the mixed content policy especially in mobile safari and firefox because there is no way to allow these connections. The simplest solution would be to host the whole thing on http only; but that would then not allow us the get a deeper github integration because sending access tokens over http would be irresponsible. I have currently no good idea on a way forward on that front.

My next steps will be to implement the basic debugging functions to defining breakpoints in the editor and having stepping controls in the debug sidebar. After that I will improve the visualization of the debug data. I think a way to open multiple projects simultaneously would be important too, will look into that.

[phoddie]

The last big hurdle is the mixed content policy...

This does seem to be an ugly challenge. I was trying to think of an example of another project that has addressed a similar problem to use as a model.

[FWeinb]

The debugging tap got some visual improvements:

[FWeinb]

I added very basic (local) project creation. You can now store multiple projects in the WebIDE and switch between them.

Just saw the Mozilla IoT WebThings announcement. Is that what the workshop you are planning will be about? Saw a reference to the moddable WebThing documentation too.

[phoddie]

Cool. Yes, that works.

[FWeinb]

@phoddie I added support for binary files to the WebIDE and creating a very basic image viewer:

Just as a side note: I had to switch to a different storage solution to support binary files. All saved projects will be gone with this update.

The compiler currently just ignores these files because I do not know how I would go about uploading these files to the device. I just noticed that many of the examples provided in the moddable repository need some assets to be available on the device and implementing this wasn't that complicated.

Next I will definitely look into extending the debugging support to have a fully working debugging experience right inside the browser.

[phoddie]

@FWeinb - cool to see you added image support. When we dive into the wasm tools work, we plan to look at getting the other tools running. For images, they need to be run through png2bmp to convert to a microcontiolller friendly format. That opens up some interesting questions, because the format varies depending on the device and display. But we'll get to that...

[phoddie]

@FWeinb - I found some time to work on the protocol changes I've been thinking about. They are working on ESP8266,. I'm going to clean them up in the next day or two and post them for you to try. Once that works, bringing up the ESP32 is straightforward.

The new protocol completely eliminates the HTTP server, moving all communication to the WebSocket connection. That has a bunch of nice benefits:

• Less code and less RAM use on the microcontroller.
• Faster, because the TCP connect is eliminated from each request
• Commands (e.g. install, restart, etc.) can be sent to the microcontroller while stopped at a break point or even while the script is running in a very long loop.
• Ensures there is only a single development client at a time (HTTP requests can come from multiple machines simultaneously, which could quickly get unmanageable)

The elimination of the HTTP server, of course, breaks the WebIDE as now implemented. Routing the commands over the WebSocket connection is easier than sending the same commands over HTTP. The data is always sent in WebSocket binary messages, which is a simple way to segregate the command messages from xsbug traffic. For example, restart is:

xsbugWebSocketClient.send(Uint8Array.of(1));

[FWeinb]

@phoddie awesome work! Eager to try this. Moving all communication to the web socket makes total sense.

[phoddie]

Excellent. I'll try to have it committed before the weekend.

[phoddie]

Changes are committed. The docs have been updated as well. The patches changed, so be sure to reapply those. It is a lot of changes, so it wouldn't be surprising if there are problems. Let me know how it goes.

[phoddie]

(Now with ESP32 support)

[FWeinb]

@phoddie Got it working in the WebIDE. There was a minor issue, it seams that you are using an update version of the core websocket module. For me Server.handshake and Server.subprotocol where undefined. I changed these to the numeric values and everything started to work.

[phoddie]

@FWeinb That's great! I'm glad it wasn't too tough to make the changes. I will try it out a bit later.

Apologies for my oversight on the WebSocket constants. Those will be in our next Moddable SDK push.

FWIW -- we have some progress on the wasm tools work. That should be available soon.

[phoddie]

@FWeinb - Just a quick note here that we've pushed an update to the Moddable SDK with a substantial revision to the wasm tools support. I've opened a separate issue for discussion of that. It includes a link to documentation as well.