WebIDE Integration - Githubissues

FWeinb commented 5 years ago

As discussed in Moddable-OpenSource/moddable#132 I am opening an issue here.

I just had a minute to test the integration of the new WebSocket-Debugger into the WebIDE. Looking well so far I started by just piping in the device log into the log area:

What would be great is a way to detect when the mod was uploaded and run. I am currently querying /mod/install but it returns as soon as the upload was completed.

Another problem is the mixed-content problem regarding https/http on resources hosted on the device itself. Chrome is blocking these requests and you need to manual allow loading the execution. This is problematic for uploading and connecting to the unsecured websocket connection.

FWeinb commented 5 years ago

@phoddie that looks great! I will look into it and try to update the WebIDE with it. Might be a good idea to make that manifest file editable from within the WebIDE too.

Really great to see your efforts in pushing these WASM tools.

FWeinb commented 5 years ago

@phoddie I think you will like this! Have some progress regarding the debugging story inside the WebIDE. I did not have time to implement the the exception handling but local/global/module frames are implemented as well as the list of call sites. I will need to integrate all of that a bit more with the file editor but it is a start.

As always, I made a little video showing it in action (click on the image).

Creating breakpoints from within the editor and sending them to the microcontroler will be my next objective.

phoddie commented 5 years ago

Superb! And it is fast too.

FWeinb commented 5 years ago

While implementing this I noticed ~two~ three things about the communication with the device.

Currently if we connect to it the device performs a debug stop and needs to get the <go/> command to allow to install a new mod.
Another thing was that explicitly closing the socket to the device from the client using socket.close() seams to destroy the underlying socket and I am not able to connect to the device again, need to perform a rest of the device.
when the device is restarted via the debugger or after uploading a new module the device should refuse any attempt to connect to it immediately. I try to reconnect to the device as fast es possible but currently have to inject a delay because I don’t want to connect to the device before it rebooted.

As you said. Everything did turn out pretty fast in the way the debugging connection is handling all the traffic while toggling the debug frame information. Good idea to eliminate the http server and only have the web socket connection!

phoddie commented 5 years ago

All clear points. I will look into those and get back to you.

phoddie commented 5 years ago

Currently if we connect to it the device performance a debug stop and needs to get the command to allow to install a new mod.

In my testing, this works. I tested with the xsbug.js script by waiting three seconds after connecting and sending a step, install, and restart:

setTimeout(function() {
    xsb.doStep();
    xsb.doInstall(0, helloWorldXSA);
    xsb.doRestart();
}, 3 * 1000);

If you still see the problem after updating, please let me know the steps to reproduce it.

Another thing was that explicitly closing the socket to the device from the client using socket.close() seams to destroy the underlying socket and I am not able to connect to the device again, need to perform a rest of the device.

The microcontroller was not handling the close request from the browser. The result was that the microcontrollers believed it was still connected and the browser was waiting for a reply to its close request. WebSockets close is now implemented. I was able to connect and disconnect several times in row without restarting.

when the device is restarted via the debugger or after uploading a new module the device should refuse any attempt to connect to it immediately. I try to reconnect to the device as fast es possible but currently have to inject a delay because I don’t want to connect to the device before it rebooted.

This one is complicated. I agree that a delay isn't a great approach. Let's try to eliminate that. After you send a restart, the WebSocket connection should cleanly close. If you wait until the web socket close is received, the device's web socket listener should be gone too. Terminating the listener on receiving the restart command is tough, because it is owned by the application script which is many layers away from the debugger bridge. I'll think about that some more.

FWeinb commented 5 years ago

Thanks for looking into this

Regarding that example:

setTimeout(function() {
    xsb.doStep();
    xsb.doInstall(0, helloWorldXSA);
    xsb.doRestart();
}, 3 * 1000);

As I am thinking about it my initial idea was to remove the need for doing a doStep there to conceptually split the concerns of the installation process from the debug process. I might not be explaining this good enough. Just from a conceptual standpoint it is strange to have to do a doStep before you can install a new module onto the device, by mixing two concerns debugging and installing.

The last command xsb.doRestart() is currently optional, should we make that explicit or should we keep the current behavior that doInstall will reboot the device automatically?

WebSockets close is now implemented.

Great! In the next update of the WebIDE I will close the connection to the device.

phoddie commented 5 years ago

Just from a conceptual standpoint it is strange to have to do a doStep before you can install a new module onto the device ...

The communication protocol does not require you to stop in the debugger in order to install. I gave that example as I understood your previous comment as indicating install did not work when stopped.

That said, I think you should. Depending on the host, you may be overwriting the executing byte code for the currently running mod. That is effectively corrupting the running code and will surely lead to problems. On the ESP8266 it is currently safe because the upload goes to a separate section prior to being installed, but on the ESP32 the upload goes directly to the location where the code executes. They are different because of architectural differences in the hardware.

The last command xsb.doRestart() is currently optional, should we make that explicit or should we keep the current behavior that doInstall will reboot the device automatically?

Install does not perform a restart. The install over http did restart, so perhaps you are recalling that. I changed it in the WebSocket version to give the IDE more control over the process (e.g. perhaps after install it wants to do something else, like update a preference, before restarting).

FWeinb commented 5 years ago

That said, I think you should. Depending on the host, you may be overwriting the executing byte code for the currently running mod.

Okay that makes sense. Let’s keep it as is.

Install does not perform a restart. The install over http did restart, so perhaps you are recalling that.

Yes I did mix that up. But isn’t this dangerous in regards to your previous point? After and install the device must be restart, continuing will lead to a crash on the esp32.

phoddie commented 5 years ago

But isn’t this dangerous in regards to your previous point? After and install the device must be restart, continuing will lead to a crash on the esp32.

My feeling is that a debugging interface is inherently dangerous because it needs broad access to do its job. There are lots of ways to crash or otherwise cause havoc. I'm not sure it is possible to make it safe in all case. I fear it could be a lot of work to try.

At the moment, my focus is making the debug inteface functional, reliable, and fast. That's already challenging. Any tool using the debug interface (e.g. WebIDE) needs to be aware of what is safe and implement against that. And we are still discovering what is safe. ;)

FWeinb commented 5 years ago

@phoddie Got the breakpoints from the WebIDE working. Click to see it in action:

phoddie commented 5 years ago

Very nice. It works just the way it should. ;)

FWeinb commented 5 years ago

After some testing today I ran into an issue that I can't solve. The Moddable Zero is always crashing after uploading the mod via the web browser. I tried to reflash runmod but I still run into these two crashes.

I tried to not run the mod on boot but still after trying to flash it via the WebIDE the ESP is crashing.

phoddie commented 5 years ago

The first stack trace is the installation of a new mod after rebooting (fxMapArchive). I've never seen a crash there. Either the mod data is a corrupt or it is a previously unseen bug in the install process.

The second trace is less obvious. It looks like it is trying to load and run the mod (fxResolveModule). That it crashes again might be because the mod data is corrupt, or something else entirely. Tough to say without more info.

If you haven't already, try resetting your flash and reinstalling. It shouldn't matter but...

esptool.py erase_flash

And just in case... I posted a fix a few days ago to a bug that corrupted mod installs. Please make sure you are running with that.

FWeinb commented 5 years ago

@phoddie Yes, there was indeed a corrupted mod stored on the device. Erasing the flash and re-flashing runmod fixed it. Still strange how this could have happened, I can't reproduce it now.

FWeinb commented 5 years ago

@phoddie Added breakpoints to the debugging tab to have an easy way to enable/disable them during runtime.

phoddie commented 5 years ago

Nice. First feature debugging feature not also in xsbug.

phoddie commented 5 years ago

@FWeinb - Both of us have occasionally encountered situations where a mod is corrupt. This seems to happen when the transfer is interrupted -- for example by unplugging the device in mid-transfer. The transfer is usually so fast that there isn't much opportunity to happen, although over Wi-Fi stalls do happen so it is more likely. Yesterday, I found a bug in the SPI read/write functions that caused occasional crashes while uploading a mod. I fixed that and will push in the next day or two.

That got me thinking about how to ensure an interrupted mod upload doesn't leave a partially installed (and consequently corrupt) mod in place. The solution I found can be implemented in the WebIDE without a change to the device code. It relies on writing the device header only at the end of the transfer, rather than at the start.

Upload first 4 bytes of mod. This triggers the erase of the full block of flash, which resets any existing signature to ~0.
Do not upload bytes 4 to 8 of the mod. This is the archive signature.
Upload the remaining bytes of the mod (from 8 to the end) as usual.
Go back and write the signature (bytes 4 to 8).
Restart.

FWeinb commented 5 years ago

@phoddie I will try to update the WebIDE as soon as I can.

FWeinb commented 5 years ago

@phoddie Update the WebIDE.

I guess I have circled in on my problem regarding malformed xml data. If you set a breakpoint (e.g. by adding one using the WebIDE) and re-upload the same mod, in 99% of all cases I will get malformed XML from the device. No idea if this is related just wanted to let you know.

phoddie commented 5 years ago

I guess I have circled in on my problem regarding malformed xml data. If you set a breakpoint (e.g. by adding one using the WebIDE) and re-upload the same mod, in 99% of all cases I will get malformed XML from the device. No idea if this is related just wanted to let you know.

A (more or less) reproducible case is great. I assume this is only over serial/USB, not WebSockets. Once you upload the mod, you then restart, I assume. What are the commands that get sent after that?

phoddie commented 5 years ago

I ran into a situation this weekend where during the mod upload I was also configuring the debugger (setting breakpoints, etc). That was too much over serial for the ESP8266 and bad things happened. I fixed it by making sure that during an install nothing else was happening over serial. Since after installing the next step is rebooting, that isn't a problem. Perhaps related to your serial corruption problem.

phoddie commented 5 years ago

@FWeinb - I just pushed some changes.

The patches are now removed from this project as they are integrated in the Moddable SDK. That seems to be working well.
There is a new command to set the time, timezone, and daylight savings on the device from the IDE. This is convenient as there is no battery backed real time clock. It works well on ESP8266; on ESP32 the time is set but the timezone and daylights savings are not.
There is a command to set the baud rate. It isn't current used, but does work. It may be useful in some circumstances.
There is a new command to load a module by name. This has some interesting uses. For now, it allows us to simplify the runmod host. Instead of setting a preference to tell runmod when to execute the mod, the IDE can request that itself when it wants. This gives the IDE more control of the timing and allows a consistent IDE implementation across USB and WebSockets (send the load module message in onLogin, for example). See doLoadModule in xsbug.js for details of the binary message format. It also means that the mod does not have to be named mod (though it cannot be main because that is taken by the host).

If you use the load module command, we can simplify runmod/main.js to this:

import Net from "net";
import MDNS from "mdns";
import {Server} from "websocket"
import Preference from "preference";

class ModDevServer extends Server {
    callback(message, value) {
        if (Server.handshake === message)
            ModDevServer.debug(this.detach());
        else if (Server.subprotocol === message)
            return "x-xsbug";
    }
    static debug(socket) @ "xs_debug";      // hand-off native socket to debugger
}
Object.freeze(ModDevServer.prototype);

export default function() {
    trace(`host ready on serial\n`);

    if (Net.get("IP")) {
        const hostName = Preference.get("config", "name") || "runmod";
        new ModDevServer({port: 8080});
        new MDNS({hostName}, function(message, value) {
            if ((1 === message) && value)
                trace(`host ready at ws://${hostName}.local:8080\n`);
        });
    }
}

phoddie / runmod

WebIDE Integration #1