Ability to send ESP32 programs to run

[Serentty]

It seems to me like a lot of potential for graphics acceleration could be gained if it were possible to send programs over the serial link, which could run on the ESP32. This would allow much more direct manipulation of graphics in the VDP’s video memory.

Perhaps it could be implemented by adding some sort of mechanism for callbacks, where code could be run every frame.

[lennart-benschop]

This opens up really interesting possibilities, but you need to somehow define the plugin ABI for the downloaded code, so it can do useful things.

• To what resources of the VDP should the downloaded code have access? Should it be able to call functions in in the VDP? It should have access to the buffers where graphics data are stored and it should have access to the audio output.
• How will the downloaded code be invoked?
• How will the downloaded code communicate with the eZ80? This can possibly be done via buffer that the eZ80 can read and write via a VDU command.
• If the downloaded code ends up in dynamically allocated memory, how will it be relocatable or position independent? It will invoke a lot of design and prototyping before it is ready for release.

[Serentty]

Absolutely, some set of expectations would need to be set, or else unassuming code might simply take the ESP32 off in its own direction and the VDP code might either not run at all, or have its data structured stomped on.

My thinking was that the MVP would not have access to the VDP’s functionality directly—that could be added step-by-step when it proves useful. Rather, it would essentially just be a compute API. Later on, a specific API would have to be made for things like manipulating VDP graphics.

For the invocation, it could follow some standard function prototype where it is passed certain information. This could include things like a memory range which it is allowed to use. It would have to be a sort of cooperative multitasking system, where the code is allowed to save state in between invocations, and each invocation is a time slice where it is allowed to run.

For communication, some sort of VDU command like you suggested makes sense to me. Just a raw byte stream that gets ignored by the VDP and forwarded to the specific ESP32 program. A port number could be included to disambiguate, similar to a TCP packet.

Relocating code is definitely a bit tricky. I don’t know if something as fancy as an ELF loader is necessary.

[lennart-benschop]

Machine code has to live in flash or in IRAM, a specific area of RAM. I don't know how much of IRAM is already in use by FabGL. In any case we need a specific buffer allocator for machine code

[stevesims]

IMHO allowing arbitrary code to run directly on the VDP is not something we should look to add.

This would be a major security issue, and have the potential to allow errant code to brick people's machines.

I see two more practical alternatives.

The first would be to add the ability to store VDU byte streams on the VDP and allow them to effectively be called as functions. This would need just two new commands added to the VDP, one to "store function" and another to "call function".

This approach would obviously be rather limited, but would potentially still be useful. The obvious limitation is VDU commands are a simple byte stream with no flow control possible, and no variables. The lack of variables could potentially be addressed by including a command(s) to alter bytes within a stored function. (Overwrite byte, increment/decrement byte, etc.).

Another alternative would be to add support for a rudimentary programming language to the VDP. Forth would be a good candidate, as it's very light weight and should be fairly easy to implement.

Implementing Forth on the VDP would be an interesting companion to the z80-based Forth interpreter you produced @lennart-benschop 😁

[VOC-LINK]

Such Forth for ESP32 already exists: https://esp32.arduino-forth.com/

[stevesims]

the VDP now has a "buffered commands API". see #97 this implements all of the rough ideas I outlined in my earlier comment about repayable VDU byte streams, and extends that idea further.

those enhancements were written in such a way as to potentially allow for other language interpreters to be run on the VDP, such as Forth or Lua. the VDU system essentially has become a stream processor, so a language could use it as an I/O stream. before this work was done @astralaster had already experimented with integrating a Lua interpreter, and it's possible that work could be revisited at some point in the future.

the "buffered commands API" goes a long way to providing the functionality that @Serentty originally suggested, in terms of sending graphics programs over the serial link. what is missing from it right now from that request is the likes of callbacks and running a routine every frame.

I have some ideas about how to add that functionality, but it is likely that some other enhancements to the buffered commands API will come first. specifically in this area, and in line with other parts of this general discussion, ways to integrate the buffered commands API and the bitmap/sprite API are being considered and discussed.

for now, "running a routine every frame" can be achieved via sending a VDU command from the ez80 every frame to call a command buffer

[snakebyte69]

Using Forth and Lua is a good idea, the original goal is just to get the ESP32 to run code, the method is not that important. This should be done in a way were the number of languages could be expanded.