[FR] AI and Automation support for Machine Learning and 3rd Party Programs.

[Michaelangel007]

An idea I've had for a while now is a .dll plug-in where a 3rd party program is able to start/stop execution, can inspect memory, can send keypress/joystick input, save screenshots, etc. of the emulator.

This context are programs/bots that can generate auto-maps of games, can use Machine Learning to play games, etc.

See Seth Bling's video: MarI/O - Machine Learning for Video Games

Questions that need to be answered:

• How are other emulators doing this?
• What API does the .dll need?
• What features do ML programs needs?
• How is disk swapping to be managed?

[sicklittlemonkey]

In the video he mentions that it was written in Lua for the BizHawk emulator. The description has a link to the source code: https://pastebin.com/ZZmSNaHX

[hasseily]

Grid Cartographer is a map-making Windows software that allows you to make maps, but also to run something called "GameLink". Under GameLink mode, the screen is split in 2 panes. The left pane has the map and an icon for the player position, and the right pane has the screen output of a supported emulator. You can fully interact with the emulator in Grid Cartographer (GC): select the right pane, and every key/mouse input is sent to the emulator, which then sends back the framebuffer for display in GC. Furthermore, GC has the ability to read certain memory locations in the emulator in order to link the player position to the correct map on the left pane. So when you're moving around in the dungeon, the player's icon is synchronized to the map on the left (maps are automatically selected or created if new). This way you can either load a ready-made map and navigate with a modern auto-map, or manually draw the map as you move in the dungeon.

The dev of GC forked DOSBox and implemented GameLink on it. The way he does it is as follows:

• Hijacks the memory allocator of DOSBox and sends it to the new GameLink code that is compiled within DOSBox (something that ideally should be done via a DLL and not static compile)
• The code creates a shared memory allocation that includes:

1. The original required allocation
2. The input struct to send to the emulator (mouse x/y, btn, keyboard)
3. The memory peek struct that is used for linking the player position
4. The audio volumes for L & R
5. The framebuffer struct with width, height, aspect ratio, format and buffer

There's an In() function that receives input and audio, and an Out() that sends all the rest The emulator is responsible for handling the input and audio, and when the emulator finishes going through the runloop iteration, it fills the framebuffer and sends it along with a a few other pointers within the shm.

Then GC grabs the framebuffer and displays it, and can access the whole memory of the emulated program (since it's shm) to grab the player position, map id, etc...

It's basic and rough but it works well. All you really need is to know the name of the SHM and fill the framebuffer and handle the inputs on the other end.

[hasseily]

If I understand correctly, the main user input event handling for AppleWin is via the CALLBACK FrameWndProc(window, message, wparam, lparam). Let's say I want to send a pause message to AppleWin via a shared memory structure like the one I described above. The pause code inside FrameWndProc is simple but I don't want to replicate it.

Is there a way to manually call FrameWndProc to emulate a keypress? That would go a very long way to automate AppleWin's behavior: we create a shared mem structure accessible from outside programs that send in commands, and AppleWin parses this and sends the keypresses to the callback proc, emulating the user. I guess I could directly call FrameWndProc() but is there a better way?

[sicklittlemonkey]

There would be a little overhead - checking a synchronization object to manage the shared memory. Possibly after every instruction in debug mode, say.

The downside to this over sockets or pipes as a debug API would be that the client couldn't be remote. It would be faster though, assuming memory or screen dumps are desired.

[hasseily]

There would be a little overhead - checking a synchronization object to manage the shared memory. Possibly after every instruction in debug mode, say.

No, the Main (and possibly Aux) memory that will be exposed as shared should be read-only from other apps. It's only to analyze the memory of the running app, and then you send back the commands to Applewin via the input struct.

[Michaelangel007]

@sicklittlemonkey Thanks for the link to the Lua source code!

@hasseily There needs to be an API for writing to memory as well. While the other apps can directly read memory there are use cases where they will want to:

• Set RAM values
• Set IO values -- example, feed joystick button presses

This needs to be done through the memory sub-system so as to keep the emulator in the correct state for peripherals, etc.

[tomcw]

Semi-random thought: you could add a GDB (or lldb) server to AppleWin.

[sicklittlemonkey]

@hasseily https://github.com/hasseily is correct that sending keypresses might be enough for some games. Windows messages could be sent to AppleWin's window to do this I suppose.

[hasseily]

I forked the codebase and am working on an implementation that highlights this potential. I'll revert back when I've got the whole in/out working. Right now I have applewin exposing its main+aux ram to 3rd party programs, as well as the complete video stream. I'm working on the input process right now.

[sicklittlemonkey]

Cool! It looks like at least Michael and I are interested in this topic, but AppleWin thrives on contributions from people who "scratch their own itch"!

[tomcw]

My interest is from a auto testing AppleWin POV. Eg. from RESET, set a breakpoint (BP), then boot the Apple II. On hitting the BP (or a timeout), then compare the video memory with a "golden image".

This is why I suggested gdb/lldb above.

[Michaelangel007]

Looks like we have a bug in the initialization order for the debugger.

For example, this doesn't work. (Can't save memory, won't break when ProDOS is loaded.)

File: DebuggerAutoRun.txt

bsave "debug.0000.bin",0:FF
bpx 2000

Something like this does work:

echo "Hello Debugger!"

WinMain() calls RepeatInitialization() which has this order:

DebugInitialize();
MemInitialize();

We need to initialize the debugger after memory and video. Probably safe to move it to the end of the function?

[Michaelangel007]

@tomcw A GDB interface would be pretty cool! Would be awesome to be able to use a modern IDE to real-time debug 6502 code ;-)

I didn't realize GDB has a built-in UI! gdb -tui

Do you have any more details on how we can interface with gdb?

Handy refence for gdb <-> lldb commands:

• https://lldb.llvm.org/use/map.html

[tomcw]

We need to initialize the debugger after memory and video. Probably safe to move it to the end of the function?

I've spun this out to #855.

[tomcw]

Do you have any more details on how we can interface with gdb

No, I've never looked. It'd be interesting to do some investigation, but currently it's not near the top of my priority list.

[sicklittlemonkey]

It would be cool, and could be used in conjunction with or separate from the shared memory.

@Michaelangel007 You've never had to debug from the command line? Luxury! ; - )

[hasseily]

I've looked at triggering

@hasseily https://github.com/hasseily is correct that sending keypresses might be enough for some games. Windows messages could be sent to AppleWin's window to do this I suppose.

I've looked into doing that. In theory the proper way would be with SendInput(), correct? But it doesn't work because AppleWin still supports WinNT 4. I need to revert to using keybd_event(). But is there a better way and just call FrameWndProc directly?

[Michaelangel007]

We need to initialize the debugger after memory and video. Probably safe to move it to the end of the function?

I've spun this out to #855.

Thanks Tom!

[Michaelangel007]

It would be cool, and could be used in conjunction with or separate from the shared memory.

@Michaelangel007 You've never had to debug from the command line? Luxury! ; - )

LOL! I've been spoiled with a 2nd monochrome monitor debugging with Borland C++ back in 1990s. ;-)

Seriously, though, I've tried gdb over the years. It is painful. MSV's Watch window solves 95% of my bugs.

[hasseily]

I've got Applewin completely controlled by Grid Cartographer. I've implemented a very light framework where any type of remote control can be hooked into Applewin. Currently I only have Grid Cartographer's GameLink running, but it's relatively trivial to add any new one. I handle everything with shared memory, and there's passing of keystrokes, video, audio. There's also a special out-of-band channel.

https://github.com/hasseily/AppleWin/tree/v1.29.14.0-rc

[sicklittlemonkey]

This sounds cool, but I don't see a free version of GC. Is it only commercial?

[hasseily]

Unfortunately GC is commercial. However I've set up the code base using a RemoteControlManager() that abstracts the control apis and any number of such apis could be used. Also nothing stops anyone from using the existing Gamelink system from GC, it's really just a SHM structure.

I needed something exiating to make it work and didn't know of another program that did something like this.

In fact I'm thinking of building a small skeleton open source program that will link to Applewin and expand its visual space. Think of a plane's MFD or secondary display. Like in a RPG you could always display all the stats and inventory from all the characters while you play the game on the main Applewin screen.

[hasseily]

Pull request:

https://github.com/AppleWin/AppleWin/pull/860

Here you can see AppleWin running, and it's also controlled from a 3rd party app that allows full play and autoloads relevant maps with proper player positioning.