Advanced declarative non-uniform bank paging

[lmartorella]

Here a viable implementation of "non-uniform" bank switching support via configuration. The implementation is still completely based on SimulatedMemory, but it offer the possibility of having non-uniform slot sizes. In real vintage hardware, in case the addressing is not covering the full 64kb space (e.g. old arcade multi-Z80 boards or custom industrial boards), different circuitry could be used to decode the address and enable different hardware.

So with this PR is possible to declare the memory layout using ranges, e.g.:

{
    // ...
    "customMemory": [
      // 0-3FFF: 16kB ROM
      { "range": [0, "3FFFh"], "rom": "rom.hex", "name": "ROM" },
      // 4000-6000: RAM: smaller slot of 8kb
      { "range": ["4000h", "5FFFh"], "name": "common" },
      // 6000-8000: Not populated
      // 8000-FFFF: paged, 16 banks of 32Kb
      { "range": ["8000h", "FFFFh"], "banked": {
          "count": 16,
          // I/O port 80 selects the active page, with the lower 4-bits
          "ioMmu": {
              "port": { "mask": "FFh", "match": "80h" },
              "dataBits": [0, 1, 2, 3]
          }
      } }
    ]
}

So in this case the memory layout is 16k/8k/8k/32k:

In addition, the PR contains the implementation of automatic bank switching logic listening to I/O out operation.

A new class CustomMemory does all the logic. In order to share the same settings with the paired CustomMemoryModel, a CustomMemorySettings class contains all the decoded information from the public settings.

In addition, I've added the support to Intel .hex file (for ROM content). I've found a handy npm dependency for that, but rather poorly tested. By the way, the file format is simple enough for the external code to be rewritten inside this project (if you don't like the proliferation of 3rd party dependencies).

I was mentioning the declarative bank switch support in maziac/DeZog#74, and I understand your concern about the real necessity of declarative syntax: in the 99% of cases, custom code will be required to complete the simulation.

However, I would like to see "natively" supported the most common patterns when declaring the hardware of a custom board. Having a sort of guided "auto-complete" data object is always preferable to write custom code: code is always prone to copy/paste error propagation (unless you offer a strong API that requires simple configuration objects), and less controllable in terms of correctness of operation / overall performances.

I'm working on a project around this custom extension board: https://github.com/lmartorella/olivetti-plu, and I found DeZog incredibly flexible as a reverse engineering tool / prototyping tool. I see a lot of potential in this project, and it is not so unlikely that it can become the universal IDE for all the 70s/80s/90s consoles and computers based on the Z80 CPU (and their variant too... now thinking to 8080, to CP/M, etc..), exactly as VS Code is growing fast to cover all languages and environments.

Thanks! L

NOTE: Other MMU implementation not based on I/O are obviously possible (e.g. via memory write, like in the ZX Interface 1 and in Sega Master System 1M cartridges), so this is only to cover the most common way to do bank switching. However it is quite easy to extend the configuration to support all the possible cases.

[maziac]

Wow, a comprehensive change you did. I need some more time to check it. I'll come back to you.

Did you do any investigation what and how many boards can be simulated with this approach. Actually I could imagine that a lot use the I/O bank switching, but that is just a guess.

Already a comment for the hex file: We can include the Intel hex type. Also the npm is fine as long as it is MIT license. But maybe it is better not to distinguish the file type via the file extension. In the "assembler" area there are often a lot of file types used and they often share same names. I have also seen that the Intel Hex Type already has plenty of file extension (.hex, .h86, .hxl, ...) So maybe better to use "loadObj" for raw data and maybe a new "loadIntelHex" for the hex file format.

[lmartorella]

Hi, time is not a problem. Working on emulating 40+ years old electronics often requires relaxed pace :)

I think that basically all "modern" systems with a plain Z80 on board that mounted more than 64Kb of memory uses bank switch commanded by CPU (I/O or men triggered). Perhaps there could be some exceptions in mainframes, in which multi-processes/multi-user could have be implemented with hardware timers for bank switch, but this is IMO virtually impossible to implement without properly saving the CPU state before the switch (hence using an interrupt anda shared rom/ram).

The more variability comes instead in the "fixed" allocation of slots. Actual hardware could have banks that starts with a BIOS rom and then switch to RAM (to allow boot from removable disk), banks that can be moved between slots (e.g. Spectrum +2/+3). However I feel that these could be treated as exceptions, or simply with a more expressing syntax. For this reason i kept the logic in a separate class without interfering with the underlying memory manager.

For the hex file you are right. However I believe that a "smarter" configuration should be required sometimes to avoid discourage people to try his own. There are other ways to do "auto sensing" (like try parsing the file), but you are right, let's keep it simple as of now.

Thanks! L

[maziac]

I have a few questions regarding the intel format.

• more of curiosity: is there really a use in the z80 area? What (z80 related) tools/assemblers create such a file type?
• the file type itself: I don't fully understand it. The wiki page describes there is a record type data (00) with start address and length. I.e. I assume that a hex file can have several of these records. On the other hand the intel-hex tools rpm package just returns a struct with one start address. What if the file contains several data records?

[maziac]

Hi,

I have now looked through the PR, not yet understanding everything, but I think I got the bigger picture. One thing first: I really like to read your code. Here and there I find new things for me to try in the future.

Now for the PR: I still don't know where this whole paging/banking will lead us to. Or, in other words, what is the 'ultimate' goal?

DeZog was primarily implemented as a source code debugger. The disassembly is only a fall back. The simulator (zsim) was implemented to overcome a few drawbacks of the other emulators and to have an internal simulator so that dezog can run without other dependencies. I used dezog a lot for debugging ZXNext (own) sources. There was a shortcoming: the paging/banking was in-transparent to dezog. The problems were:

• several source files got the same addresses, i.e. it was not possible to decide which file to show for a certain PC value.
• a breakpoint set in one bank would also lead to a break in other banks at the same address

For the ZXNext this was completely solved: I needed also the help of Ped in sjasmplus because it was required that the paging information is available for the labels. I.e. to know that a label not only has an address but also to know in which bank it resides. And also the zsim has the capability to work with these, so-called, long addresses that consist of 64k address + bank information.

(bank_nr+1)<<16 + address

If we now implement this flexible way of defining the memory/banking layout for zsim, it would mean we also need the other way: how to get from a current PC value + state information to a 'long address' or similar.

I think this is done in your CustomMemory constructor. (Please confirm). So it should be covered. BTW I'm really impressed how well you understood my code and extended it (I sometimes have a hard time myself to read it after a while).

But still missing are the source files, i.e. the long addresses together with the source files. I.e. some kind of assembler that outputs banking/paging info. I only know of sjasmplus. It supports ZXNext but I don't think it's flexible enough to allow any kind of banking.

So, we end up at object code debugging only (what was not dezog's goal in the first place). The custom memory change would be a zsim thing only. (Or do you see any assembler being involved here as well?)

And, if it is zsim only. Does it really help? I mean: this would support the memory management but if you want to simulate an older system don't you require more? E.g. display output etc. Or are you going to implement this via custom code?

Don't get me wrong. I'm open to add this PR. It's well coded, there are unit tests, I think it will fit smoothly in zsim and it would extend zsim capabilities.

I'm just not sure if it really helps or if this would also require more (programmatical) extensions to dezog anyway (e.g. display emulation) to make this useful.

[maziac]

Any comments?

[lmartorella]

• more of curiosity: is there really a use in the z80 area? What (z80 related) tools/assemblers create such a file type?

As far I know, the .hex file type was widespread in the area of CPU with linear addressing, and it become unpractical after introduction of segmentation/relocation (e.g. 8086). It is still commonly used for example in MCU area, Microchip microcontrollers still uses .hex format as "binary" format after emit from compiler/assembler and before the actual flashing. EPROM programmer of the age (like mine PGM8ME) uses .hex format for the chip content. For example here you can see the dumped ROM of the Olivetti board.

• the file type itself: I don't fully understand it. The wiki page describes there is a record type data (00) with start address and length. I.e. I assume that a hex file can have several of these records. On the other hand the intel-hex tools rpm package just returns a struct with one start address. What if the file contains several data records?

The hex file is "optimized" to be shown on screen, more or less like the Memory View panel. So each record is usually 32 bytes long. This means that the file contains several data records, each of them with different start address. When assembling a program to .hex, it is usual that the memory space is not contiguous, but it only contains the bytes that must be really written to the right code/data memory address. So you will not see the empty "FF" bytes of a regular ROM dump, for instance.

[lmartorella]

DeZog was primarily implemented as a source code debugger. The disassembly is only a fall back.

Thanks, this clarify the scope of your project.

However, I still see it as a potential universal debugger/simulator for any Z80-ish based system. When I firstly approached your project, it was very quick to setup it as a debugger for my "unknown" Z80 board that I would like to covert to a single-board computer.

Out there there are obviously plenty of "specialized" simulators for the most common machines of the 80s, most of them successful clones of older projects. However, I found the lack of a real "standard" IDE very frustrating and time consuming for an occasional player, especially when coping with non-standard boards. I believe that Visual Studio Code is really becoming the de-facto standard for any activity around development, independently from the programming language, framework or operating system. It also support online web usage, being written in html/js. Installing it is super fast, like installing extensions, so becoming operative starting from zero is really a child play!

So, going back to the why, I believe that the tool has really the potential to become the de-facto standard for complete Z80-system simulation, development and reverse engineering!

And also the zsim has the capability to work with these, so-called, long addresses that consist of 64k address + bank information.

Well, this only works when the memory can be imagined as a "flat" memory contiguous space. Let's for example use my Olivetti board: it has a ROM mounted in the lower 16Kb, then 8Kb of RAM and then a whole 512Kb banked memory mapped in the upper 32Kb addressing space.

It is not easy in that case to found an "universal" way to map any memory byte to a linear addressing system. One way could be to imagine that only the upper space uses extended addressing memory, but the lower space uses a sort of not-banked "zero" page.

(In the Olivetti board case, the code is actually loaded in the ROM, so not paged at all. The huge RAM amount was used to implement a fast DB for price look-ups). (But I have plans to have a better use of such RAM amount, implementing some sort of multi-process system, or hacking the board to use a slightly different memory schema to run CP/M 3.x on it).

In a more generic way, it is not always possible to linearize a "wild" non-uniform hardware-defined addressing space (and so in turn the program disassembly).

I think this is done in your CustomMemory constructor. (Please confirm).

Well, actually it is not my intention to linearize the addressing space. The idea was simply to reuse the current implementation in order to simulate a non-uniform banking schema. In simpler terms, to me the PC register will always be 16-bit wide. Trying to disassemble a generic program that spans a non-uniform banking scheme is simply not possible without giving more context to the simulator.

I sometimes have a hard time myself to read it after a while

Ahaha, it happens to everyone :) But Typescript helps a lot in defining structures.

The custom memory change would be a zsim thing only. (Or do you see any assembler being involved here as well?)

Well, my idea is to avoid interacting with other parties. DeZog is always including native support for everything, so it would be sufficient to me. In the future, it would even be possible to fine tune even not covered areas (like the precise timing) to close the gap with other simulators.

I mean: this would support the memory management but if you want to simulate an older system don't you require more? E.g. display output etc. Or are you going to implement this via custom code?

You got the point! If you think to the ZX ULA simulator you wrote in DeZog for simulating the screen memory, and you think to make it "pluggable", one can realize that DeZog can be used to modularize a system, exactly like a hardware designed would do in the golden era of 8-bit CPUs.

For example, in order to simulate a CP/M system, my idea was to plug a simple class to implement a console in/out based on some I/O port, and then write a generic reusable "console" UI screen (and implement there the logic for cursor ASCII codes, colors, etc..). Then at least one disk drives in needed (converting the more standard interfaces of the era).

With only that it would be possible to hardwire whatever CP/M systsem with DeZog!

Now, the missing resource is... to find the time :)

Thanks! L

[maziac]

When assembling a program to .hex, it is usual that the memory space is not contiguous, but it only contains the bytes that must be really written to the right code/data memory address.

So the 'StartLinearAddress' is more or less the first address and data.size the size overall bytes including the unused FFh.

[maziac]

"precise timing"

accuracy was also not my goal, at least not for the zsim. I leave that to the "real" emulators. "Precise timing" is something where I would spent a lot of effort in, but that does not really interest me. (As said, therefore are the the external emulators).

"DeZog can be used to modularize a system"

Actually this was the idea with zsim. Instead of simulating a special HW one could simply "attach" HW with the configuration. I'm e.g. thinking of adding a ZX81 screen output. Some Olivetti or other console output is also possible. (Maybe already possible with the custom code, I guess no-one tried yet)

"linear address space"

Why shouldn't it be possible to have a linear address space? It's simply putting all banks one after the other after 0xFFFF. If this would be addressed via the "long addresses" it's just a question how to permutate the bank number into the higher address number. Of course, more advanced, if the block sizes are not constant size, but still possible.

But without an assembler supporting it, it is rather useless.

The disassembler is not really capable of "long addresses". E.g. it can't display code from 2 different banks. Although it has knowledge of the bank when setting a breakpoint, it will always show only one bank. Also does the disassembler change it's contents sometimes. My idea was here that it only has to show small amount of code (e.g. if you call a ROM routine from your assembler code). I.e. it re-organizes what it displays after some time. In those cases the breakpoints vanish from the vscode UI (a DeZog error, maybe you have faced that already). But I think to make it more capable a complete re-design would be necessary. Not sure if I want to put effort in that direction. In the very beginnings my idea was also to add support for MAME, but I'm far from that at the moment. DeZog is not specialized on reverse engineering. This would require more stuff like that discussed in #83.

Anyhow, I will take a 2nd look at the PR and maybe just focus on the bank switching in zsim. Other stuff may (or may not) come later...

(BTW I just found that the disasm.asm file extension is wrong, issue #87)

[lmartorella]

Why shouldn't it be possible to have a linear address space?

Of course it is possible to make any banked set of memory linear. Your example of concatenation of bank bytes is an example.

However, as you already mentioned, this virtual addressing space is rather useless.

The Z80 doesn't have any MMU in hardware that allow addressing more than 16-bit of memory. Bank switching can be seen as a particular case of "overlaying", the more generic programming technique that allows to run programs larger than 64kb. In the more generic implementation, overlays can even be stored on slower mass storage like disks. When loaded, the overlay is copied in the RAM assigned portion and then the CPU will jump in. Memory banks is only a faster implementation, because overlays can be prepared at application load, and then simply switched.

However, the common issue is that a portion of the application must be always "resident", usually in the lower addressing space where the interrupt vectors are stored (e.g. can be ROM or even RAM after bootstrapping). Such resident page will contains the actual code to swap in/out overlays (e.g. accessing the disk or even simply do a OUT to select the bank with overlay), and typically the CPU stack memory.

Splitting an application in overlays is hardly a simple or automatic operation. Some high-level languages (like the Pascal/MT+) supports overlays at language level. The produced assembly code will contains such harness to switch overlays in/out from file.

However, the debugging support can hardly support addressing overlays in exact way.

Let's try to draft an example of a possible way to do in assembler.

    ; this code is running from common bank, SP in common bank
    CALL_OVERLAY my_function_in_ov1   ; macro that will activate the bank and do the call
    CALL_OVERLAY my_function_in_ov2

This code could produce (using I/O fast bank switch):

    ld c, bank_port    

    ld a, ov1_bank_id
    out (c), a
    call my_function_in_ov1_phis_address

    ld a, ov2_bank_id
    out (c), a
    call my_function_in_ov2_phis_address

Should be - for example - both my_function_in_ov1_phis_address and my_function_in_ov2_phis_address to be at 100h as physical address, I would expect a sort of "segment" prefix when looking at the address.

    ov1_bank_id: 1
    ov2_bank_id: 2
    my_function_in_ov1_phis_address: [1]:0100h
    my_function_in_ov2_phis_address: [2]:0100h

And this is the first pitfall of the linear addressing space: I would like to see in the lower 16-bits of such addressing, the same physical address (e.g 0100h) and not another numbers, otherwise when I look at the CPU registers I would always need a translator (and that is not possible, since the CPU doesn't support "segments" to translate addresses to memory space, like the 8086).

This means that the addressing virtualization should always know where the bank can be loaded in physical memory. Banking mechanism like the Spectrum +2/+3 that allows the same memory bank to be mapped at different addresses (I'm referring the ROMs 0/1 that can be mapped in the top addressing slot) can be hardly be used to do overlays, since any reference to absolute address there will fail.

The second pitfall is that the debugger could resolve the virtual address in the above case, since the CALL was correctly assembled from source code:

    ld a, ov1_bank_id
    out (c), a
    call my_function_in_ov1_phis_address    ; could show [1]:0100h, but this ONLY relies on the correct 
                                                ; execution of the before instructions

Then, once the overlay is loaded in memory, the debugger should know what banks are activated. As far I have seen, this is something that zsim is capable of, so resolving "overlaid" map files could be possible.

Debuggers like gdb seems capable of using overlays with symbolic information (even if gdb is designed to target 32-bit machines). The key point is to have a portion of memory in common area where the application should mirror back the state of each overlays.

Then, if the underlying "virtual" addressing space is linear or segmented, is only a matter of the underlying memory implementation and debugging symbol linking. These addresses should never surface the debugger UI in my opinion (registers, etc..), and neither the annotation in the debugging screen (because they are depending on the current banking state + PC value only).

I'm not sure what Z80 compilers are supporting overlays (and so, indirectly, banking) with debugging information, but now I'm intrigued to make a search in the CP/M era :) )

L

[maziac]

Hi, just to let you know: I have not forgotten your PR. It's just I'm currently working on "DeZog as reverse engineering tool" and would like to postpone the PR until I finished that. If you like you can also try it out. It's on the develop branch. At the moment it's kinda working (tag rev_eng_1). The process is described here: https://github.com/maziac/DeZog/blob/develop/documentation/ReverseEngineeringUsage.md

[lmartorella]

Will do, looks promising!

[maziac]

Quick question about your code. Can you confirm that it is all MIT licensed.

[maziac]

I'm still working on this. 2 questions:

• do you have an intel hex file for testing?
• can I use some of your code? I.e. do you put your PR under MIT license?

[lmartorella]

Hi, I've dumped a ROM in hex format in this project: https://github.com/lmartorella/olivetti-plu/blob/main/reng/PLU10.HEX (Since the ROM has the higher A13 pin wired to 1, the first half is never used, as described here: https://github.com/lmartorella/olivetti-plu/blob/main/README.md#memory-banks ) For this reason I introduced a offset flag in the memory setup: https://github.com/lmartorella/olivetti-plu/blob/prototype/.vscode/launch.json#L17

Yes, the entire my PR is under free/MIT license. Thanks! L

[maziac]

Thanks.

[maziac]

Hi,

please checkout the first 3.0 release here: https://github.com/maziac/DeZog/releases/tag/v3.0.0-rc1 I have not directly incorporated your branch but I put in a lot of your ideas and code as well. Actually the changes were bigger than first thought because I wanted to support also banking for reverse engineering. And the disassembler now also supports long addresses. Hope the result is fine with you.

I will close the PR here. We can discuss further under 'Discussion' of the release.

[lmartorella]

That's great! I'll definitively have a look, thanks!