DEVICE documentation - Githubissues

Hi,

first let me say that I really like to see the new ZXNEXT device. But I also have a few difficulties to understand how to use this new feature.

So this here is a request to enhance the sjasmplus documentation on the DEVICEs.

Could you please explain in more detail what the DEVICE directive (or real device emulation) especially for ZXSPECTRUMNEXT does.

I think that during compilation an address space bigger than 64k is used (virtual space), i.e. the 1.75MB of memory. I guess this would then also be true for the list file, i.e. the list file would then also include addresses bigger than 0xFFFF, e.g. 0x1AB05, but this is not explicitly documented somewhere (I think). What about ORG statements? Can I do "ORG 0x20000" or something. It would also be nice if the default mapping (memory to slots) would be more clearly specified. E.g. the default mapping for ZXSPECTRUM128 is not found under ZXSPECTRUM128 but under ZXSPECTRUMNEXT. The ZXSPECTRUMNEXT mapping is not explicitly given.

The ZXSPECTRUMNEXT mapping is not explicitly given.

Wait a second... In the Real device emulation mode there is Predefined devices list with:

ZXSPECTRUMNEXT ZX Spectrum Next, has 8 slots (0-7) of size 0x2000 and 224 RAM pages (0-223) totalling at 1.75MiB of memory. The default mapping is similar to ZX128, paging in: {14, 15, 10, 11, 4, 5, 0, 1} pages. The page 10 (bank 5) contains also ZX48 sysvars ("USR 0" mode) and clear ULA screen. (SAVESNA works only in ZX48 and ZX128 modes, in NEXT mode only SAVEBIN or saving specific TAP blocks work)

I basically didn't touch the ZX128 docs, now I see they are kinda too simple, so you have to check the DEVICE directive docs itself for the default mapping {7, 5, 2, 0}. (and the four slots are induced from ZX48 description) ... so this part sort of requires you to eplore the docs thoroughly, including the example code and both the chapter 4 section over devices and the directive help (actually exploring the accompanied tests/ sources may help a lot too, at least tests/devices).

That can be probably improved a bit, but all info is technically there, in the end, even for 128? How to use devices is kinda lacking lot more, and that's probably what is making it double difficult to grasp this info and understand it is actually complete, so I'm just verifying it is all there, but I'm not happy with form either.

If you can pinpoint which part you did completely miss, and which you did see, but didn't pay attention and if you have some ideas how to improve it, let me know...

About device memory (how to), this is a bit more complicated, the old docs have description of DEVICE, SLOT and PAGE, even mentioning directives which are not (never were) implemented, and these are quite sparse.

I did add new MMU ... reading it... hmm... pretty much expects you to pick up the mechanism from the old three above. :D

And you are expected to get the idea from that. :)

So how it works:

I guess this would then also be true for the list file, i.e. the list file would then also include addresses bigger than 0xFFFF, e.g. 0x1AB05, but this is not explicitly documented somewhere (I think). What about ORG statements? Can I do "ORG 0x20000" or something.

Unfortunately no, list file is from Z80 point of view, and that's also the DEVICE interface point of view, i.e. almost everything is 16b only (0000-FFFF addresses) (ORG too). To access particular page of the large RAM you have to first map it into the 16b Z80 address space.

I departed from this by the new directives (SAVEDEV, SAVENEX) and that departure is probably not described that well either, but those in some cases do operate on the simple linear 0..1.75MiB address space, ignoring current SLOT/PAGE settings of virtual device.

So the base sjasmplus is expecting you to do (for example in ZX128 device, to map page 6 into range 8000-BFFF): SLOT 2 : PAGE 6 : ORG 0x8000 and the following machine code will land into page 6 and labels will be in 0x8000+ form (truncated to 16b) (after reaching 0xC000 you will start to overwrite different page, which is mapped into SLOT 3).

The new MMU can do the same by MMU 2, 6 : ORG 0x8000, but as long as you specify only single slot in MMU, you can add extra option, to either error/warn you upon leaving the slot (emit of machine code byte at 0xC000), or to make it auto-warp back to 0x8000 with ++page (MMU 2 n, 6 will thus map pages 6 and later 7 into 0x8000-0xBFFF slot in ZX128 device). This can be used to INCBIN large files, or to generate large binary blocks by db/ds/....

And now the elephant in the room... What if you did generate large block like that (pic0: incbin "pic0.bin" / pic1: incbin ...), and now you want to know which label did fall into which page (as they are all 0x8000..0xBFFF range)?

And the answer to that is, you have to use $$ operator right there upon defining the label, like pic0page EQU $$ to grasp current page (and hope it will be correct, but I think I fixed it to a state when it will work as expected). There are no other tools to work with labels in the big-memory address space, except the obsolete LABELSLIST dump for Unreal emulator, which actually shows label page and page offset (but it's missing the Z80 16b offset, so to collect the complete info, you have to also check the listing file labels table). (I partially "fixed" LABELSLIST as debug info while fixing the labels internals, to see if the pages are set correctly (they were all zero in older versions of sjasmplus))

This is ongoing effort to improve this somehow, but it has two problems a) it's lot of work, as even the old system didn't work properly and had to be fixed first b) the old system is designed in this Z80 point of view way, which annoys me greatly, but actually some users of sjasmplus even insist on it (when I was discussing 16+b address space with them, i.e. they expect/want the assembler to behave sort of like emulator of ZX128 - don't ask me why somebody would want that). But it's deeply baked in, and I can't simply move to 16+b address space, as that would break all older sources.

So right now I'm considering how to proceed further, if you will check the TODO.TXT, you can find collection of my thoughts about this and some possible proposals, but I'm not sure about these yet, so any opinions/comments are welcome.

Thanks for the detailed explanation. I thought it would have an effect on the list files as well but as I understand now it doesn't. It instead has an impact on saving to bins etc. OK.

Yes, it's already usable even for real production, but it involves lot of manual setting up everything, very little feedback/validation from assembler, and some planning ahead. Seems like most of the authors I did consult this with actually want it like that and prefer this way, so I'm kinda stuck how to resolve it, I have some ideas in that TODO file, but I'm not sure which direction to follow, I will keep it in my head for some time, maybe even get back to writing some more Next real code like tests, so I get some own experience how things work.

Basically the two major paths are:

stay at v1.x sjasmplus, i.e. trying to be backward compatible as much as possible -> that means to keep most things as is, and add new operators to have more power to work with memory layout, like labels operators, and maybe having & operator to turn expression into large address, i.e. something like savedev "data.bin",0,&mydata,300000 will be possible then, but most of the code will be written in Z80 16b address space way. The 32 bit addresses would surface only here and there (maybe also supporting things like org &0x123456, although that opens another can of worms with how "for which slot to format label usage").
fork or v2.x version, intentionally making old sources/syntax invalid, and working straight in the 32b address space (there's still need for the new ways of how to specify slots when turning large address into actual 16b value in instruction).

I'm not yet firmly decided, because if you are familiar with the internals and understand the system well, it seems to me the option v1.x is still bearable (with extended operators and quality of life extensions, maybe allowing large addresses in list files and exports under some options, etc.).

Then again many times it feels like just to delete it all and write it from scratch as v2.0, as that would make many things lot more easier to maintain and modify, but the initial phase until something usable would emerge would take easily months, or maybe years, and I think I want rather to do even some own project.

As I understand it, you are working on the tooling mostly too, and that's a bit tricky, often it requires a "trial by fire" of some real medium/large project to see if the design of the tooling holds up. So if you know about somebody interested to run "serious" project in sjasmplus, it would make this process of design decision probably lot more easier, seeing the needs of real project.

At this moment I put my thoughts mostly against two things: 1) old sjasmplus sources and how to keep backward compatible as much as possible 2) what I will probably need in my dream project.

As with the include nesting, if you can describe what precisely you are looking after, maybe it will be possible to get it into the sjasmplus, but just changing to linear large address space is eluding me, can't see how to manage that one without actually breaking everything old.

Actually I'm working on the tooling and on the Z80 programming itself. I have currently a few projects running:

the z80-debug-adapter which takes the list file associates it with the addresses got from the (zesarux) emulator and associates the list file position with the "real" source file. So it's possible to show the cursor at the right position (PC) when stepping through the code.
a z80 unit test framework: that works in conjunction with the z80-debug-adapter to execute unit tests from with the IDE (vscode).
a ZX Next game framework: A set of libraries that allows to use sprites, music, sound, graphics etc. more easily.
and an actual game which is a port of an old game of mine, written completely in z80 assembler (and a little part written in C-code, but I wouldn't do that anymore.)

So the tooling and the Z80 development is complementing each other.

The current 16 bit address space is OK for me. But with the z80-debugger I have the problem that I can't map the right addresses if different pages are used. I.e. in this case the user has to choose the right list file already in the configuration (of the z80-debugger). But if a different memory is paged in you see the wrong file listing. With 32bit addresses (and the knowledge about the slot/page mapping) it would be possible to associate the right addresses with the right files without bigger problems. (Through zesarux it is possible to retrieve the current memory page.) So this would be a clear advantage.

But I also see the additional work also on my side. Since I'm supporting also 2 other assemblers it makes it more difficult if they behave differently. So this is a little bit similar to you: 32bit addresses would be my dream but then I have another bigger part to implement. (And I don't have the problem that I need to seek for work...)

So whenever you change to 32 bit I will probably follow, but I don't have to push you in that direction.

This is all fascinating. We generally reframed problems around the necessity of bank-switching before flat addressing. The QL got it right incompatibly, but the SAM ‘32K window safe-16385 byte copy’ has a lot to commend it if your data straddles all RAM (as SAM BASIC can). But write-through ROM (as on SAM and other late Z80s) is best for big bitmaps, can Next FPGA do that? Existing asm will cope fine 🧙🏼‍♂️

Maziac: well, for the moment you can maybe take a look at LABELSLIST which for devices ZX128 (and more) and NEXT does produce page:offset of labels, and then you can show the label per mapped page, and calculate it's actual 16b address by the position where it is mapped (in interactive tools). The page 0 will be unfortunately a bit polluted by equ/=/defl "value" labels, I have some ideas how to minimize this in future (to give them page above the device max like 254). (zx48 will produce 128-like pages, which is probably good for use with next, but zx128 has 16ki pages and next has 8ki pages, which makes supporting both a bit more tricky I guess) (and of course the final blow for sjasmplus case: you can use multiple device directives through source, but the labelslist is produced at the very end, with last device set active, i.e. if somebody would produce two (zx128 and next) binaries form same asm source, the exported labels will be mix of those with some of them having wrong paging - don't see any "fix" to this, one has simply write source in a way to not break the tooling)

I was checking also CSpect/snasm map files, can add support for them easily (in principle it's precisely the same thing as the "Unreal" dump above, just a bit different formatting and includes the expected 16b value after mapping) - if you have already support for these?

BTW, whatever path the further development will take, it's highly likely the labels list (in list file) will converge toward this kind of view, i.e. to display also memory page where it resides. Also thinking about changing listing addressing scheme, but that will require at least one new column (at least 7 chars), which makes listing a) incompatible, not the first time b) even more demanding on screen width

Simon: unfortunately I don't have any knowledge how those things work, so some practical examples would help.

Next can map any page into any slot - there are eight 8ki slots (start Z80 addresses: 0000, 2000, 4000, ..., C000, E000), and there are 96 or 224 pages available (0.75MiB or 1.75MiB, depends whether machine has extended RAM). Default map on HW is {255, 255, 10, 11, 4, 5, 0, 1} mimicking ZX128 {rom, 5, 2, 0} default mapping (8kiPageNum = 16kiBankNum*2), where page "255" is ROM. (the sjasmplus does map zx128:bank 7 (next:pages 14, 15) into ROM area by default, it has no "ROM" pages yet ... writing about it, that should be actually trivial to add, hmmm)

Plus on top of that, you can redirect writes into 0000-3FFF address range (ROM) to any two pages from the 0..31 range (first 512kiB memory chip on board which has 256kiB reserved for ROM/divMMC and second 256kiB are pages 0..31), through "Layer2 over ROM" extra feature. But this is like purely runtime feature, don't see any need/way to support it in assembler.

I'm still unsure where to focus to fix this bug (documentation part) - for me it seems quite complete, but maybe chaotic. Need some insight which parts are most problematic and most to difficult to find, or grasp.

About Maziac's tooling part - the new CSPECTMAP plus my new comments in #48 shows where this may go... I guess in the end the labels listing part (--lstlab option) will be changed more toward the CSpect MAP format, but with further extras about label definition/usage (which I'm afraid to add to CSpect MAP, to not break the emulator itself).

At this moment I think the "v1.x" path is most likely to happen, i.e.:

adding few more tools to work with large addresses in asm (new operators)
changing labels listing to include lot more internal information (mix of CSpect MAP and ideas in comments)
but generally staying in 16b address space (maybe adding extra info into listing hidden behind some switch, or changing the format one more time)

And I have no idea "when" ... but judging by the previous versions and my TODO, v1.14.0 or v1.15.0 seems likely (as I'm now generally quite OK with current sjasmplus version, not aware of any huge issues, IMO it's "production" quality now, ready to be used even for large projects) - which will hopefully happen still this year, if my real life will not divert me from sjasmplus completely.

Hi, originally this issue was about documentation. I think this was mainly my own problem understanding how it works, don't know how to update the spec for this.

But a question: I would like to use the CSPECTMAP or LABELSLIST option soon and add support to my vscode z80-debug extension. I have to decide which of the 2 options to support. Basically they contain the same information. Which one is more "future proof" or subject to change?

And a small comment to the --lstlab option: In general I find it more useful not to add this info to the end of the listing but to a different file. This makes life easier for people who want to automatically parse the files :-) (This is, of course, not a real issue as --lstlab is optional, but maybe it is useful input for any future changes...)

CSPECTMAP depends mostly on the #CSpect specs, i.e. it's as future proof as Mike decides.

LABELSLIST is unreal emulator specific, with Next device producing something with it, sort of accidentally... there will be probably no good reason to change this one intentionally (and the docs are already formalizing it a lot, although the Next slots are only 0x2000 in size, not 0x4000). So I guess I will update the docs to make the format more precisely defined and let it be like that indefinitely.

I'm toying around with the idea of introducing extra "ROM" (or you can think about it as "scratch" space too) pages in devices, like FF, 100, 101, maybe breaking the slot size limit to make Next 16k first-ROM labels produce FF:0000 .. FF:3FFF output, so you don't need to leak into "100" as long as you work on only 16kiB ROM image (or maybe even go for full 64kiB range for ROM/scratch pages, postponing need of "100" page as long as possible, as that will probably break anyone parsing it as two-hexa-digit field) - but this shouldn't affect you for regular sources, it will be added extra, used only by sources written after the feature being added.

The usage details and other labels extra info will probably land to listing labels table and EXPORT (in the form of EOL-comment with some strict syntax to make it machine parse-able).

So as temporary thing before we will pull out something smarter and prettier with EXPORT/listing table (who-knows-when), I would go by LABELSLIST, that one should stay as is for some time (it's mildly tested, I think it should work as described, but if not, I can probably fix it within the v1.x effort).

Oh, I did check source now, and the page number is actually decimal... and I'm not even sure if this is correct (never used the unreal). Does it support machines with more than 10 pages, like pentagon/etc?

So I'm not sure what is the proper format of LABELSLIST :/ ... if somebody will be able to define how Unreal works, that will become the definition for LABELSLIST too, meanwhile I will keep it as is, so for Next machine one should parse the page as decimal value and it may be also 3 digits long, not just two.

Meanwhile there was also $$<symbol> operator added to get labels' page easily and the comments in this thread are somewhat obsolete.

And I'm still not sure how to improve the DEVICE documentation, there's nothing obvious missing, and there's no obvious simplification possible, so it's about getting through it, at this moment, or just asking in issue for help with particular topic.

(also there are now public examples of sjasmplus in action, like the https://github.com/ped7g/SpecBong project, where the DEVICE usage can be seen)

So I'm closing this for now.

z00m128 / sjasmplus

DEVICE documentation #55