fasmg ez80 integration take 2 - support needed

[stevexyz]

I'm back on trying to let the fasmg compiled code to be source level debugged by DeZog!

Last time I started two routes:

r1. Convert the source listing to a supported one (here the old details)

r2. Complete new native integration, where I was not able to finish the changes since it took more than expected (it is my first .ts project) and then I changed computer loosing all development environment setup and didn't come back to finalize until now

Now, I would need some suggestion if to continue with route 1 or to continue with route 2.

For content:

r1. The current listing available from fasmg is in the ".lst" file. From it I was already extracting (with a php program) the ".sym" files in order to debug on no$zx and EightyOne, and for DeZog I created the conversion (with a python program) to the ".list" dezog compatible file

r2. I've rebased my old "draft" code changes on the new main line upstream (available to be seen in the fasmg-ez80 branch of my github fork)

Now:

r1. For the listing adaptation seems that the accepted structure has changed. In this case would be nice to have the details of the new format. And even nicer would be to have an "assembler neutral" (fully DeZog oriented) format in order everyone willing to integrate with DeZog would simply adapt this format (that taking into consideration the specifications in AddingNewAssemblers.md could be for example something like a series of lines "L|labelname|labelvalue" and "A|address|filename|linenumber")

r2. To continue on the native integration I will surely need some more support than r2, and in case this would be suggested would be nice to get some feedback on my current changes (settings.ts and labels.ts)

Thank you in advance for the suggestions!

[stevexyz]

An update: I might have found the specifications of the new source listing format at https://z00m128.github.io/sjasmplus/documentation.html#c_sld_data

Despite the worrying sentence "The file format is still under development. If you are working on support in debugger, stay in touch with us, so we can discuss any modifications to current format with you." I can try to adapt to it if ok

Another option cold be to duplication (and maybe simplyfication) of the current parser to create the before mentioned "assembler neutral" one so it can stay the same even if the assembler dedicated one will change but @maziac the decision

[ped7g]

@stevexyz hi, I'm co-maintainer of sjasmplus. The SLD format was initiated by Chris Kirby (haven't heard anything from him for longer time :/ ), and I added it to sjasmplus, then later it was adopted by Maziac in DeZog and somewhat extended to support DeZog needs. And DeZog is currently the only active project using SLD files AFAIK.

As long as you will stay around these projects and follow discussions in issues of dezog and sjasmplus, you will catch all the changes to the format and could provide your own input and directions.

If you have any questions about it, or comments, let me know, thank you. (currently any change to it will be driven by particular real-world use case, the format looks "good enough" to me, until some real use case proves it is not)

[maziac]

The main reason I used the sld format was the problem of "long" addresses. The normal address space of Z80 is, of course, only 16 bit. I.e. 0-0xFFFF. If a program uses banking 2 source files could share the same 64k addresses. In that case it would be impossible for DeZog to decide what source file to present for the current PC (program counter). In sld format the labels are extended with the banking/slot information. Internally DeZog creates these "long" addresses > 64k from the data in sld and works with it.

Now for your assembler, fasmg, the main question would be: does it's output also allows to include this banking information? If yes, it makes sense to take the sjasmplus parser as a starting point. If not, it is probably better to look at the z80asm list file parser.

I don't know fasmg, so maybe that is something you can tell better.

Here is a summary of questions, maybe you can answer:

• is there support for modules?
• does it support local labels?
• can macros be determined in the list file? If e.g. the macro is expanded without notice it will become harder to parse.
• is it possible to tell when an include file starts and what file name it has?
• can one determine how many bytes are assembled in a list file line? Some assemblers add e.g. "..." if the number of bytes gets too long. But this makes it harder to determine the size.
• does it support "long" addresses / banking information?

Parsing a list file can get quite complicated therefore I think you are always better off if you implement a parser rather than trying to convert your list file to one that already exists.

Maybe you can add here an example list file output of fasmg with macros, includes and all label types included. It's easier to discuss...

[stevexyz]

Let me start saying that fasmg is a "generic" assembler, and I use it for Z80 assembly with the ez80.inc of @jacobly0; for this reason I'm always referring to it with the syntax fasmg-ez80. In general it can assemble for many other processors, do other generic processing and even solve math equations, but all is outside the scope of this development.

Then regarding the questions:

1. Module (inlcudes) are the base of fasmg works
2. The fasmg label management is pretty advanced (with namespaces, structures, etc), but I think that these "advanced" features would not be common in compiled programs
3. Macros are expanded in the listing file
4. All file starts/end are present (and infact in my previous source conversion I extracted the information, see the ".list" file above)
5. Bytes assembled are present so is possible to determine the number/size
6. For now banks / long addresses are not used

Parsing a list file can get quite complicated therefore I think you are always better off if you implement a parser rather than trying to convert your list file to one that already exists.

For me is rather the opposite since I'm pretty familiar with many languages (e.g. python I used for the previous conversion) but never developed .ts code extensions (even if code seems simply to be read and infact as you can see above I already started in the past). Moreover the new "sld" format is simpler than the old one and adapting the old converter is pretty easy (I'm already trying! :)).

Maybe you can add here an example list file output of fasmg with macros, includes and all label types included. It's easier to discuss...

The ".lst" (without "i") example above is simple, but I think it can serve the purpose.

To finish I have a question for SLD: the addresses and value of labels can be expressed in exadecimal? In case which are the valid formats? (e.g. 0xFF , $FF , 0FFH , ...)

Thank you!

PS: let me add that also the current "listing creation" is done with a macro included in the compiled source file, and so could even be changed in case to adapt immediately for example to SLD, even though I'm still preferring the post processing for easiness of development. Here the include file that do everything: https://github.com/stevexyz/DeZog/blob/fasmg-ez80/examples/zx80zx81-1k-kit%20v2/includes/fasmgutils.inc

[ped7g]

SLD vs value formatting: I didn't specify that, hm. That's somewhat oversight. I think I had request from one of the tool's author (probably from Kirby?) to keep values decimal, but can't recall it clearly, maybe I'm just imagining it.

If DeZog does parse also hexa values, I'm open to discuss addition of that in the spec, but a) it may hypothetically make it harder to implement new tools, b) the 0xab form only (no $ab or 0abh) (case sensitive or not? not sure... I don't care while outputting, will be lowercase, but tool parser is here the one on the receiver end)

[maziac]

I think I had request from one of the tool's author (probably from Kirby?) to keep values decimal, but can't recall it clearly, maybe I'm just imagining it.

:-) During development I once asked for hexadecimal output, but you were reluctant to change, I guess because of Kirby.

Now, I would be the one who wouldn't want to change it anymore.

[maziac]

For me the fasmg list files seems quite easy to parse, especially because it already includes the source (asm) file in each line.

A few questions:

1. does the filename also include the complete path with folder name? In case the files are located in sub folders.
2. can you point me to an example for a macro resolution.
3. Modules: with modules I didn't mean include files but more the modules in the sjasmplus sense. I.e. you have a special modules keyword to organize your code. The modules name would be prepended to the labels. It is a nice feature, but hard to parse. Probably this is the namespace you mentioned. Can you point me to the usage in your listing?
4. Can you point me to a local label?
5. "For now banks / long addresses are not used" What does it mean? Does fasmg support it but you don't want to implement it yet or is it not yet support in fasmg but on it's roadmap.

[stevexyz]

1. does the filename also include the complete path with folder name? In case the files are located in sub folders.

Yes if a relative or full path was specified in the source to find them it will be reported also in the listing in the same way

2. can you point me to an example for a macro resolution.

Macros are "transparent" in the listing file, meaning there is just the end result. In the "simple example" I added the utilization of the "jrp" macro (that is automatically selecting between jr and jp). There are others that might "inject" piece of codes, etc; anyway the good news is that you can ignore macros, since in the listing the source file and line number reported are the proper one of the actual instructions in the source file after all macros have been processed

3. Modules: with modules I didn't mean include files but more the modules in the sjasmplus sense. I.e. you have a special modules keyword to organize your code. The modules name would be prepended to the labels. It is a nice feature, but hard to parse. Probably this is the namespace you mentioned. Can you point me to the usage in your listing?

As you described might be namespaces, but as said I think few people would ever use them (just the ones that are building "framework" utilities like the ez80.inc or the listing utility). So in my opinion the festure can be skipped at least in a initial phase. If for curiosity you want a proper description of namespaces (and macro and the rest) here you can find the reference manual: https://flatassembler.net/docs.php?article=fasmg_manual

4. Can you point me to a local label?

Same consideration as of point 3

5. "For now banks / long addresses are not used" What does it mean? Does fasmg support it but you don't want to implement it yet or is it not yet support in fasmg but on it's roadmap.

Fasmg supports "everything" :) ... or "nothing" :D ... depends how you think of it. Fasmg is a "generic" framework in which you implement specific assemblers, in my case Z80 one implemented via the ez80.inc macros mentioned before, that for my current knowledge it is not currently supporting paging, and as before would probably be a rare feature to be used so in case it might be implemented in the future (and in case is something that we might ask in the ez80 repository itself)

[jacobly0]

Fasmg is a "generic" framework in which you implement specific assemblers, in my case Z80 one implemented via the ez80.inc macros mentioned before, that for my current knowledge it is not currently supporting paging, and as before would probably be a rare feature to be used so in case it might be implemented in the future (and in case is something that we might ask in the ez80 repository itself)

Paging wouldn't be a feature of the instruction set include file, see here for an example.

[maziac]

Thanks for the answers. Here are my comments:

namespaces: If a new assembler is included into DeZog it should also be able to cope with possible list files generated by that assembler. Therefore we should also take namespaces into account. And for the effort during parsing it is crucial if we have to create the full label during parsing by ourselves or if the list output already contains it. For the namespace it means: If a namespace is defined, does a label that occurs in the list file is a fully qualified label or just a part of it? E.g. if namespace is 'mymodule' and we define a label called 'lbl' will 'mymodule.lbl' occur in the list file or 'lbl'. Similar for the local labels, e.g. if we have a label called 'main' followed by a local label in one of the next lines called '.loop' will '.loop' appear as 'main.loop' in the list file or as '.loop'. If you can provide these 2 examples (namespace, local labels) it would be nice.

banking: I'm confused now. Probably I don't understand the fasmg syntax good enough. How would that look like in the asm or list file when used?

[jacobly0]

There seems to be some confusion about fasmg, so I will try to clarify. The fasmg program does not contain any builtin instruction sets or listing capabilities. Instead you write include files that do anything from defining instruction sets to performing arbitrary processing of the rest of the program. This allows you to create listing/symbol/map/debug or whatever postprocessed file you want. Therefore, the features and format of the listing file are entirely dependent on the code in the listing include. I haven't looked into the one in question to see which of these features it does or doesn't support and what it outputs in those cases, the simplest way would be to create a test program and find out.

Here's the usage of the banking code I linked, but of course you could define it to use whatever syntax and feature set you wanted:

page ; page 1
label1:
    ret
page ; page 2
label2:
; pageof label computes the page of a label
    ld a,pageof label1 ; 1
; offsetof label computes the offset into a page of a label
    ld hl,offsetof label1 ; $4000 (pages are mapped at address $4000 by default on the hw this was written for)
; offsetof is automatically applied to immediates
    ld hl,label1 ; $4000
    ret
dw label1 ; $4000
; in order to do non-linear math with the address you need to use offsetof
db offsetof label1 shr 8 ; $40

I'm not following this issue too closely, but if the problem here is not understanding fasmg well enough to write a correct listing include, then I might be able to help. Given the expected output from a given input that uses all the features that need to be supported I could probably put something together.

Also before it leads to confusion, fasmg-z80 and fasmg-ez80 are separate repos, and there would be no reason for the ez80 repo to contain any paging features.

[maziac]

Do I understand correctly: by modifying the fasmg include file one could basically create any output listing.

That would mean there would also be a 3rd implementation alternative: If the fasmg include file would create the same output list file/sld file as sjasmplus does, it could be used without any change in DeZog. Is this right?

[jacobly0]

Yes, that's the method I would suggest, but I realize that not everyone understands fasmg features well enough to do so, hence my offer to help.

[stevexyz]

I mentioned this solution at the end of my third post, saying also that, with my level of proficiency, I prefer the "conversion" that in the meantime I started and is already almost there, here an example output: https://github.com/stevexyz/DeZog/blob/fasmg-ez80/examples/zx80zx81-1k-kit%20v2/simpleexamplezx81.sld

@jacobly0:

1. I din't know that you created also a "Z80" version, and I always used the EZ80 one with the assume adl = 0 (as you suggested here): which are the differences?
2. if you implement a new listing to be directly compatible with DeZog even better, but in the meantime can I ask if you are able to add the variable value (in the "equ" or "=" instruction) in the current listing here? Currently I'm able to convert when is a direct value, but when is an expression I'm not able to extract the value (e.g. line 251 here)

Thank you!

[stevexyz]

For an update: I got some strange errors with the autoload of the executable in Zesarux, but doing the load manually it works: the conversion make a file that is "digested" by DeZog and I finally did a test debug with the simple example program! Tomorrow I'll try more

[maziac]

If you can provide your project together with the list/sld , obj and the launch.json file I can check the problem with the loading.

[stevexyz]

If you can provide your project together with the list/sld , obj and the launch.json file I can check the problem with the loading.

Everything is in the forker repository under examples zx80zx81-1k-kit v2: https://github.com/stevexyz/DeZog/blob/fasmg-ez80/examples/zx80zx81-1k-kit%20v2/

[maziac]

I cannot test at the moment but from the launch.json I see that you are loading with

        "load": "simpleexamplezx81.p",

"load" can only load nex, sna or tap files. And tap files only with zesarux. p-files it cannot load.

You should create an object file (raw data) instead and load that via "loadObjs", see syntax in the Usage.md.

[stevexyz]

"load" can only load nex, sna or tap files. And tap files only with zesarux. p-files it cannot load. You should create an object file (raw data) instead and load that via "loadObjs", see syntax in the Usage.md.

Yes, didn't notice that load was limited to Spectrum files (this was a ZX81 one). Anyway I tried with the loadObjs (you can see in the updated repository) but also unfortunately without success

[maziac]

The obvious thing that is missing is the executable address. If just a raw file (loadObjs) is loaded it requires to also set the address to jump to the start. I.e. in you case it is (I think):

            "execAddress": "maincycle"

But still this does not work. When examine I found that the code (from loadObjs) is not loaded correctly in Zesarux. You can easily examine with

-mv 0x4009 100

in the debug console.

I have to investigate what the problem is with DeZog/Zesarux in this case.

Anyhow, if you use the internal Z80 simulator the file is loaded and you can now debug your code:

            "remoteType": "zsim",
            //"remoteType": "zrcp",

[maziac]

Found it: The path contained a space and I forgot to put it in parenthesis when passing to Zesarux. I uploaded a fix to the the DeZog repo.

[stevexyz]

For me we can close this issue. Now there is a "makesld.py" script that is able to convert the listing in a DeZog compatible way! :) Nice to have, that can go in parallel, are the two points mentioned above for @jacobly0 (but we can discuss on the proper repository) and the possible extension of DeZog to be able to load directly the ZX81 ".p" files (but no problem in loading manually or in case with loadObjs). Thank you and happy hacking! _Stefano

[jacobly0]

Discussion continued at jacobly0/fasmg-z80#3.