rfc: get rid of INIT_TEXT?

[stsp]

Commit 2319c138 removed the last real use of INIT_TEXT. Now the question is: should we completely eliminate the INIT_TEXT itself? I hate doing that for 2 reasons:

1. That would be a biggest departure ever from freedos architecture. After that we can't say that we compile freedos sources.
2. That would be a major reduction of fdpp scope and the code. Support of INIT_TEXT is the biggest deal in fdpp. Just passing the flat 64bit pointers as a short pointers to asm, costed months of considerations and resulted in a darkest code magic in the world. :) And there are more.

So instead of doing that, I'd like to ask @bartoldeman @tkchia and @PerditionC if they actually need INIT_TEXT. Can it be removed also from freedos? See 2319c138: signon() was called before init_kernel(), so it couldn't use HMA_TEXT. But I swapped them and now it can. Every other use of INIT_TEXT seems to happen even later - when HMA_TEXT is also usable. So... is there any reason why INIT_TEXT cannot be removed from vanilla freedos?

[tkchia]

Hello @stsp,

My understanding is that INIT_TEXT is --- as its name suggests --- used to hold kernel code that is only used during intiialization. Once command.com kicks in, the kernel can jettison the contents of INIT_TEXT, and free up the memory so that user programs can use it. This is better than having the code uselessly lying around and taking up memory space (especially when 1 MiB is not exactly a lot of space).

Whether HMA_TEXT is usable or not (?) is, I think, not really the point here.

Thank you!

[stsp]

Whether HMA_TEXT is usable or not (?) is, I think, not really the point here.

Not sure I understand your point. The important thing here is that INIT_TEXT contains the same code as HMA_TEXT. Take execrh.asm as an example:

segment HMA_TEXT
EXECRH:
        EXECRHM

%ifndef WATCOM

segment INIT_TEXT

INIT_EXECRH:
        EXECRHM

So it expands EXECRHM macro for both INIT_TEXT and HMA_TEXT. Or take iprf.c as an example:

/* init code printf                     */
/* simply include prf.c while defining  */
/* _INIT: reduces command line length   */
/* and simplifies make procedure        */
#define _INIT 1
#include "prf.c"

Everything is simply duplicated. If you have HMA_TEXT usable from the very beginning, then why would you need such a duplication? And it gets worse because in other places there are subtle differences: for example INIT_TEXT variant takes the short pointer, and HMA_TEXT counterpart takes far pointer for the same argument. So basically you have both: lots of duplications and lots of subtle differences between the same functions for init/non-init. Do you think its really needed?

[stsp]

Or take DosExec() in intr.asm as an example:

segment HMA_TEXT

;; COUNT ASMPASCAL res_DosExec(COUNT mode, exec_blk * ep, BYTE * lp)
    global RES_DOSEXEC
RES_DOSEXEC:

Wow, cool, seems specific to INIT_TEXT? Except that its not. You have its counterpart in task.c:

COUNT DosExec(COUNT mode, exec_blk FAR * ep, BYTE FAR * lp)

And now you see that it takes FAR args while its INIT_TEXT counterpart takes near args. Note that, besides being a mess, its actually quite difficult to support, i.e. half of fdpp is needed only to support near pointers passing to INIT_TEXT (I'll probably drink if its removed :)

[tkchia]

Hello @stsp,

The important thing here is that INIT_TEXT contains the same code as HMA_TEXT.

There are also quite a number of functions in INIT_TEXT that are not in HMA_TEXT --- Init_clk_driver (), dsk_init (), etc.

Since these routines are --- in this implementation --- currently in INIT_TEXT, they will be thrown away --- and their memory freed up --- after command.com starts.

If these routines go in HMA_TEXT, they will be sitting around uselessly in main memory after command.com starts.

Or take DosExec() in intr.asm as an example:

(You need to read the code more carefully. res_DosExec(...) is the user-side routine that just calls the syscall. (Plus, it is not in INIT_TEXT...) DosExec(...) is the kernel routine that actually implements the syscall.)

Thank you!

[tkchia]

Hello @stsp,

If you ask me: in the case of fdpp --- which will (most probably) run in a 64-bit hosted OS with support for virtual memory and "swapping out" unused pages --- it is probably not so important to have a separate INIT_TEXT. You might be able to get away with unifying INIT_TEXT and HMA_TEXT into one segment, or some other shenanigans.

However, for vanilla FreeDOS, INIT_TEXT is still a useful thing to have.

Thank you!

[bartoldeman]

There are two reasons for INIT_TEXT:

1. Organizational: it makes sure that the functions that are only used at init time are placed together so a contiguous block can be jettisoned.

2. Segment issues: for Turbo/Borland and GCC compilers (perhaps it's improved for GCC but that was the case before) INIT_TEXT+HMA_TEXT is bigger than 64k, so only in those cases the init and resident functions live in different segments and that's the reason we need all this gymnastics and duplication. An alternative is far calls to the common code but that then increases code size. The situation with Watcom is considerably simpler, that's the reason for %ifndef WATCOM and the duplicates were eliminated because CS: it the same for HMA_TEXT and INIT_TEXT, there it's just INIT_TEXT at the end as a grouping.

printf is actually only present in INIT_TEXT in production builds, it's only in HMA_TEXT as well for debug builds. Since printf is so big in production builds only a few functions from prf.c are in HMA_TEXT as duplicates.

In the end you can pretty much see the init code as an independent DOS program that communicates with the kernel mostly using int21 calls (there may be some exceptions though I can't remember).

[stsp]

You need to read the code more carefully. res_DosExec(...) is the user-side routine that just calls the syscall.

Yes, sorry for a mis-name.

Plus, it is not in INIT_TEXT

See init_DosExec() in the same file. Its what I had to name initially, not DosExec() in task.c. The only difference they have, is the label name: no_exec_error vs exec_no_error to avoid clash.

There are also quite a number of functions in INIT_TEXT that are not in HMA_TEXT --- Init_clk_driver (), dsk_init (), etc.

OK, those are the C routines, so I didn't know they also belong to INIT_TEXT - definitely not in fdpp, so thanks for mentioning them. For fdpp they represent no problem. I would only suggest getting rid of an asm part of INIT_TEXT then.

[stsp]

Hi Bart,

An alternative is far calls to the common code but that then increases code size.

Yes, this is exactly what my suggestion is. This increases the C-compiled code size, you mean? How much?

The situation with Watcom is considerably simpler, that's the reason for %ifndef WATCOM and the duplicates were eliminated because CS: it the same for HMA_TEXT and INIT_TEXT, there it's just INIT_TEXT at the end as a grouping.

Hmm, I can't understand that part. If there is just an INIT_TEXT, then what is to call after init?

printf is actually only present in INIT_TEXT in production builds, it's only in HMA_TEXT as well for debug builds. Since printf is so big in production builds only a few functions from prf.c are in HMA_TEXT as duplicates.

OK, makes sense.

[stsp]

There are also quite a number of functions in INIT_TEXT that are not in HMA_TEXT --- Init_clk_driver (), dsk_init (), etc.

Hmm, if I am reading the freedos's makefile right, then even config.c, main.c and most of other C code goes to INIT_TEXT? Well, ok, in that case I realize it can't be removed from freedos. Its just that in fdpp the Cish funcs do not go to INIT_TEXT, so obviously we are not on the same train here.

[tkchia]

Hello @stsp,

Hmm, I can't understand that part. If there is just an INIT_TEXT, then what is to call after init?

If I understand @bartoldeman correctly, if HMA_TEXT + INIT_TEXT combined is small enough, it becomes possible to put them in the same segment group, so INIT_TEXT kind of becomes a "part" of HMA_TEXT; something like this:

+---------------+
| HMA_TEXT      | <- offset 0 for both HMA_TEXT & INIT_TEXT
| +-----------+ |
| | INIT_TEXT | |
| +-----------+ |
+---------------+

Once the kernel initialization is done, INIT_TEXT can be freed up as a single memory block.

Thank you!

[stsp]

An alternative is far calls to the common code but that then increases code size.

Now as I realize most of the C code goes to INIT_TEXT in freedos, another question: you mean it increases the code size of INIT_TEXT?

Once the kernel initialization is done, INIT_TEXT can be freed up as a single memory block.

But if its freed, there is a problem. Because obviously those %ifndef WATCOM that we see, prevent the generation of the HMA_TEXT counterparts. In which case, after INIT_TEXT is freed, there is none at all. No execrh() for example. So the only interpretation of Bart's statement I can think of, is that with watcom INIT_TEXT is not freed? Still strange.

[tkchia]

Hello @stsp,

Hmm, if I am reading the freedos's makefile right, then even config.c, main.c and most of other C code goes to INIT_TEXT?

The C code that is needed after initialization are mainly covered by the make rules that use the default recipe, e.g.

blockio.obj: blockio.c    $(HEADERS) $(TARGET).lnk  
break.obj: break.c        $(HEADERS) $(TARGET).lnk  
chario.obj: chario.c      $(HEADERS) $(TARGET).lnk  
...

The code for these modules, if I understand correctly, will go into HMA_TEXT.

Thank you!

[bartoldeman]

An alternative is far calls to the common code but that then increases code size.

Yes, this is exactly what my suggestion is. This increases the C-compiled code size, you mean? How much?

Too much. But even worse is that you then get different issues, namely that the compiler generates near calls to functions that do long multiplication/long division etc, and you need to somehow deal with those.

The situation with Watcom is considerably simpler, that's the reason for %ifndef WATCOM and the duplicates were eliminated because CS: it the same for HMA_TEXT and INIT_TEXT, there it's just INIT_TEXT at the end as a grouping.

Hmm, I can't understand that part. If there is just an INIT_TEXT, then what is to call after init?

with watcom there's a single code segment for HMA and Init code. https://github.com/FDOS/kernel/blob/35a18350a0ab27e28799122fb075adb55ee1f7f0/kernel/segs.inc#L51

group   TGROUP          HMA_TEXT_START HMA_TEXT HMA_TEXT_END INIT_TEXT_START INIT_TEXT INIT_TEXT_END
%define IGROUP          TGROUP

in this case far code is only needed to call things in LGROUP (0070:xxxx).

The init code can then freely call things in HMA_TEXT using plain near calls while it runs, and once it finishes the memory past INIT_TEXT_START is released.

[bartoldeman]

you mean it increases the code size of INIT_TEXT?

both INIT_TEXT and HMA_TEXT since they both then need to use far calls for common functions.

Once the kernel initialization is done, INIT_TEXT can be freed up as a single memory block.

But if its freed, there is a problem. Because obviously those %ifndef WATCOM that we see, prevent the generation of the HMA_TEXT counterparts. In which case, after INIT_TEXT is freed, there is none at all. No execrh() for example.

See also https://github.com/FDOS/kernel/blob/35a18350a0ab27e28799122fb075adb55ee1f7f0/kernel/init-mod.h#L50 With Turbo you have two execrh()s: execrh() and init_execrh(). With Watcom, there's only one, in HMA_TEXT. The init code calls execrh() in HMA_TEXT using a near call.

So the only interpretation of Bart's statement I can think of, is that with watcom INIT_TEXT is not freed?

It is freed.

[tkchia]

Hello @bartoldeman,

Segment issues: for Turbo/Borland and GCC compilers (perhaps it's improved for GCC but that was the case before) INIT_TEXT+HMA_TEXT is bigger than 64k,

Well, I guess I can try to continue improving on gcc-ia16, if it still needs improving. :-)

Improving on the Borland C++ toolchain is another matter though. :-| How feasible might it be to get Borland C++ to use the some sort of fastcall calling convention throughout an entire program? Will this help to alleviate the problem?

Hello @stsp,

Its just that in fdpp the Cish funcs do not go to INIT_TEXT, so obviously we are not on the same train here.

I guess the equivalent in 64-bit x86_64-linux-gnu-gcc land, might be to use __attribute__ ((section ("..."))) qualifiers, or some sort of objcopy hackery, to place fdpp's C-ish initialization routines into their own little section. Then you can deal with the initialization routines as a single block of memory (or pages) in the 64-bit flat address space.

This might be useful to do, but as I said earlier, it is probably not so important.

Thank you!

[bartoldeman]

Well, I guess I can try to continue improving on gcc-ia16, if it still needs improving. :-)

for all I know it's already good enough, it may well be. It's just been quite a while since I last looked at it...

Improving on the Borland C++ toolchain is another matter though. :-| How feasible might it be to get Borland C++ to use the some sort of fastcall calling convention throughout an entire program? Will this help to alleviate the problem?

mostly we support different compilers because often an issue surfaces with one that doesn't with another, it's just better coverage. It does come at the cost of ugliness though.

With Turbo C 2.01 (yes the old museum compiler) an advantage was also that we had the smallest compressed kernel with it, due to the codegen being more boring so more compressible, which was nice for floppies.

[stsp]

both INIT_TEXT and HMA_TEXT since they both then need to use far calls for common functions.

OK, but what exactly makes an increase? call far vs call? Because I look to execrh() and it seems to be taking FAR pointers in both cases, and same with init_DosExec() vs res_DosExec() which both take near. So what would differ besides the call far?

With Turbo you have two execrh()s: execrh() and init_execrh(). With Watcom, there's only one, in HMA_TEXT.

Oh, I see! Again wrong freedos reading on my side.

[stsp]

Hmm... I grepped for WATCOM in all asm sources of freedos, and it seems to me that only execrh() gets protected by such ifdef. I think init_DosExec() INIT_CALL_INTR etc are all compiled nevertheless. What am I missing again?

[stsp]

See also https://github.com/FDOS/kernel/blob/35a18350a0ab27e28799122fb075adb55ee1f7f0/kernel/init-mod.h#L50

OK even from that list I can see that only execrh() is protected and string.h functions (string.h ones are irrelevant for fdpp). Why other init functions do not have a WATCOM guard? If they did, I could disable them also for fdpp.

[bartoldeman]

init_call_intr doesn't have an equivalent call_intr -- so it only exists in init code, and one could rename it to call_intr without issue. Once upon a time a seperate call_intr for resident code existed but not any more. All interrupt calls by DOS resident code go via asm functions other than intr style functions.

For init_DosExec vs res_DosExec, it looks like indeed for Watcom, init_DosExec can be eliminated and calls to it replaced by calls to res_DosExec, i.e. a missed optimization.

[stsp]

For init_DosExec vs res_DosExec, it looks like indeed for Watcom

res_read() vs read()? But yes, not many missed. So I probably disable a few and close this.

[stsp]

In fact, in fdpp both res_DosExec() and res_read() are implemented in C and take the far pointers, unlike their INIT_TEXT counterparts (and unlike their impl in freedos). So I will probably not do anything at all. Thanks for explanations!