Add NEC PC-98 support to ELKS

[ghaerr]

This issue is being opened to discuss porting issues and merging the development @tyama501 is doing into master.

This continues the discussion started in #1040.

I will produce a more detailed list of discussion items shortly.

[ghaerr]

PC-98 attributes does not have foreground color or background color. It only have 3bits color which is assumed foreground.

I see. Well, if PC-98 cannot display any background color other than black, then you may have to add/keep the code for reversing fg/bg like you've done. Is there a specification page URL that shows the format of the PC-98 attribute word?

Thank you!

[tyama501]

I mentioned above but there is in here. Text V-RAM http://software.aufheben.info/contents.html?contents_key=textvram

pc-98 text screen is overlapped on graphic screens but they are separated. If we fill graphic screen with other colors it may change.

Thank you.

[tyama501]

Hello @ghaerr ,

It was inconvenient that I cannot use arrow keys so I added kbd-poll-pc98.c and modified conio-pc98 to use additional INT18 AX=0407 to see the arrow keys are pressed or not as follows.

https://github.com/tyama501/elks/commit/6cd90cf42e68f70b2ce5b45fb8f4ad867a5e8926

Is it ok to add kbd-poll-pc98? Tell me if there is better solution.

I could use arrow keys for command line history and move cursors in "kilo" but I am still suffering (or even more?) for slow speed.

Thank you. I wish you a Happy New Year.

[ghaerr]

Hello @tyama501,

I wish you a Happy New Year.

And a Happy New Year to you too!!

Is it ok to add kbd-poll-pc98?

Yes, that looks good, go ahead. I would suggest renaming all the 'early_xxx' routines to 'bios_xxx', with the exception of 'early_putchar', which is named early_putchar since it's used in the pre-boot sections of the kernel for output, outside the console driver.

I could use arrow keys for command line history and move cursors in "kilo" but I am still suffering (or even more?) for slow speed.

I know that kilo is a quite slow program in general. Is the speed problem just since you've added the 'early_arrow' check, or is it just the kilo screen redraw speed, or all screen drawing that is slow?

Thank you!

[ghaerr]

I could use arrow keys for command line history and move cursors in "kilo"

If PC-98 supports the six Home/End/PgUp, etc, keys, those should also be added to kbd-poll-pc98.c. That will complete the special key processing that the shell and screen editors use.

[tyama501]

Thank you @ghaerr ,

Especially slow in kilo that I can see redrawing even I used 25MHz, 386 emulator but I am not sure this caused by adding arrow function yet. I think whole display is slower than before.

I think pc-98 has Roll up / down / Ins / Del and there is no Home and End. http://nkmm.org/yagura/kbd/

What are Home and End for?

[ghaerr]

I think whole display is slower than before.

This could be because of the conv_pcattr() call, or it could be because of the kbd-poll-pc98.c scheduling a timer every 8/100 seconds for the INT 18h AX=0407h call. You can increase the timeout by changing the 'timer.tl_expires' value, but I don't expect that to do much, unless the INT 18h call is very slow.

It is likely the conv_pcattr() call. If so, you might special case the most often used attribute (A_NORMAL) and do a single if check and return the pre-calculated value for that. That should help a lot since the screen is blanked by 1920 A_DEFAULT attributes all the time, and it may be too costly to compute the converted attribute.

[ghaerr]

What are Home and End for?

Usually top and bottom of file. What does the PC-98 keyboard look like?

[tyama501]

@ghaerr , There is pictures in the following URL. http://nkmm.org/yagura/kbd/

Roll Up stands for Page down and Roll Down stands for Page up.

[ghaerr]

If so, you might special case the most often used attribute (A_NORMAL) and do a single if check and return the pre-calculated value for that.

For fastest speed, perhaps something like:

int attr = C->attr == A_DEFAULT? 0x??: conv_pcattr(C->attr);
...
pokew(addr, AttributeSeg, attr);

everywhere that it is used, and take the attr calculation out of any loops.

[tyama501]

Thank you @ghaerr ,

Nice idea. I will try that.

[tyama501]

Hello @ghaerr

With your ideas the screen speed improved a lot except for kilo. Thank you. I reversed code and found that slow speed of kilo is not caused by adding attribute nor adding arrow keys.

When I used kilo, the moving cursor can be seen with redrawing as the following video.

https://user-images.githubusercontent.com/61556504/147815677-f9110bf9-4a2a-43dd-bcce-b536fa172b39.mp4

Are these also can be seen in slow IBM-PC? Or does the IBM-PC disable cursor when the redrawing?

[ghaerr]

Hello @tyama501,

With your ideas the screen speed improved a lot except for kilo. Thank you.

Good. It sounds like PC-98 will be good for ELKS because it depends on 8086 code being as fast as possible. As you point out other possible slow areas of operation, we can determine how an algorithm might be rewritten, possibly in ASM, to help.

I reversed code and found that slow speed of kilo is not caused by adding attribute nor adding arrow keys.

When kilo was first contributed, I took a look at it internally, since it is a pretty cool screen editor for being in a single source file. However, this small size comes at a price, namely the draw code when inserting any characters redraws the entire screen below the cursor, and cannot easily be made faster. So kilo on ELKS will always be slow.

Are these also can be seen in slow IBM-PC? Or does the IBM-PC disable cursor when the redrawing?

I don't have my slow real PC available to test right now, but I seem to remember it is quite slow, as described above. The cursor remains on when kilo redraws the characters, so it can be seen. However, the direct console does not update the cursor during the clear functions, just like PC-98. So I think your PC-98 direct console is OK.

I just looked again at the PC-98 direct console, and I think the last potential speedup that might be useful would be to change the fmemcpyb (far memcpy byte) to fmemcpyw (far memcpy word). This would have the effect of moving words instead of bytes, and should speed scrolling up a bit. The last argument would be shifted right to convert bytes to words.

[EDIT: actually, the fmemcpyb code already tries to do word memory copy so the above change won't make any noticable difference.]

Thank you and Happy New Year!

[tyama501]

Hello @ghaerr ,

Thank you for the comment. I will create PR for updates for the direct console.

I also modified conio-pc98 and kbd-poll-pc98 to add Page Up and Down function. I will make another PR for these. Insert and Delete seems not included in the original kbd-poll.

[tyama501]

Use default attribute value for PC-98 in console-direct-pc98 #1083

[tyama501]

Add kbd-poll-pc98 #1084

[tyama501]

Hello @ghaerr ,

Next I will try reading SCSI drive but one thing we need to consider is whether it is ok or not to assume hda=SCSI drive.

Many of PC-9801 in 80's and early 90's uses SCSI(or SASI) drives but PC-9821 in mid 90's has IDE bios and uses IDE drives. (These PCs have 486 or Pentium for CPU and may be not main target of ELKS since these can use other Unix OS like FreeBSD/pc98.)

Maybe I will first try with hda=SCSI.

[tyama501]

Hello @ghaerr ,

After I merged the latest upstream master, the following /dev/fd*** messages appeared. It says sectors = 616.0kb and fd0/1 are duplicated. Do I need to do something for this?

I also modifying bioshd_gethdinfo for harddisk but I still having trouble with getting CHS parameters with int1B.

These fd messages appears even I disabled CONFIG_BLK_DEV_BHD.

Thank you.

[ghaerr]

Hello @tyama501,

After I merged the latest upstream master, the following /dev/fd*** messages appeared.

Yes, I found out about that code after Conrad reported a bug with CONFIG_SMALL_KERNEL not defined, and set it as the default display. Seeing these new issues about the group of fd's being duplicated, I may revert it.

It says sectors = 616.0kb and fd0/1 are duplicated. Do I need to do something for this?

Oops! Forgot about the 1K sectors. Also, not sure why the entire group is duplicated, this will likely be a problem on other systems as well. I'll fix it.

These fd messages appears even I disabled CONFIG_BLK_DEV_BHD.

You mean hd messages in the last screenshot? Ok, I'll look into fixing that also, thanks!

I also modifying bioshd_gethdinfo for harddisk but I still having trouble with getting CHS parameters with int1B.

I'll fix the above problems, and when you get hard disk support working for PC-98 please submit a PR.

Thank you!

[tyama501]

Thank you @ghaerr ,

〉You mean hd messages in the last screenshot?

No, I mean these messages are not caused by where I modified since these also appear when BHD is not defined.

[ghaerr]

No, I mean these messages are not caused by where I modified since these also appear when BHD is not defined.

Thanks, I understand now. I have fixed this issue in #1096.

Is there a way to determine how many physical drives are actually installed on PC-98 system? If so, it would be nice to set 'ndrives' to the number of installed drives from bioshd_getfdinfo() for the CONFIG_BLK_DEV_BFD_HARD case. Then, only the actually installed floppy drives would be displayed, instead of the (now fixed) four (/dev/fd0-3).

If you can get the above dynamic number of floppy drives installed from PC-98 BIOS, then you can probably move to using just CONFIG_BLK_DEV_BFD instead of using the hard-coded (CONFIG_BLK_DEV_BFD_HARD) option, although the disk format will still be hard-coded.

I also modifying bioshd_gethdinfo for harddisk but I still having trouble with getting CHS parameters with int1B.

I can see the biggest problem is that you are setting C and H to zero, which causes a BIOSHD read error.

[tyama501]

Thank you @ghaerr ,

I couldn't find the actual drive numbers in bios so now I'm trying to prove each disks in gethdinfo. If this works, may be it can be used to getfdinfo too.

[tyama501]

Hello @ghaerr ,

OK, I think I could get the right CHS values for hda with the following modification. https://github.com/tyama501/elks/commit/67586e48695b91f4eb2667b03c27938f85de6aba

CHS = 666,8,25 is correct but it seems the total amount is calculated with 1024Bytes sector. The hard drive is 512Bytes sector and the total amount should be 68MB not 130MB. Where should I modify for this?

mount is still failing.

I am now using "drive" variable directly for SCSI ID. This means I can use SCSI ID from 0 to 3 from smaller value. I think I need to scan SCSI ID 0 to 6 and create map for the drive variable.

Thank you.

[ghaerr]

Hi @tyama501,

Thanks for the information about the PC-98 INT 1B operation.

OK, I think I could get the right CHS values for hda with the following modification.

That looks good, when HD is fully working send that over and we'll commit it. I would suggest though perhaps on lines 183-184, instead of just increment ndrives on call_bios success, also "break" the for loop on first failure. That is, we don't support "holes" between say, /dev/hda and /dev/hdc, the drives must be contiguous for the code in the following for loop at line 198 to work.

CHS = 666,8,25 is correct but it seems the total amount is calculated with 1024Bytes sector. The hard drive is 512Bytes sector and the total amount should be 68MB not 130MB.

Hmmm - this is going to be a problem, as right now, the sector size support in ELKS is currently limited to a single fixed size per system - that is, SECTOR_SIZE is set to a constant in config.h and cannot be different for different drive types. (It is currently set as 1024 for CONFIG_IMG_FD1232 and 512 bytes otherwise).

I suggest, if possible, for the time being that you get hard drive support working with CONFIG_IMG_FD1440 (not CONFIG_IMG_FD1232), which will set SECTOR_SIZE to 512. When you get that working, we can commit support for PC-98 hard disk. After that, I am going to have to devise a way for SECTOR_SIZE to be parameterized on a per-device basis for the FAT filesystem code, the low-level block I/O code, and the SSD and RD (ramdisk) block drivers. I would like to have this work using a mechanism that does not increase the kernel size, except for on PC-98, as that will be the only system requiring it.

If you can develop and test hard drive support for 1440k floppies first, that would be great, and I will add kernel support for variable sector sizes afterwards, which will then allow 1232k floppies and hard disk on PC-98. Does that sound like that could work?

[ghaerr]

Hello @tyama501,

I have added changes required to bioshd.c for variable sector size support in PR #1098. PC-98 with FD1232 and FD1440 disks should continue to work. This should allow you to add your hard disk support as discussed above, but more changes will be required for PC-98 hard disk support to work when CONFIG_IMG_FD1232 is defined.

Thank you!

[tyama501]

Thank you for the update @ghaerr ,

I will debug FD1440 with the emulator first. (I don't have a real bootable device for FD1440 now.)

[tyama501]

Hello @ghaerr,

I got one question while modifying bioshd. In do_bios_readwrite(), there is the following code where checking out_ax value for retries. Does this mean not only AH but also AL should be 0 after bios call? If so I need to mask AL after int1B since the returned value of int1B does not have AL value. If this is true I will do that in int1B.S.

            set_ddpt(drivep->sectors);
            if (call_bios(&bdt)) {
                    printk("bioshd(%d): cmd %d retry #%d CHS %d/%d/%d count %d\n",
                        drive, cmd, MAX_ERRS - errs + 1, cylinder, head, sector, this_pass);
                out_ax = BD_AX;
                reset_bioshd(drive);
            }
    } while (out_ax && --errs);     /* On error, retry up to MAX_ERRS times */

Thank you.

[tyama501]

The amount of the disk seems OK when FD1440. I still cannot mount it so I will debug. I wrote mbr so no mbr also not true. BTW even when I have more than 1 partition, is the following command correct? (Although I deleted 2 partition and created 1 dos partition that made no difference)

mount -t msdos /dev/hda /mnt

[tyama501]

I used PC-98 formatter and the mbr might slightly different from PC/AT.

I will try fdisk of ELKS tomorrow.

[ghaerr]

Does this mean not only AH but also AL should be 0 after bios call?

In the IBM PC BIOS, AH should be non-zero after an error (Carry bit set), and AL is the number of sectors transferred. However, the 'out_ax' variable is initialized to 0 and only set on error. Thus, it is not really clear that AL is useful.

If so I need to mask AL after int1B since the returned value of int1B does not have AL value. If this is true I will do that in int1B.S.

It seems IBM PC AL value is not really useful on error and should probably be rewritten to use 'out_ax & 0xff00' instead. For now, lets have you mask AL off in int1B.s since I cannot easily test this code on IBM PC. This must be rewritten in bioshd.c if PC-98 call_bios may need to return a valid AL value.

[ghaerr]

I still cannot mount it so I will debug. I wrote mbr so no mbr also not true.

It seems you have two problems with the hard disk port. First, for some reason, hard drives 2, 3 and 4 are somehow marked as being valid, so the partition recognition code in genhd.c is trying to read them and getting a bioshd: I/O error. This should be fixed.

The other problem, of course, seems to be that MBR is not recognized. What I suggest in this case is to temporarily skip using partions altogether, and just have the hard disk created such that the (non-MBR) boot sector is the first sector of the drive, with no partitions at all. The resulting image would be similar to a floppy disk image, except different CHS.

Also, when mounting hard disks, you must use '/dev/hda1' to mount partition 1, etc. /dev/hda is used only when mounting a non-partitoned drive. Thus your mount commands would always fail, as the incorrect drive is being specified. This is why I suggest trying no partitions first, then moving to testing partitions later, if possible.

BTW even when I have more than 1 partition, is the following command correct? (Although I deleted 2 partition and created 1 dos partition that made no difference)

No, you should be using /dev/hda1 as explained above.

The partition code is more complicated by the fact that the partition type/ID must be correct, which could possibly differ for PC-98, as well as the location of the magic 0xAA55 magic number, etc.

[ghaerr]

Hello @tyama501,

PR #1099 finalizes the changes required to support disks with different sector sizes on ELKS and PC-98. You should now be able to configure CONFIG_IMG_FD1232 along with hard drives, with each having sector sizes of 1024 and 512, respectively.

The bioshd.c driver should automatically handle this for you, as all disks have sector sizes of 512 except for the 1232K PC-98 floppies, and I have tested PC-98 and IBM PC operation. (I have not tested PC-98 hard disks though, but your work should fit in without problems, hopefully).

Thank you!

[tyama501]

Hello @ghaerr ,

Thank you for supporting the variable sector size! I could merge it.

I didn't have a way to create the large image without partition so I tried writing partition table with fdisk in ELKS. This seemed work to recognize mbr and hda1 while booting.

But I still couldn't mount it. It says all parameters are zero for partition 1. I need to debug int1B.S as well as num_drive.

BTW is there also format tool in ELKS?

[ghaerr]

I didn't have a way to create the large image without partition so I tried writing partition table with fdisk in ELKS.

The file image/hd32-fat.bin is a non-partitioned image that can be used for testing. Type "cd image; make images" to build all of them. (Requires the 'mtools' distribution to build FAT images).

This seemed work to recognize mbr and hda1 while booting.

That code only reads sector 1, so it's possible the calculations in int1B.S are still not correct, but the first sector can still be read.

But I still couldn't mount it. It says all parameters are zero for partition 1.

I would guess a sector full of zeros was returned?

BTW is there also format tool in ELKS?

For physical floppies, no. For virtual images, mfs is used for MINIX filesystems (see elks/tools/mfs and images/Make.image for details). For FAT filesystems, the above-mentioned mtools mformat program is used (see images/Make.defs and images/Make.image for details).

[ghaerr]

Hello @tyama501,

I also noticed during compilation of ELKS PC-98 for testing that console-direct-pc98.c gives several compiler warnings for non-forward declarations of functions I assume are in conio-pc98-asm.S. When you find time, you should probably fix that for best compiler checking and no warnings emitted. You're welcome to create a conio-pc98-asm.h for all the functions in the .S file. This could also be used when compiling PC-98 headless console using conio-pc98.c for no warnings.

Thank you!

[tyama501]

Thank you for the comment @ghaerr, I will work for those.

After I merged updates of your commits yesterday it seems FD1440 image cannot boot and there is the following error when building. Can you build it?

FD1232 image can boot. I also noticed that FD1232 image still says the double amount for PRINT_DRIVE_INFO of harddisk.

Does the following need to change?

ifdef CONFIG_IMG_FD1232

            __u32 size = ((__u32) drivep->sectors) * 10;    /* 0.1 kB units */

else

            __u32 size = ((__u32) drivep->sectors) * 5;     /* 0.1 kB units */

endif

[ghaerr]

Hello @tyama501,

Thanks for the report. I have fixed both problems in #1103.

[tyama501]

Thank you!

[tyama501]

Hello @ghaerr ,

OK, I could mount and read/write 32Mbyte image without partition.

I will create PR after debugging some issues and checking with the real device. I am also planning to add scanning SCSI ID.

[tyama501]

Hello @ghaerr ,

I added SCSI ID mapping as the following commit. https://github.com/tyama501/elks/commit/055299654a43a4402cf8897dcb63762abc6b2f76 Please tell me if the modified part needs to move to somewhere else.

It seems working on the emulator. I put 2 images, one for SCSI ID 0 and the other for SCSI ID 3. They are detected as 2 drives and assigned to hda and hdb. hda has a partition with IBM mbr and it can be mounted as hda1.

I will check with the real device from tomorrow.

BTW .img is not supported file for NekoProject21/w. But it can read so may be it's ok.

Thank you!

[ghaerr]

I added SCSI ID mapping as the following commit.

Nice work!

Please tell me if the modified part needs to move to somewhere else.

Lets move the last two changes (lines 741-748 and 815-822) to a function call map_drive to keep the ifdefs out of the main routine(s) and only a single ifdef in map_drive. Also put the scsi_drive_map definition at lines 131-134 above with MAXDRIVES to keep multiple ifdefs to a minimum.

The other changes at lines 203-207 and 178-187 are ok for now.

BTW .img is not supported file for NekoProject21/w. But it can read so may be it's ok.

What does this mean, that you have to rename the file back to .bin in order to use it?

thank you!

[tyama501]

Thank you @ghaerr ,

I will modify as you mentioned.

〉What does this mean, that you have to rename the file back to .bin in order to use it?

I just need to select "All files" instead of "Supported files" when reading the file.

FreeDOS(98) which is PC-98 version of FreeDOS also using .img for the FD so maybe it is ok for ELKS too.

[tyama501]

Hello @ghaerr ,

I could mount the partition with the real SCSI emulated device. I will make PR after merging some upstream.

Thank you!

[ghaerr]

Well done @tyama501, you are making wonderful progress!! Thanks for the picture. Send your changes over when you're ready, thank you!

[tyama501]

Add SCSI hard drive support for PC-98 #1115

[ghaerr]

Add SCSI hard drive support for PC-98 #1115

Thank you, I'll look at it. For some reason, GitHub didn't send me an email about this or the PR being submitted!

[tyama501]

Hello @ghaerr ,

I would like to discuss what to do for next as I wrote on #1115.

(1) PC-98 style partition support. Maybe we need to modify genhd a little.

(2) CONFIG_TRACK_CACHE I have not try track cache before neither FD nor HD. I may try it. Does this get faster disk access?

(3) XMS for 286 I see some codes for XMS in ELKS. I assume int15 is used for 286 and unreal mode is used for 386. Is this correct? My PC-9801RX21 has 2MBytes XMS/EMS. It has intel 286 so the unreal mode cannot be used. Also int15 maybe different from IBM. FreeDOS(98) has XMS driver so there is a way to control it, but I need to find out how.

[ghaerr]

Hello @tyama501,

(1) PC-98 style partition support. Maybe we need to modify genhd a little.

Yes, that would be the place to do it. I propose writing an entirely seperate procedure for PC-98 there, if it is much different than IBM PC layout.

(2) CONFIG_TRACK_CACHE Does this get faster disk access?

Yes, definitely. It depends on the BIOS being able to read an entire track quickly, with ELKS increasing the sector count from the first sector required until the end of track. This is then stored in the DMASEG track cache, which should be equal to the sectors/track of the floppy. Note: currently, ELKS also uses the same track caching buffer for hard drives, which is not very beneficial and will be removed someday; so no need to increase DMASEG to handle HD sectors/track.

3) XMS for 286 I see some codes for XMS in ELKS. I assume int15 is used for 286 and unreal mode is used for 386. Also int15 maybe different from IBM.

Yes, basically. ELKS doesn't use any MSDOS-style actual XMS API, the "XMS" term just references any memory above 1M in ELKS, not the older API.

For systems where unreal mode does not work (like 286 or Compaq Portable 386), the INT15 option calls out to a BIOS INT 15 subfunction that copies XMS memory. This subfunction could easily be replaced with a similar PC-98 function and ELKS XMS should work properly.

My PC-9801RX21 has 2MBytes XMS/EMS.

Currently, ELKS can use only about 256K of the 2M, or the buffer pointers to that 256K otherwise consume too much kernel data segment memory. However, even that amount of cache greatly helps ELKS speed.

FreeDOS(98) has XMS driver so there is a way to control it, but I need to find out how.

As mentioned above, ELKS doesn't actually use the MSDOS-style XMS API, so this won't be of any use.

Thank you!

[tyama501]

Thank you for the information @ghaerr ,

Maybe I will try track cache first.

Oh , and may I write about PC-98 in the following file? Documentation/text/porting-guide.txt

[ghaerr]

Oh , and may I write about PC-98 in the following file? Documentation/text/porting-guide.txt

Yes, that would be a great place to document the PC-98 BIOS calls that ELKS is dependent on!