Clarion Professional Developer 2.1 - Cosmetic Issue: Extraneous blinking cursor.

[zydanielson]

The blinking cursor shown in the above screen shot is not caused by the Clarion 2.1 program. This cosmetic problem did not exist in DOSBox-X version 0.83.6. It seems to have been introduced in 0.83.7. It is a cosmetic problem only, not functional one. But it annoys the operators using the system.

The extraneous cursor is not random because it can be reproduced. But look at this series of screen shots:

There is no blinking cursor initially in the Master Menu. This is as it should be.

The user uses arrow keys to move the highlighted menu selection to "Payables". With the first movement of the highlight, the blinking cursor mysteriously appears.

The user presses Enter to load the Payables Menu. When the menu comes up, there is no cursor. As it should be.

The user presses the Down Arrow key to move the highlighted selection to "Invoices and Credit Memos". The blinking cursor appears again.

Brecon's operators initially reported this as a "random" bug because the cursor was there sometimes and not other times. However, after I experimented with it a bit, I discovered I could reproduce the above sequence of events every time. So it is not random.

As I said, this is a cosmetic problem only. It does not impact the functioning of the Clarion programs. But Allen says it distracts the operators and they complain.

The way I know this wasn't a problem in 0.83.6 is because that was the version I was running in my dev machine, and I could not get the random cursor to appear no matter what I did. Unfortunately, that's when I noticed I was using the older version of DOSBox-X and decided to use the Windows Installer to upgrade to 0.83.7. When I did the upgrade, the problem started showing up on my dev machine exactly as it is showing in Brecon's server and workstations.

Here is my conf file in case you need to check it. I saved it as a TXT file so it would upload: dosbox-x.conf.txt

[Wengier]

@zydanielson Thanks for letting us know of such an issue. Could you please upload a test package that can reproduce the said issue? (I did check an earlier package you uploaded previously, but I think it may not be quite applicable here). Also, can you please check if the said blinking cursor also shows up with the default setting of DOSBox-X 0.83.6 (dosbox-x -defaultconf)? Thanks!

[joncampbell123]

@zydanielson Can you use DOSBox-X's debugger to see if it is calling INT 10h AH=01h Set Cursor Shape?

Use "BPINT 10 1" to catch calls to INT 10h AH=01h.

The handling of that call does set cursor shape but it does not consider any case (from the BIOS data area) that the cursor might have been made invisible.

https://github.com/joncampbell123/dosbox-x/blob/master/src/ints/int10_char.cpp#L398

I'm currently looking for anything else in INT 10h emulation that might affect the VGA CRTC registers related to cursor start/end scanlines, because those registers ALSO control whether or not the cursor is hidden.

[joncampbell123]

Looking at INT 10h AH=1, it's supposed to accept the cursor start/end AND two bits that correspond to the cursor invisible/blink bits as they existed on CGA hardware.

http://www.ctyme.com/intr/rb-0086.htm

Obviously on EGA/VGA only bit 5 has any meaning, so the cursor should be invisible if INT 10h AH=1 and bit 5 of CH is set.

I don't see DOSBox-X's INT 10h AH=1 emulation doing that. I see it checking if bit 5 is set, and setting the start/end backwards to make the cursor invisible, but not setting the invisible bit.

I'll make a commit that I think might resolve the issue soon.

[joncampbell123]

Um, actually looking at INT10_SetCursorShape, it's testing some bits of BIOSMEM_VIDEO_CTL in a way that seems backwards according to documentation here: https://www.matrix-bios.nl/system/bda.html

It's checking BIOSMEM_VIDEO_CTL which is 40:87h

Also: https://www.stanislavs.org/helppc/bios_data_area.html

[joncampbell123]

If you can compile from the latest commit, see if it fixes the cursor issue.

[zydanielson]

Thank you for replying so quickly on this. I was working on something else most of the night and didn't get back to this issue until now.

@Wengier if the package you are talking about that I uploaded previously was in a folder called ClaTest, that should be enough to reproduce this, though I haven't tested it. That package contains the Master Menu (MASTER.PRO). Running that with (cpro master) should reproduce it. You don't have to load anything from the Master Menu. Just use the arrow keys to move the highlighted menu selection and the blinking cursor should appear using the 0.83.7 that is in the Windows Installer. If this doesn't do it, I'll put together another package tomorrow. Also I'll have to test the 0.83.6 (dosbox-x -defaultconf) tomorrow since I already upgraded to 0.83.7 on my dev machine and will have to do a new installation of 0.83.6 to do the test. I'm too beat to do it tonight. (5:12 am here.)

@joncampbell123 I'm afraid I don't know anything about DOSBox-X code or the debugger. Neither have I ever done a compile from a commit. (Sorry.) If you or @Wengier can explain how to do the compile, or upload a compiled dosbox-x.exe from the latest commit, I'll be happy to test it. Also, that will have to wait until tomorrow. My brain is a bit fried tonight. 🙄

[Wengier]

I have tried the ClaTest program (run with CRUN MASTER) you uploaded previously in a real Pentium computer running MS-DOS, and I can confirm that the cursor does not show there.

@joncampbell123 with the latest DOSBox-X code I notice that the test program (CRUN MASTER) does make multiple calls to INT 10h AH=01h, and the modified code first=0x1e | 0x20 is executed 4 times before the main menu shows up. But the cursor is still blinking as demonstrated in @zydanielson's first post.

@zydanielson With the earlier test program CRUN MASTER will run a script before showing the main menu. With the official DOSBox-X 0.83.6 version and the setting shell=4dos.com that script would be skipped due to a bug in that version and the cursor will not show, but with the default shell the the blinking cursor appears to show up here too. On the other hand, with the official DOSBox-X 0.83.7 version and the current code that script will run (even with shell=4dos.com as the said bug was fixed), and as with DOSBox-X 0.83.6 using the default shell the blinking cursor will show up in the main menu. So with this in mind while the earlier test program may be helpful to show the problem, I don't think it is the best test program for showing the difference between 0.83.6 and 0.83.7. If possible you can certainly upload a new test program to show the difference between the versions.

Also, a new compiled package can be downloaded from the link below. In my testing I did not see real changes here yet with the default output, but you can still test it on your system and see how it works. If you like, you can certainly try to use the new TrueType font output (set "output=ttf" and optionally a TTF font like "ttf.font=consola" to use it) and run the program. Hopefully the blinking cursor will not appear there.

Download: dosbox-x-update.zip

[zydanielson]

@Wengier I downloaded the update and tested it with output=ttf and ttf.font=consola. I got this error: even though Windows tells me this font is installed.

So I changed the conf to output=ttf and ttf.font= That loaded okay. The blinking cursor problem was absent. However, random little lines appeared on the screen:

I played around with different ttf.winperc numbers and the random little lines changed where they appeared, but I could not get them to go away:

I'm pretty sure Allen isn't going to like the little random lines any more than he liked the blinking cursor.

What should I try next?

[joncampbell123]

It could mean the font bitmaps returned by FreeType aren't always big enough to fill the fixed character cell expected by the TTF output, and the rendering code needs to fill in the extra column properly.

[zydanielson]

@joncampbell123 can anything be done about this? Or is it something we just have to live with?

[joncampbell123]

I'm saying if the TTF font rendering took that into consideration, the alphanumeric cells could be filled out properly and the black lines would no longer appear. Those black lines may just be cases where FreeType's rendered bitmap is not quite wide enough for the cell. Notice the black lines are consistent for the lower case 's' and lower case 'o' for example.

[Wengier]

@zydanielson For now consola.ttf simply needs to be in the same directory as DOSBox-X.exe for ”ttf.font=consola” to load. The package I posted earlier already includes consola.ttf, why not you extract it too? You won’t encounter the random little lines when you use the consola font. (the random little lines only occurs with the built in font, which I am aware of and will fix later).

[zydanielson]

@Wengier I copied the consola.ttf to the DOSBox-X directory and it now runs fine with:

It's interesting that your conf defaults to ttf.lins=24. Clarion expects 25 x 80. This isn't a problem because I corrected the conf. I just wondered if other DOS systems defaulted to 24 x 80? If the 24 x 80 default isn't standard, you might want to make the default 25 x 80. That's up to your discretion, of course. I don't know which is more standard.

Unfortunately, I now found an extraneous cursor. It doesn't blink, of course, but it also should not be there:

In order to reproduce this, I would have to upload the entire Order Entry and Client system to you, complete with all order and client data. This is something Allen would not want me to do for security reasons. If necessary, I can write a series of dummy programs with test data that might reproduce what you're seeing above. I'd rather not take this detour unless there is no other way for you to track this down. It might be easier for you to remote into my dev machine while on the phone with me, but I don't know what sort of remoting software you might have.

Pros and Cons

1. PRO: Using ttf.font and ttf.winperc makes for a much sharper font that is easier on the eyes and can't be resized to something difficult to read.
2. CON: The color of the cursor (yellow/red) doesn't go well with our screen color scheme (blue). Is that cursor color configurable in any way? Black/gray or yellow/black would look better.
3. CON: When using the Insert key to toggle editing insert or overwrite, when in Insert mode, the bottom of the cursor looks slightly disaligned to the top, making the letter the cursor is on difficult to read.

Allen's Decision at this point

Is to update to the latest build, using ttf because he likes the crisper, more legible font. But we're wondering if we can get rid of that extraneous cursor appearing in the upper left corner at times, and if a change to the color of the cursor is possible.

[Wengier]

@zydanielson The Windows binary I posted earlier was the initial test version of DOSBox-X with TrueType font support. I have made various improvement to it today, and you will immediately notice the following changes (among many other changes).

1. ttf.font is now default to 25.
2. ttf.font=consola will try to load the font from the Windows font directory too.
3. The built-in TTF font has been updated which should look better and will no longer have the random little line issue as encountered earlier.
4. The cursor in TTF output can be made blinking as well (with ttf.blinkc option).

You can download the updated Windows binary still from the link below. Please check it out:

Download: dosbox-x-update.zip

P.S. The color and sharp of the cursor is really set by the CRUN program, not DOSBox-X, but you may not see the extraneous cursor any more.

[zydanielson]

@Wengier thank you. You continue to amaze me with your work and the speed with which you do it.

We are so close now to something that will make Allen very happy.

The Good News

The extraneous cursors are completely gone (as far as I can tell).

The Bad News

There is no cursor at all, even in an input field where there should be one.

TTF Conf Settings

[zydanielson]

I thought I would mention one other small cosmetic issue. It's not extremely important and I'm sure Allen can live with it just fine as it is. So it all depends on how much perfection you want in DOXBox-X. Clarion has a feature called a "tracker" that draws lines on the screen, creating boxes, etc. This is how it looked prior to using TTF: Notice how both vertical and horizontal lines are smooth and continuous.

This second screen shot is taken using TTF: You can see that vertical lines are still smooth and continuous. Horizontal lines, however, are slightly discontinuous and jagged.

Again, getting the cursor back inside input fields is the more important issue. That is a functional issue, whereas the rendering of drawn boxes is cosmetic only.

[zydanielson]

Results from further testing

1. I have written all programs so that when an input field receives focus, it is initially in Overwrite Mode. This is when (at present) there is no visible cursor at all.

2. When the operator presses the Insert key the mode is toggled to Insert Mode (pushing characters to the right with typing). When in Insert Mode, the cursor is visible. Toggling it again to Overwrite Mode causes the cursor to disappear.

3. As a final observation, when in Insert Mode, I can still produce the extraneous cursor. So something links the toggle of Insert and Overwrite mode to the original extraneous cursor problem. While still on this screen, toggling to Overwrite Mode makes the extraneous cursor disappear.

I hope this information is helpful.

[joncampbell123]

@Wengier @zydanielson

If you comment out this call to INT10_SetCursorShape(), does it fix the issue?

https://github.com/joncampbell123/dosbox-x/blob/master/src/ints/bios_keyboard.cpp#L576

Real hardware doesn't change cursor shape within the keyboard ISR when you hit the Insert key on the DOS prompt or anywhere else. That cursor shape change is usually done by the program, outside the ISR.

[joncampbell123]

Another suggestion, for the border characters, is to take the bitmaps given by FreeType and alter the bitmap to make the line continuous. Specific code ranges known to correspond to those single and double line character codes could be modified by code that looks at the edges and extends the line to the edge of the bitmap so it is rendered as a solid line.

[Wengier]

@joncampbell123 Yes, if I comment out the call to INT10_SetCursorShape() in that line, I believe the blinking cursor will not show in the CLATEST program. On the other hand, with this code change pressing the Insert key on DOSBox-X's DOS shell will not change the cursor shape any more, but I have tested this key in a real MS-DOS system and pressing it does change the cursor shape in MS-DOS.

@zydanielson You can try the latest DOSBox-X build with this change applied. Since the test program you uploaded earlier only allows me to go into the main menu, but not anything else, I am not able to test the interfaces as you showed above, and at this time you have to check out the result yourself. Also, for the drawn boxes rendering I am also unable to see it myself because of this reason, and I want to know if you are using the Consola font or the (new) built-in TTF font? Could you post a (new) test package that allows me to check it out myself? Thanks!

In any case, you can download the updated DOSBox-X package from: dosbox-x-update.zip

[joncampbell123]

As long as the DOSBox-X command shell is emulating DOSKEY-like editing controls, the shell needs to handle the insert key itself, rather than IRQ1 handling it for the shell.

[Wengier]

@joncampbell123 I see. So basically that line was in the wrong location and needs to be moved to the shell code to be directly handled by the shell instead.

[joncampbell123]

Correct. The shell input code already handles the Insert key in PC-98 mode, it needs to do the same in IBM PC/XT/AT mode.

A quick modification to the code that would also solve the problem, would be for that IRQ1 handler code to check whether the cursor is invisible according to the BIOS data area, and OR the starting scanline with 0x20 to keep it invisible.

[joncampbell123]

I just pushed a change to that ISR that retains the cursor shape change but preserves the invisibility state. Does that help the issue?

[zydanielson]

@joncampbell123 and @Wengier: This is perfect, concerning the cursor problem. No extraneous cursor shows up anywhere I used to find them. The cursor changes shape correctly both under Clarion control and in DOS. ttf.blinkc works correctly as you intended it to. Personally, I like the blink on. But I'll let Allen choose which way he wants to see it.

@Wengier I put the consola.ttf included in the package into the DOSBox-X directory, so I assume that is what is being loaded, and not the one from Windows. With ttf.font=consola the horizontal lines are still jagged:

When I leave out the font name and run with ttf.font= am I right in assuming that is using the built in TTF? In any case, running under that configuration the horizontal lines are smooth and continuous. Notice the lines are dimmer but look better. They are just slightly harder to see. I don't know that this matters. I'll check with Allen on Monday to get his opinion.

I used an old copy of Fireworks to show the difference of the two TTF renderings. The one on the left is the built in TTF and the one on the right is the consola.ttf in the package. Don't worry that it is slightly blurry—that is from my imaging software. It is not blurry in DOSBox-X. What I'm trying to show here is the difference in luminosity of the two fonts. I think the built in one is just slightly dimmer than the one in the package. This is true of the drawn box lines as well as the letters of the font itself.

My Prediction Concerning Allen

1. He will like the brighter font of consola.ttf in the package better than the built in TTF.
2. He will like the smooth vertical lines of the built in TTF better than the consola.ttf in the package.
3. One of these is likely to satisfy him even if no further work is done on the rendering.

So where we go from here is up to you.

You folks ROCK!

[zydanielson]

@joncampbell123 I was posting when you put in your last reply:

I just pushed a change to that ISR that retains the cursor shape change but preserves the invisibility state. Does that help the issue?

I don't know if you still need me to do further testing, or if we're done. In any case, I'll test the change you pushed when @Wengier gives me a download link to the compiled package.

It seemed to me that the change of cursor shape was already working in both Clarion and in the DOS shell.

[joncampbell123]

@zydanielson I'd like to know if the ISR change I pushed solves the issue as well. If it does, the DOS shell can keep it's current insert behavior while allowing your software to keep the cursor hidden.

Remember the removal of that part was a hack to solve the issue, but it also prevents the shell as implemented currently from reflecting insert/overwrite behavior with the cursor shape.

Long term, the shell should handle it itself, but that's more work than just fixing the ISR as it exists now to preserve the cursor invisibility state.

[joncampbell123]

@Wengier On an unrelated note, I noticed the dosbox-x.reference.conf file isn't showing some settings in the [dos] section. "minimum mcb segment" isn't there, even though it still exists. Did some setting remapping break things?

[zydanielson]

Okay @joncampbell123. I'll do further testing when @Wengier links me to the latest compile.

I realize I'll be more useful to you if I can learn to compile it myself. I'll look at the documentation and see if it's something I can learn without too much trouble.

Thanks for all you do.

[joncampbell123]

@zydanielson If you are on Windows, and you have Visual Studio 2019, there is a VS project file in the vs2015 directory you can use to compile DOSBox-X.

@Wengier The TTF output works here on Linux. I might be able to modify the code to make room for the menus so they don't draw on top of the TTF output.

[joncampbell123]

@Wengier I just found a silly bug in your TTF output. Apparently it assumes 80-column output. It breaks if you set up 40x25 text mode. It needs to use whatever the CGA/EGA/VGA emulation says the character cells per row is, whether 40, 80, or anything else. I think I can fix that.

[Wengier]

@zydanielson The updated DOSBox-X package is now available from below. Please check it now and see if there are improvements in the cursor behavior.

• dosbox-x-update.zip

@joncampbell123 Thanks for the cursor fix and the modification of the code to make room for the SDL draw menu. I have further updated it so that the SDL drawn menu should appear for this output by default. Also, the default settings for ttf.lins and ttf.cols are now both 0, which means it will use the current settings automatically when possible. And the "minimum mcb segment" option now appears in both dosbox-x.reference.conf and dosbox-x.reference.full.conf.

[joncampbell123]

@Wengier I just pushed a fix for the SDL drawn menu to fill the width of the window properly. Also it seems that once you set 40x25 mode, 80x25 mode becomes inaccessible (it's stuck at 40x25 mode). I'm going to mod your code to not alter VGA mode 3 if ttf.lins and ttf.cols are zero (not specified).

[Wengier]

@joncampbell123 You are certainly more familiar with video modes in general than me. Please go ahead to change the code so that it will work better.

[joncampbell123]

@Wengier I just pushed more changes to the TTF output. Is there a reason the code in vga_draw.cpp copies the full 16-bit char + attribute, then copies only the foreground/background color, interleaved together?

Also, your TTF code seems unable to render foreground=0 and background!=0 making it impossible to render inverse text in PC-98 mode.

Since the draw char field is 16 bits wide, any plans to allow the full 16 bits to hold a unicode character? So that, say, PC-98 mode can convert unshifted JIS to unicode and render Japanese properly?

[zydanielson]

@joncampbell123 the cursor works perfectly now and looks identical to what we used to see in VMware and in DOS. Excellent work. It's just a cursor now with perfect color matching our color scheme. I left ttf.colors= blank. I'll take a look at dosbox-x.reference.full.conf tomorrow to see what that does and play around with it a bit. But I can tell you right now that we have achieved the goals I reported when creating this issue. There are no extraneous cursors, ttf.blinkc works as expected, and pressing Insert to toggle typing mode works exactly as it should, including the shape of the cursor. All of this is tested in both Clarion and in the DOS shell.

@Wengier using consola.ttf in the package still produces slightly jagged horizontal lines. Using the built in TTF has smooth horizontal lines, but the lines and the letters of the font are noticeably dimmer, the same as the screen shots above. I don't know if you intend to smooth out the horizontal lines in the packaged consola.ttf or not. I'll show all this to Allen in the morning and let him make his choices as to what is available in the present build. I'll let you know what he says.

Thank you both.

[joncampbell123]

One final issue (whether I can figure it out or you do) @Wengier is that CGA/EGA modes and TTF output have incorrect colors and PC-98 mode is unable to render inverse text (foreground=0 && background!=0).

[zydanielson]

@Wengier it's the strangest thing. I installed the latest update to Allen's server. At first he could see the jaggedness of the horizontal lines. But as we both played with it on his computer and several of his workstations, neither of us could detect the jaggedness any longer! I even wondered if the only reason I could see it at first was because I knew it was there and maybe Allen could see it because I told him to expect it. However, Allen is quite happy with what we have now. And if the jaggedness is so subtle that it disappears as you are looking at it, it's probably not worth doing anything about it.

Here is the TTF configuration we're going with:

I'll leave this issue open since you and Jon are still discussing related issues here. I'm totally satisfied at this point.

Many thanks.

All my best, Zy

[Wengier]

@zydanielson The console.ttf file included in the package is the unmodified original console TrueType font. This is only one TTF font out there, but there are so many different monospaced TrueType fonts available, so if you wish you can certainly try different (monospaced) TrueType fonts and see the result, instead of limiting to these two (consola and the builtin TTF).

@joncampbell123 TTF output on PC-98 mode seems to be reliable right now. Japanese characters can already be rendered properly at this time.

[joncampbell123]

@Wengier Not 100% completely though. Border characters do not appear, the ASCII backslash needs to render as the Yen symbol, and single-width katakana are not rendered yet by the TTF output (so, some text including File menu entries in EDIT.COM taken from a boot disk do not appear). Also some elements, including the function row at the bottom, expect certain control characters (like CR) to render as ␍ (which is fortunately a Unicode code point and therefore SHOULD be easy to map to the TTF renderer and font) and not as the code page 437 music note.

[zydanielson]

@Wengier

... there are so many different monospaced TrueType fonts available ...

Thanks for this reminder. Since Allen is happy now, I'll probably just continue to use consola.ttf in the package. But I'll remember what you said as an alternative, should he ever bring it up again. ☺

When you and Jon are done with this issue, go ahead and close it.