For some reason, placing 0 in long press option causes the software to freeze, moving joystick to 100%

[asdf1280]

With long press option like the following: -1|-5:370 -5|-1:370 1|5:370 5|1:370 there's NO performance issue. However, if long press is set like 0|-5:370 -5|0:370 0|5:370 5|0:370 the software freezes and lever moves to 100%. I'm using master controller.

Probably it has something to do with 'handleVControllers()' in FormMain.cs, but can't exactly find why.

[DerJantob]

For me, "-1|-5:370 -5|-1:370 1|5:370 5|1:370" is also freezing the program for a short amount of time. Actually it should always freeze if a long press is triggered, because int the ConvertLongPress function

Keyboard.HoldKey(Keyboard.ConvertStringToKey(vc.increaseKey), dauer);

is not handled by another thread. In the handleVControllers function, I am using

new Thread(() =>{
//Code
}).Start();

That prevents the freezing for general controller movement. I have made it that way, because i wanted to make sure, that the ConvertLongPress gets triggered only once. I have tried putting

ConvertLongPress(vc);
if (vc.istStufenlos)
{
}
else
{
}

into the thread function, but that resulted in Long Press getting triggered multiple times. Because Longpresses do not get triggered so often, I would say that it isn't such a problem.

Today I have also found out that you are the creator of the TSW autopilot (really awesome project btw!). Does the program freezing only bother you in the autopilot, or also when using the program normally?

Regards Jannik

[asdf1280]

First of all, huge thanks for acknowledging my AP.

I had to use long press without 0, that is, like '-1:-5' to partially fix the freeze.

But please note that the freeze I mentioned in this issue is not that small amount of freeze. Using 0 for long press for some reason causes the 'entire software' to be unresponsive for a few seconds or even require a force quit. The long press might be being called excessively. It needs a careful look.

[asdf1280]

Does the program freezing only bother you in the autopilot, or also when using the program normally?

A: I actually had to remove joystick support so as that I can make a stable autopilot wrapper. However, it works the same way in handling VControllers. Maybe can you try to reproduce this whole freeze by switching window to TSW2_Controller when it's pushing lever all the way to 100?

FYI) My autopilot only applies 80% throttle. Try 80% throttle and see how it pushes all the way to 100% and freezes.

Maybe another cause is too high long press delay. But I doubt that should be a reason for the software to completely freeze and die. So it needs a fix anyway.

[DerJantob]

Using 0 for long press for some reason causes the 'entire software' to be unresponsive for a few seconds or even require a force quit.

I am somehow unable to reproduce that. Could you maybe send me the train configuration where you experience that issue?

[asdf1280]

04:43:45             :Master Controller:61 ( 61% Brake)
04:43:45             :Master Controller:move from -61 to -58
04:43:46             :1. input:Master Controller 57% Brake
04:43:46             :Master Controller:Found Indicator (Master Controller)
04:43:46             :Master Controller:Is braking area [Brake]
04:43:46             :Master Controller:57 ( 57% Brake)
04:43:46             :Master Controller:No movement because of tolerance of +-1
04:43:46             :1. input:Master Controller §5% Brake
04:43:46             :Master Controller:Found Indicator (Master Controller)
04:43:46             :Master Controller:Is braking area [Brake]
04:43:46    -Error-  :Could not get a number out of (removing percent method)  §5
04:43:46             :Master Controller:5 ( §5% Brake)
04:43:46             :Master Controller:move from -5 to -58
04:43:46             :1. input:Master Controller 100% Brake
04:43:46             :Master Controller:Found Indicator (Master Controller)
04:43:46             :Master Controller:Is braking area [Brake]
04:43:46             :Master Controller:100 ( 100% Brake)
04:43:46             :Master Controller:move from -100 to -58
04:43:47             :Master Controller:move from -58 to -50
04:43:47             :1. input:Master Controller 76% Brake

Hello. The issue seems to be because of text recognition. It looks like sometimes '9' is detected as '§', and it caused extra unexpected input. Maybe consider adding a replacement source code.

Another alternative I recommend is using Tesseract.NET v5 and tessdata_best for OCR. In my autopilot software, this way, 80% of detection issues were resolved. The only issues left were missing detection of '.(dots)' in numbers, however this is not the case for TSW2_Controller.

Also, tesseract model in Deutsch is likely better at number recognition than English model.

[DerJantob]

Oh, I completely forgot that updates for NuGet's exist haha. I will also test tessdata_best in the next days and see how it preforms.

But back to your problem. The error in the log file is very likely not the problem with the freezing. The Error just says that the program could not read the number in the first try. As you can see in the time stamps, before the error it says 04:43:46 and after the error it still says 04:43:46, so no delay there. If you go to "C:\Users\username\AppData\Local\TSW2_Controller\TrainConfigs" you should find your configurations. Maybe I can help you if I test your configuration. Just don't forget to tell me on which train you are having the problem with :)

[asdf1280]

It's hard to reproduce that exact same issue. And while trying to reproduce that, I found the issue above. This probably resulted in some kinda freeze before in the worst case scenario... I think so because the detection issue like the one above causes my throttle lever to go all the way to 100% and return, again and again. Changing OCR engine definitely helps.

[DerJantob]

Ok now I get it. So the program detected §5% instead of 57% So the program then thinks it is in the wrong position and moves the lever to 100% to compensate for the wrong detection. Maybe I could try to force the program to get the same reading 3 times in a row to increase accuracy. And of course, as you've said, improving OCR should help as well.

[asdf1280]

Actually, OCR is likely to produce same result for same number. So that could end up with performance issues. Rather, one more method can be vertically extending the image to scan. For example, when generating an image for scanning, giving padding to text seemed to work for me.

[DerJantob]

I have updated Tesseract and am now using tessdata_best, and I have to say that it significantly improved things. Although just updating made it far worse than it was before, but after using tessdata_best, the common issue with 1A being read as 14 has been resolved. Thank you for that! I have also made some changes in the Screenshot function. That also improved things for me. If you want, you can try it out and see if this resolves your issue. TSW2_Controller_PreRelease_1.zip

Here is the part inside the Screenshot function that I have changed, if you want to import it into your AP.

for (int y = 0; (y <= (bmpScreenshot.Height - 1)); y++)
{
    for (int x = 0; (x <= (bmpScreenshot.Width - 1)); x++)
    {
        Color inv = bmpScreenshot.GetPixel(x, y);

        //Farbanpassung um möglichst nur die Schrift zu erkennen
        //          Helligkeit          ||                      RGB-Werte alle fast gleich
        if (inv.R + inv.G + inv.G < 500 || !(Math.Abs(inv.R - inv.G) <=2 && Math.Abs(inv.G - inv.B) <= 2))
        {
            inv = Color.FromArgb(0, 0, 0, 0);
        }

        if (inv.R > 0) { inv = Color.FromArgb(255, 255, 255); };

        //invertier das Bild
        inv = Color.FromArgb(255, (255-inv.R), (255-inv.G), (255-inv.B));
        bmpScreenshot.SetPixel(x, y, inv);
    }
}

[asdf1280]

Honestly, I'm a bit concerned of many Color.FromArgb calls that can waste memory. You may want to see my code for autopilot distance reading.

        static void OptimiseForOCR(Bitmap image, int maxColor) {
            for (var y = 0; y < image.Height; y++) {
                for (var x = 0; x < image.Width; x++) {
                    Color inv = image.GetPixel(x, y);

                    if (inv.R < maxColor || inv.G < maxColor || inv.B < maxColor) {
                        image.SetPixel(x, y, Color.White);
                        continue;
                    }

                    image.SetPixel(x, y, Color.FromArgb(inv.ToArgb() ^ 0xffffff));
                }
            }
        }

if (inv.R < maxColor || inv.G < maxColor || inv.B < maxColor) { should be different because my code is only used to processing fully white texts.

[asdf1280]

For your function, I'd change like this.

for (int y = 0; (y < bmpScreenshot.Height); y++)
{
    for (int x = 0; (x < bmpScreenshot.Width; x++)
    {
        Color inv = bmpScreenshot.GetPixel(x, y);

        //Farbanpassung um möglichst nur die Schrift zu erkennen
        //          Helligkeit          ||                      RGB-Werte alle fast gleich
        if (inv.R + inv.G + inv.G < 500 || !(Math.Abs(inv.R - inv.G) <=2 && Math.Abs(inv.G - inv.B) <= 2))
        {
            bmpScreenshot.SetPixel(x, y, Color.White);
            continue;
        }

        // if (inv.R > 0) { inv = Color.FromArgb(255, 255, 255); }; is this needed?

        //invertier das Bild
        bmpScreenshot.SetPixel(x, y, Color.FromArgb(inv.ToArgb() ^ 0xffffff));
    }
}

[DerJantob]

many Color.FromArgb calls can waste memory

Oh didn't know about that.

// if (inv.R > 0) { inv = Color.FromArgb(255, 255, 255); }; is this needed?

Nope, you are right. Noticed it myself, but you responded very quickly :)

Quick question, wouldn't

bmpScreenshot.SetPixel(x, y, Color.Black);

be better than

bmpScreenshot.SetPixel(x, y, Color.FromArgb(inv.ToArgb() ^ 0xffffff));

[asdf1280]

Tesseract worked better for me when it had some gray colors around text like anti-aliasing.

[DerJantob]

But 0xffffff is also white and not gray, or did I misunderstand your point?

[asdf1280]

It's a code used to invert the color. You can search for bitwise operators for more information regarding this.

[DerJantob]

Alright, I think this is a bit advanced for me ^^ But I don't think that color-inverting is necessary anymore. With this:

(Math.Abs(inv.R - inv.G) <=2 && Math.Abs(inv.G - inv.B) <= 2)

I have made sure that the RGB-values are almost the same. This changed screenshots looking from this

to this

So I think then forcing the remaining pixels to be black increases contrast and makes it easier to read for OCR. That changed it to this:

Do you know what I mean?

[asdf1280]

I understand. However inversion really worked better for me, despite of lower contrast. You should try them both, though. Also, in the image you posted, 'M' in the second line seems to be partly erased out, so you should loose the filter a bit to prevent errors.

I have made sure that the RGB-values are almost the same. This changed screenshots looking from this

This is quite new to me. For my software, the scanning area is always covered with dark background, so I never had to cope with this.

[DerJantob]

This is quite new to me. For my software, the scanning area is always covered with dark background, so I never had to cope with this.

This is just happening if you are moving the camera around or if you are going into the external view. Maybe I have to fiddle around with the values a bit more.

[asdf1280]

I think trying to find that gray color in text indicator and erasing everything to the left can work too. I use similar method to accurately detect the numbers in my system. When reading kilometres to the next station, the OCR sometimes doesn't detect '.' dots. For exmaple, 1.5km can be 15km from time to time. I have made a fix by looking for a single dot at specific Y axis in the image.

[asdf1280]

                {
                    var dotY = spdLimDistanceBounds.Height - 6;
                    var dCnt = 0;
                    for (var x = 0; x < bitmap.Width * 0.6; x++) {
                        var i = bitmap.GetPixel(x, dotY);
                        var i2 = bitmap.GetPixel(x, dotY - 3); // 1
                        var i3 = bitmap.GetPixel(x, dotY - 8); // 7
                        if (ColorDist(i, Color.White) > 200) {
                            dCnt += 1;
                        } else {
                            if (dCnt > 0)
                                if (dCnt > 2 && dCnt < 6 && ColorDist(i2, Color.Black) > 200 && ColorDist(i3, Color.Black) > 200) {
                                    dotExists = true;
                                }
                            dCnt = 0;
                        }
                    }
                }

This is my code. It involves some complex logic that helps to only detect '.' but not thin numbers like 1 and 7.

``

This code can completely fix the OCR detecting 2.5 km as 25 km.

Similarly, I think your software can scan the image from x=0 and find that gray colour, then erase everything to the left.

But sometimes, the color in the game can be similar to gray. This is dangerous.

[DerJantob]

The problem is that the gray box fades into other colors, so I think I can't reliably detect the end of the box (e.g. when playing at night). I thought that I could use the white stripe below, but I just noticed that the distance of that white stripe has changed in TSW3 TSW2:

[asdf1280]

One dumb logic can be the 'or' logic. We can scan from the highest x point toward the left, and repeat while there's any white pixel in +2px or +n px(for TSW3) below. Actually, I begin to think that this could be the best.

[asdf1280]

I think your code is the safest form. Even this code can be broken in white background of cockpit, although that wouldn't greatly affect text detection.

[DerJantob]

I don't know, I think that would not be so performant in the long run. The program already deletes everything left of the text indicator (e.g. throttle). So "§8 adgh Throttle 23%" gets converted to "Throttle 23%".

[asdf1280]

Agreed. The new methods I tried to introduce involve more risk.

[DerJantob]

Maybe we should focus on the problem with the false Reading 04:43:46 -Error- :Could not get a number out of (removing percent method) §5

I am wondering how it could detect §5% instead if 57%. It doesn't make sense, because the 7 is in between the 5 and the 5

[asdf1280]

It was 95%. I think there was a misunderstanding. The throttle rised rapidly between two scans. Fixing the OCR will fix the issue.

[DerJantob]

Ok that makes sense again haha

[asdf1280]

If we plan to only detect numbers, we can greatly improve the performance by running engine.SetVariable("tessedit_char_whitelist", "0123456789%");. If you can separate the indicator reading and number reading, this can prevent some fake character recognition too. Looks impossible, though.

[DerJantob]

That's an interesting Idea and could really improve things. I think there was a method with tesseract to get the coordinates of Text. Maybe with this I can split the image. But of course that would take a bit until I have implemented that

[asdf1280]

Yeah you can just keep it in mind.

[DerJantob]

I will probably take a closer look a little later today

[DerJantob]

I did some testing and this is what ive got so far: For me "Leistungsregler 71%" is very often getting read as 1%, so I have done the Testing with 71% Also, using 1920x1080 is more difficult for tesseract, so I also changed my resolution.

• engine.SetVariable("tessedit_char_whitelist", "0123456789%"); does not help, because tesseract detects the characters as usual and just ignores characters that are not inside the whitelist. So §5 would still get detected as 5.
• If a Number got read incorrectly, then stretching the image horizontally did sometimes get the correct result, but I didn't find a good way to decide when to use the stretched image. Just using it every time was more inaccurate, and reading the confidence-level of Tesseract also didn't help much, because it was often very confident in false readings.
• Denoising or resizing the Image didn't help
• And I think I know what caused your freezing. During testing, I noticed that extremely rarely the program reads a number that is way too high, for Example I think 96% got read as 9606, so the program thinks that the lever is at 5606% and wants to move it back to 56%. And that takes time. Maybe I can limit the scanned number for continuously variable levers, but I don't know if that breaks someone's configuration.

But what did improve the results is by changing the Screenshot function to this:

if (inv.R + inv.G + inv.G < 400 || !(Math.Abs(inv.R - inv.G) <= 15 && Math.Abs(inv.G - inv.B) <= 15))
{
    bmpScreenshot.SetPixel(x, y, Color.Gray);
    continue;
}

//invertier das Bild
bmpScreenshot.SetPixel(x, y, Color.FromArgb(inv.ToArgb() ^ 0xffffff));

Using gray instead of white seems to work better, but I don't know why. I also expanded the range of allowed colors, but that only improved it a small amount. Maybe you could test it on your end and see if that helps.

Regards Jannik

[DerJantob]

Did it improve anything?