Add glyphs for empty unicode characters used by bots

[rbjaxter]

Cheaters in TF2 commonly use certain glyphs that don't appear when vote-kicking them, causing confusion when they match the name of someone else in the server.

The above table shows the most commonly used unicode characters by bots/cheaters/banned accounts, and was pulled from pazer's tf2 bot detector: https://github.com/PazerOP/tf2_bot_detector

Editing fonts is by no means an area I'm very familiar with, but I believe that I could search for these specific characters and add in some sort of glyph to indicate when a name is using a hidden/invisible character.

[rbjaxter]

First attempt at doing this did not succeed, sadly. Modifying the font unicode characters and then setting the range did not seem to do the trick as I was anticipating.

[andy013]

Hey, I already did this and created a little mod you might be interested in here: https://github.com/andy013/votehud_custom_font

If you would rather create your own font I could probably help you if you have any issues getting it to work. There's a bunch of hacks you need to do to get certain characters to display.

[rbjaxter]

Hey, thanks for the comment!

I actually started exploring this idea because someone in my Discord had mentioned what you had made.

I wanted to try and do it on my own just to get a better grasp of how everything functions, but I've seemed to hit a bit of a wall when it comes to the proper generation of the font. I understand everything else relating to this UI customization, but I'm personally god awful when it comes to working on and generating fonts that I've modified.

After trying on my own, I also tried to reverse-engineer yours a bit (replacing your font with mine and adjusting font definitions accordingly), and it does in fact seem to be an issue with the font I'm generating. When I first tried this on my own, I tried replacing every empty glyph but kept getting an sfnt 65535 glyph limit error (and the font file size was 11x as big once I did get the font generated). Instead, I've now just been trying to replace the most commonly used symbols from pazer's bot detector playerlist instead, but am still having issues sadly.

While I was trying to get mine to work, I was testing it with U+200F but did not seem to have any luck. Help with font generation would be greatly appreciated :)

[andy013]

The Windows API that the source engine uses will replace some characters with other glyphs depending on a few things. Unfortunately it will replace some characters with the space (U+0020) glyph. U+200F is one of those characters. Since it isn't supposed to display and many fonts don't have a glyph for it, I guess Windows just tries to display it correctly by using the space glyph in it's place.

The way I was able to get around this is a bit of a hack. In the source engine you can set the range for a custom font and the engine will splice your font together with Tahoma for any characters outside the range. I replaced the space (U+0020) character in my font with an error box, but then set the range of my custom font to be 0x0021 0xFFFF. That way Tahoma is used for U+0020 (so spaces display normally) but when other characters (e.g U+200F) get replaced by the space, it uses the error box from my font.

If you don't want to use this hack then the only alternative would be to replace the space with an underscore or something similar so that when U+200F and other characters are replaced with it, you will be able to see them. I tried this before I found the hack but it looked a little ugly with underscores all over the place.

For other characters the Windows API will fall back to another system font if it detects your font doesn't supports that script. The way it detects this is by looking at the fonts GSUB table. If it doesn't include certain entries then it will just fall back. I figured out, through a bunch of trial and error, what entries were needed for the entire range of glyphs to be used.

You can use fontforge to copy the entries from my font over to your font. Open both fonts in fontforge and on the window with your font open go to Element > Font Info > Lookups > Import then select all the entries from my font. After doing this you should see your replacement glyphs for certain languages working instead of falling back to other system fonts (If I remember correctly, this was important for Arabic and several other scripts).

Hopefully this should help you. When making my font I ended up having to write my own source mod so that I could print out all the characters on the screen and see how they displayed. Something else you should be aware of is that for some reason fonts display differently in Labels vs RichText VGUI elements. I don't know why but the space hack that I mentioned above only works in Labels and wont work in RichText areas like the chat box. The engine seems to draw spaces differently in RichText areas so a bunch of characters will always be invisible.

I've had that glyph limit error as well. I think I just remade the font and was careful to detach and remove any glyphs I wasn't using. Something you should know is that if you only replace those common characters that you have listed then it would be pretty easy for the bots to just move to other characters. There are loads of invisible characters that can be used in the source engine. As well as many non-spacing characters that will draw directly behind others and can be very difficult to see. I tried to replace the vast majority of characters in my mod to pre-emptively counter the bots moving to a different character and becoming indistinguishable again.

[rbjaxter]

This is some great information, thank you for going into such detail. I knew all of this was probably outside of my wheelhouse, but this shows that it's much more than I originally thought it to be hahaha.

I had tried setting the range as well as merging fonts prior to this but did not have luck. Given what you've said, I'll try again and see if I can get it working; I almost certainly did not do it right the first time.

Edit: I see what you mean about the lookup tables now, interesting. Definitely not something I had touched before.

[rbjaxter]

So I've imported your lookup tables, set the range, and added a test glyph (this lovable guy already in my font ) for U+0020 as you recommended as well as U+200F and continued to test with the U+200F character in my name (while calling votes).

Spaces now appear to be behaving differently, which presumably means it's properly falling back to Tacoma:

...but as far as that glyph goes, it still does not appear to show up in the vote UI:

Here are the two related files as well (my hud uses a ton of #base includes so the votehud is pretty cut down):

I feel like I'm understanding things better than I was before, but I feel I must still not be grasping something specific to do with the glyphs. Do you have any ideas?

Also, out of curiosity, is there an easier way to replace all empty fontforge glyphs than just manually doing it? I suppose I could just go through selecting all used glyphs and then inverting the selection, but this font (Lato) seems to have a spattering of glyphs throughout.

[andy013]

Could you upload the font you have edited so I can try it?

I don't think you even need a glyph set for 200F, it doesn't make any difference since it's unused anyway. Is it possible 200F gets removed from your name if you set it through Steam? When I was testing my font I always used bots with the command bot -team blue -name "<paste unicode characters here>" to test the characters.

I don't know if there is an easier way to select all empty glyphs in fontforge. I just started using it to create this mod. I did find that if you want to paste a large range, it does it quicker if the selection is off screen for some reason. You can also go to Encoding > Compact to only show characters with glyphs defined, but unfortunately I don't think selections stay when you change it back. Also there was a little script someone posted here that I used a few times to select a large range of empty glyphs: https://github.com/fontforge/fontforge/issues/2621#issuecomment-638528917

[rbjaxter]

Hmm, I just tried spawning in a bot with the character and didn't have luck.

I haven't yet tried replacing all of the unused glyphs, though; just space and U+200F. Here's the font: font-lato-semibold.zip

[andy013]

Hmm, it must be something to do with your HUD files. Are you using another version of the same font in the HUD? Maybe you can't have 2 fonts with the same name. Or maybe you are using the original version for bh_Font10? Or maybe you have this font installed on your system at C:/Windows/Fonts and that might be taking priority?

I just replaced the font in my mod with this and it is working as expected for 200F.

Also, I think what I said before about not needing a glyph for 200F is wrong. I think the source engine actually uses the width of 200F even though it gets replaced. I don't think this mattered in my font because it was monospaced but it might cause characters to get cut off if the width of the glyph for that code point is thinner than it's replacement. It's probably safe just to do what you have done and set the same glyph for both 200F and it's replacement 0020.

Lastly, you should know that replacing space like this will break a lot of other character replacements. For example, if you try and type a Chinese character that isn't included in your font, normally it will print a space in your font and then the character glyph from another font. If you replace space then it will use the replacement and the character will be unreadable. This is why I ended up printing the names of players twice in my mod in 2 different fonts, so that hopefully you could see what the character was supposed to be.

[rbjaxter]

Ah, you know what...it's because I had the old font installed on my OS. once I reinstalled it, it now appears to be working correctly as far as I can tell. doh

[andy013]

Haha, I just thought of that and edited my reply. Glad you figured it out 👍

It's probably best just to name the font something unique so that you don't have any conflicts with system fonts. You need to import it in the clientscheme file anyway for the range hack to work.

[rbjaxter]

Thanks again for all of the help :) I still need to mess around with the votehud UI and english tokens. hoping I won't have to use tokens to edit the size of the box, but given it's what you did I imagine that's the course of action needed. I had to do something similar to fix the map stamp bug issue in the store before it was fixed...