search

PE5PVB / TEF6686_ESP32

Advanced arduino based firmware for TEF6686 with ESP32
GNU General Public License v3.0
77 stars 36 forks source link

Support for German Ö and Ä in RT #10

Closed andimik closed 1 year ago

andimik commented 1 year ago

In case there is a special character in the Radiotext, this will be skipped (even for the following ones)

grafik grafik

The interesting thing is, that the TEF shows the character ü but not the other ones like ä.

Nächste Sendung: Journal um fünf gets Nste Sendung: Journal um fünf

Im Gespräch mit der gets Im Gespr mit der

IMG_20230609_163054 IMG_20230609_163256

PE5PVB commented 1 year ago

Thanks for the feedback. I'm currently working on the EBU characterset but wan't able to test it. This will be fixed soon.

andimik commented 1 year ago

17fe6d6 is a good start.

Anyway, nearly all Radiotext characters with Umlaut are cropped, only ä is replaced by a placeholder now.

RDSSpy has logged that (BER = 0)

00:41:54 dem genialen Jazzgitarristen George Benson                      
00:42:24 Ö1 Service: Tel. (01) 501 70 371 (Mo-Fr, 8-21 Uhr)              
00:42:54 Nächste Sendung: Nachrichten                                    
00:43:24 Fragen zum Ö1 Programm? Tel. (01) 501 70 371                    
00:43:55 Jetzt in Ö1: George Benson: Das Timing unbewusster Grazie       
00:44:24 (Fortsetzung). Die Spielräume Nachtausgabe verbeugt sich vor    
00:44:55 dem genialen Jazzgitarristen George Benson                      
00:45:25 Ö1 im Internet: oe1.ORF.at                                      
00:45:54 Nächste Sendung: Nachrichten                                    
00:46:25 Ö1 Club: (01) 501 70 370 (Mo-Fr, 9-17 Uhr)                      
00:46:54 Jetzt in Ö1: George Benson: Das Timing unbewusster Grazie

IMG_20230610_004604 IMG_20230610_004537 IMG_20230610_004457 IMG_20230610_004406 IMG_20230610_004333 IMG_20230610_004303 IMG_20230610_004053 IMG_20230610_004021

andimik commented 1 year ago

Good news: I found the bug in https://github.com/PE5PVB/TEF6686_ESP32/commit/17fe6d6a89bb38f03073d596268e5aebcc6d87cb

You were using wrong hex numbers.

For example capital Ö (like for Österreich) is defined in https://github.com/PE5PVB/TEF6686_ESP32/blob/17fe6d6a89bb38f03073d596268e5aebcc6d87cb/src/TEF6686.cpp#L528

but it should be

    case 0xd7: return 0x4f; break;  // Ö   Ö

grafik (Source EN 50067)

andimik commented 1 year ago

IMG_20230610_024420

PE5PVB commented 1 year ago

Did you change and test it?

andimik commented 1 year ago

Yes, for this screenshot I've changed it, compiled it and uploaded it to the TEF.

According to the RDS specification the following characters are allowed (starting with 0x30):

0123456789:;<=>?
@ABCDEFGHIJKLMNO
PQRSTUVWXYZ[\]―_
‖abcdefghijklmno
pqrstuvwxyz{|}¯ 
áàéèíìóòúùÑÇŞβ¡IJ
âäêëîïôöûüñçşǧıij
ªα©‰Ǧěňőπ€£$←↑→↓
º¹²³±İńűµ¿÷°¼½¾§
ÁÀÉÈÍÌÓÒÚÙŘČŠŽÐĿ
ÂÄÊËÎÏÔÖÛÜřčšžđŀ
ÃÅÆŒŷÝÕØÞŊŔĆŚŹŦð
ãåæœŵýõøþŋŕćśźŧ

(many thanks to https://github.com/windytan/redsea/blob/b34fd7ce72516d94407957a7fdb3e0e8eef751df/src/rdsstring.cc#L34 )

grafik

see attached spreadsheet: rds_codepages.zip

For example, Ö is 0xD7 in RDS, but 0xD6 in utf8, see https://www.utf8-zeichentabelle.de/unicode-utf8-table.pl?number=1024

Could you tell me which code page for Aura2CondensedPro is used on the TEF screen? Then I can map it and send you a pull request if you want.

The font works (at least) with umlauts, see ü I got last time. So I think we don't need to convert it to another character.

#define FONT24 &Aura2CondensedPro_Regular24pt7b
#define FONT14 &Aura2CondensedPro_Regular14pt8b
#define FONT7  &Aura2CondensedPro_Light7pt8b

Because I tried the following without success ...

case 0xd7: return 0xC396; break; // &Ouml; Ö and case 0xd7: return 0xd6; break; // &Ouml; Ö and case 0xd7: return 0x96; break; // &Ouml; Ö

IMG_20230610_105900 (sorry for the bad quality, it should be Nächste Sendung: Tipps für Ö1 Club-Mitglieder in German)

PE5PVB commented 1 year ago

It's not that easy. RDS comes as UTF8, but the font uses extended ascii. Currently I masked the öïëä as from your table, but it should be nice to have a MPX recording of that Austrian station. Please make a recording via a virtual audio cable from your SDR of about 10 minutes in WAV at 192kHz and post it via wetransfer.

andimik commented 1 year ago

Tomorrow I will add all other characters and test with Austrian and Slovene stations here. Maybe I find Italian channels with special characters, too.

It's a good opportunity as I am living next to two borders...

PE5PVB commented 1 year ago

Well, if you can help me with MPX recordings you are very welcome. In that way I can easily debug.

andimik commented 1 year ago

Sure, I will send you the link to the recordings then. Any specific format? I suppose 192 kHz 16bit mono PCM.

PE5PVB commented 1 year ago

Yes, correct, uncompressed WAV. Ofcourse the full audio range, so I can put it on my test transmitter ;-)

andimik commented 1 year ago

In the following MPX file (sampled in 192k, hopefully in mono) you will find the following radiotext messages:

Drei Kandidatinnen und Kandidaten aus ganz Österreich lösen
Bernhard Fellinger Rätsel aus den Bereichen Kultur, ...
Ö1 Club: (01) 501 70 370 (Mo-Fr, 9-17 Uhr)
Nächste Sendung: Nachrichten
Ö1 Literaturlisten: Tel. (01) 501 70 371
Jetzt in Ö1: Das Ö1 Quiz mit Bernhard Fellinger
Drei Kandidatinnen und Kandidaten aus ganz Österreich lösen
Bernhard Fellinger Rätsel aus den Bereichen Kultur, ...
E-Mail an Ö1: oe1.service@orf.at
Nächste Sendung: Nachrichten

You can read this with 

cat /tmp/a201_oe1_192k_mono.rec | redsea -E -r 192k -p | grep text | jq -r ".partial_radiotext"

The file has been created with rtl_fm under Ubuntu and a Realtek SDR dongle.

rtl_fm -M fm -l 0 -A std -p 0 -s 192k -g 30 -F 9 -f 92.8M | tee /tmp/a201_oe1_192k_mono.rec | redsea -E -r 192k -p | grep text

link to the file

https://we.tl/t-ZNVKooPIKt

PE5PVB commented 1 year ago

I can't play this file. I need .WAV format so I can playback it on my test transmitter.

andimik commented 1 year ago

Then you should convert it with sox into the right format or tell me the rtl_fm parameters.

If the format is correct, then just rename it to *.wav

PE5PVB commented 1 year ago

Usually I use a virtual audio cable and just record it with Audicity. I have no knowledge of Linux, sorry.

PE5PVB commented 1 year ago

If you can convert it, please do so.

andimik commented 1 year ago

Oh, did not know it.

The output should be signed 16 bit ints, so it's basically wav. Just try to rename it.

PE5PVB commented 1 year ago

Tried that, didn't work.

andimik commented 1 year ago

ok, let me boot into Windows :-)

PE5PVB commented 1 year ago

Conversion worked. I can now play the file. Thanks for your help

andimik commented 1 year ago

In the meanwhile I have searched for non-ASCII characters in Radiotext messages in my RDS-Spy-Archive (597 files from Austria, Hungary, Italy, Slovenia, Croatia) with 

ls -l *.spy | wc -l
597

cat *.spy | redsea -E -h | grep text | jq -r ".radiotext" | sort | uniq | grep --color='auto' -P '[^\x00-\x7F]'

and got some Radiotext messages which I did not recognize so far with RDSSpy (because RDSSpy replaces characters!)

Remark: Lines with reception errors (which produce special characters) have been filtered out of course.

16H-19H, C'EST L'HAPPY HOUR - FRANCE BLEU  ÔRIS
als 300 Seiten starke Regierungsprogramm von ÖVP und Grünen ...
ANTONIN DVOśÁK - GODALNI KVARTET ŠT. 12 V F-DURU, OP. 96 (B179),
A régió hangja!
Arsove spominčice
begibt sich auf die Spuren der tatarischen Identität, ihrer ...
Bernhard Fellinger Rätsel aus den Bereichen Kultur, ...
Bundesschulsprecherin Jennifer Uzodike über die neue Bildung ...
Cherubini: "Médée" / "Medea". Mit Rita Gorr, Guy Chauvet ...
Das Ö1 Tagesprogramm: (01) 501 70 371
Dirigent: Daniel Barenboim. Johann Strauβ: a) Ouvertüre zu "Die
Drei Kandidatinnen und Kandidaten aus ganz Österreich lösen mit
E-Mail an Ö1: oe1.service@orf.at
Filharmonija Janaček,  Dmitri Jurowski-dirigent - Simfonija št.
Fragen zum Ö1 Programm? Tel. (01) 501 70 371
Fuzzman - Hände weg von Allem
GERGELY RÓBERT - CSAK A SZÍVEM
Gremo v kino - ''V časih, ko ljudje mnenja izražajo glasno, skor
ICOTOVI ZOBI,  BIG BAND RTV SLOVENIJA - LEŽERNO JE MODERNO
INGRID SILIČ - 7 PRELUDIJEV ZA KLAVIR, OP. 11 - 1. ALLEGRO, 2. L
Inšpektor Jaka
Jetzt in Ö1: 5 Mio. Pesos Spezial: Die ...
Jetzt in Ö1: Anklang
Jetzt in Ö1: Das Magazin zum Monatsanfang
Jetzt in Ö1: Das Ö1 Konzert
Jetzt in Ö1: Die langsame Kochkunst des Schmorens
Jetzt in Ö1: Klimageschichte, Azteken, resiliente Kinder, ...
Jetzt in Ö1: Live von den Salzburger Festspielen - Wolfgang ...
Jetzt in Ö1: Mit.Schnitt
Jetzt in Ö1: Neujahrskonzert der Wiener Philharmoniker - Teil 2
Jetzt in Ö1: Ö1 Mittagsjournal
Jetzt österreichweit auch auf DAB+
Libanon+++Tönende Bilder+++Soko Döner+++Gender Equality im
Life Tirol - Der beste Mix für Tirol
"Lyrik zur Lage. Eine Rückschau, Teil 2". Es lesen Katharina
Másképpen hangzik!
mit Michael Blees.  Operettenmeister und ihre Ausflüge zur
Musik aus dem RadioKulturhaus. Schmusechor Gestaltung: Jörg Duit
Nächste Sendung: Ach, Sisi ++ Ljuba Arnautovic in Steyr ++ ...
Nächste Sendung: Die Woche im RadioKulturhaus
Nächste Sendung: Feiertagsjournal
Nächste Sendung: Jazz, Improvisation und Frauen im Musikbetrieb
Nächste Sendung: Nachrichten
Nächste Sendung: Nadia Boulanger / Raoul Pugno: "La Ville Morte"
Nächste Sendung: Ö1 Nachtjournal
Nächste Sendung: Orthodoxe Weihnachten, Diakone
Nächste Sendung: Selten wie ein Feiertag
Nächste Sendung: Tipps für Ö1 Club-Mitglieder
Neked szól!
NINA PUŠLAR,  ZLATKO - V ISKANJU SREČE
Ö1 Club: (01) 501 70 370 (Mo-Fr, 9-17 Uhr)
Ö1 im Internet: oe1.ORF.at
Ö1 Literaturlisten: Tel. (01) 501 70 371
Ö1 Service: Tel. (01) 501 70 371 (Mo-Fr, 8-21 Uhr)
Önnek szól!
Oper, zum Kunstlied und in die "seriöse" Instrumentalmusik.
PASSWORD - NICOLETTA DEPONTI, CECILIA SONGINI, NICCOLÒ GIUSTINI
Podobe znanja - Nanotehnologija odpira nove možnosti zdravljenja
Podobe zpominčice
Robert Unterköfler, Lukas Aichinger und Constanze Friedel live
Slávka Zámečníková,  Kate Lindsey,  Xavier Sabata,  Christina Bo
ŠUKAR - AMARO KHER (LIVE ST14 2010)
Več kot radio
Werke, die vor 150, 125, 100 und 75 Jahren zur Uraufführung

It seems that there are no Italian logs with special characters in my archive ...

andimik commented 1 year ago

In https://github.com/walczakp/rds-spy-logs I found:

Bäst musik just nu!
Das Ö1 Tagesprogramm: (01) 501 70 371
FONK! Det er lørdag
Förfest med Richard Herrey
HITRADIO VNıOCINA - RADIO KTERE HRAJE
Hörerservice: 0800-218 16 16
ICOTOVI ZOBI,  BIG BAND RTV SLOVENIJA - LEŽERNO JE MODERNO
Jetzt in Ö1: Live von den Salzburger Festspielen - Wolfgang ...
Klingan: Mixtejp med gäst-DJ
Kostenloses Hörertelefon: 0800 / 5900 111
Nächste Sendung: Tipps für Ö1 Club-Mitglieder
Næste: P3 Nyhede
Næste: P3 Nyheder
Næste: Radioavisen
Ö1 Service: Tel. (01) 501 70 371 (Mo-Fr, 8-21 Uhr)
P3 Musikdokumentär
Več kot radio

with 

/tmp/rds-spy-logs$ cat  */*.spy | redsea -E -h | grep text | jq -r ".radiotext" | sort | uniq | grep --color='auto' -P '[^\x00-\x7F]'
PE5PVB commented 1 year ago

We made a fontset based on the RDS charset you provided. But it seems not to work on all characters. Needs some more investigation, or a direct converter to UTF16. Will be investigated.

andimik commented 1 year ago

I got it working (at least for German characters) on the TEF, but not live with Austrian radio stations.

My script using https://github.com/ChristopheJacquet/PiFmRds/ added another byte (by mistake), and the second byte was correct (like ö) on the display.

I guess the actual problem is that UTF8 (which can be 2 byte chars or up to 3 byte chars like for EURO-sign €) needs to be converted into one hex number, see below.

codes 160-191 in ISO-8859-1 and Windows-1252 are two-byte characters in UTF-8 -- 0xC2 as a first byte, the second byte is identical to the extended ASCII-code.

andimik commented 1 year ago

Is that helpful?

https://playground.arduino.cc/Main/Utf8ascii/

PE5PVB commented 1 year ago

Can you check RDS again?

andimik commented 1 year ago

Wow! That's working now!

I can properly read the following radiotexts on the TEF, although the font does not display the Ö (capital O with umlaut) well. I guess it's a bit cropped as it looks more than a normal O. But please leave it so.

Več kot radio (on Radio Slovenija's Val 202)

Präsentation und Musikauswahl: Ursula Strubinsky dazw. 03:00 Ö1 Service: Tel. (01) 501 70 371 (Mo-Fr, 8-21 Uhr) Ö1 Literaturlisten: Tel. (01) 501 70 371 (all on OE 1 = Österreich 1)

HITRADIO Ö3 (on Ö3)

Therefore, I will close the issue.