UTF-8 characters in pipelines get mangled sometimes

jakubklimek commented 6 years ago

After importing a pipeline, one character in the whole pipeline got changed:

jakubklimek commented 6 years ago

jakubklimek commented 5 years ago

RDFParseException : Unexpected character U+FFFD at index 53: https://data.gov.cz/slovník/podmínky-užití/autorské-d��lo [line 57]

jakubklimek commented 3 years ago

Putting this text in a text holder and saving the pipeline produces many instances of the error in the pipeline file on the disk:

klimek@xrg15:/data/lp/etl/storage/pipelines$ cat 1611239277139.trig | grep -n �
2715:ěščřžýáíé.žřýěščřžýá��é.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
3167:ěščřžýáíé.žřýěščřž��áíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
3619:ěščřžýáíé.žřýěšč��žýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
4071:ěščřžýáíé.žřýě��čřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
4523:ěščřžýáíé.žř��ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
4975:ěščřžýáíé.��řýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
10398:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáí��.žřýěščřžýáíé
10850:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžý��íé.žřýěščřžýáíé
11302:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščř��ýáíé.žřýěščřžýáíé
11754:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěš��řžýáíé.žřýěščřžýáíé
12206:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřý��ščřžýáíé.žřýěščřžýáíé
12658:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.ž��ýěščřžýáíé.žřýěščřžýáíé
16274:ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýá��é.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
16726:ěščřžýáíé.žřýěščřžýáíé.žřýěščřž��áíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
17178:ěščřžýáíé.žřýěščřžýáíé.žřýěšč��žýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
17630:ěščřžýáíé.žřýěščřžýáíé.žřýě��čřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
18082:ěščřžýáíé.žřýěščřžýáíé.žř��ěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé
18534:ěščřžýáíé.žřýěščřžýáíé.��řýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé.žřýěščřžýáíé

We suspect a Java stream being split in the middle of unicode character bytes and then these split bytes being replaced by �, which is a replacement character

linkedpipes / etl

UTF-8 characters in pipelines get mangled sometimes #544