Unicode right-to-left character encoding issue

[jacobthill]

Some Arabic strings have an unencoded character in them: e.g.

• https://dlmenetwork.org/ar/library/catalog/%5B'LeBAU_b12629492'%5D and
• https://dlmenetwork.org/ar/library/catalog/%5B'LeBAU_b12629492'%5D

\u200f is a non-printing right-to-left unicode character. It is in the original data but is not visible when rendered. We didn't have an issue with this before but with the new airflow process, this character isn't getting encoded. The issue is likely in intake. We need this character to render properly or we need to remove it. I'm not sure the implications of removing it but here is a way to do that:

https://stackoverflow.com/questions/46897952/remove-right-to-left-character-u200f-in-python-hebrew

[jacobthill]

I suspect this might be solved be retransforming the collect (now from json instead of csv) and refreshing the data.