search

directus / directus

The flexible backend for all your projects 🐰 Turn your DB into a headless CMS, admin panels, or apps with a custom UI, instant APIs, auth & more.
https://directus.io
Other
27.21k stars 3.8k forks source link

No UTF-8 (special characters) on CSV export #16818

Open cs1m0n opened 1 year ago

cs1m0n commented 1 year ago

Describe the Bug

When exporting items from a collection (to a CSV file), special characters are replaced with weird characters - in my use case there are Norwegian specific characters.

I know CSV does not support UTF-8 characters by default but there is also a similar filetype called CSV UTF-8 (Comma delimited) which should support "special characters".

To Reproduce

Export an item from a collection which contains special characters in the field value.

Errors Shown

No response

What version of Directus are you using?

9.21.0

What version of Node.js are you using?

16.14.0

What database are you using?

Postgres 14.4.1

What browser are you using?

Chrome

How are you deploying Directus?

Docker

paescuj commented 1 year ago

Quick note: That would be the withBOM option in json2csv. See also https://juanjodiaz.github.io/json2csv/#/others/known-gotchas?id=unicode-support.

[...] there is also a similar file type called CSV UTF-8 [...]

What do you mean by that? I'm not aware of such a file type.

By the way: opening a CSV file containing UTF-8 characters with LibreOffice works out of the box. So this primarily seems to be a thing in Excel.

cs1m0n commented 1 year ago

From Microsoft Excel you can Save as CSV with 4 different filetypes:

  1. CSV UTF-8 (Comma delimited)
  2. CSV (Comma delimited)
  3. CSV (Macintosh)
  4. CSV (MS-DOS)

Where 1 & 2 seem to support special characters, while 3 & 4 don't.

rijkvanzanten commented 1 year ago

Linear: ENG-108

cs1m0n commented 1 year ago

This is actually a very easy fix. Check this: https://github.com/eligrey/FileSaver.js/issues/28

rijkvanzanten commented 1 year ago

This is actually a very easy fix. Check this: eligrey/FileSaver.js#28

Wanna open a PR?? 🙂 @cs1m0n

cs1m0n commented 1 year ago

16835 actually addresses this (as well as adding delimiter support) better than the fix I've suggested above since json2csv already has a withBOM option - and there's a PR open already.

Thank you for taking the time to look at this!

LevelUpGit commented 1 year ago

@cs1m0n We agree, it's a real issue both no UTF-8 support and fixed delimiter to comma.