Format of CSV Import (Configuration of Delimiter / Separator)

[cakebake]

Describe the Bug

Unfortunately, it is not possible to import the data in a slightly different format. For example, it is not possible to use the column separator semicolon (;), which is common in Germany.

I am very impressed with the software and would love to use it for our customers.

Found this discussion: #12456

Am I missing an configuration? Once this is planned, can I help develop it with an MR?

Many Thanks.

To Reproduce

1. Create CSV file
2. Import Example

   "Col1";"Col2"
   "foo";"bar"

Errors Shown

CSV Format Error.

What version of Directus are you using?

9.18.1

What version of Node.js are you using?

Docker bundled version

What database are you using?

MySQL v8

What browser are you using?

Unimportant

How are you deploying Directus?

Docker

[br41nslug]

Currently the parsing of CSV is not configurable and will default to , as separator.

[cakebake]

@br41nslug Thank´s for the clarification.

Is there a configuration planned here? Any support welcome?

[br41nslug]

I am not aware of active/planned development on that specific feature but support is always welcome 😃 You may need be a little patient when it comes to getting it reviewed tho as there is currently a big pile of open PRs.

[cakebake]

You may need be a little patient when it comes to getting it reviewed tho as there is currently a big pile of open PRs.

This is not a problem. In the meantime I'll use Adminer or a side script if it gets too complicated for the user. :) Thanks.

[cakebake]

It's getting more complicated than I thought, but I'd like to use Directus.

I looked at the extension architecture and played around with hooks a bit. Is there a hook for the import? files.upload doesn't seem to respond. The init hooks are not accepted by the software (register is not a function / maybe a bug?) because I thought I might be able to intercept the API route.

With a hook - I thought - it might be possible to modify the CSV on the fly to match the formatting.

I guess that a separate module extension with its own import logic will actually be necessary? Of course it would be nice. As far as I know, there is no UI with column assignment (CSV cols > Collection cols) in this software segment and I've been looking for it for a long time.

[br41nslug]

Is there a hook for the import?

There is no hook to intercept uploads.

The init hooks are not accepted by the software (register is not a function / maybe a bug?)

That worked last time i checked, did something go wrong with compilation there perhaps? or forgot to extract the init function maybe.

I might be able to intercept the API route.

While this is possible with the init hooks i'd generally recommend adding a new endpoints over messing with existing ones in extensions or go with the import/export approach (upload it seperately and then use the UUID to read/process the file asynchronously)

This seems to be going far beyond the scope of fixing this issue tho 😬 I'd stick with first implementing an URL parameter for separator which gets passed to the csv parser https://docs.directus.io/reference/system/utilities.html#import-data-from-file purely at the API side before moving to UI implementations

[rijkvanzanten]

Linear: ENG-170

[miagg]

I think this is somehow critical, since in most of the Europe the default delimiter is ';' The only way to import is to open the csv on a text editor and do a search-and-replace.

I hope it gets implemented soon. 👍

[LevelUpGit]

I think this is somehow critical, since in most of the Europe the default delimiter is ';' The only way to import is to open the csv on a text editor and do a search-and-replace.

I hope it gets implemented soon. 👍

I agree, we hope also, since it's really needed.

[u12206050]

Could I create a PR that would auto detect at least between if , or ; separator is used?

[rijkvanzanten]

@u12206050 I'd say lets replace the CSV parsing dependency with https://www.papaparse.com/ which has native support for auto-detecting the delimiter 🙂

[u12206050]

Agreed, I myself use that as well in a custom extensions actually :D Is it just a matter of creating a PR for it, or do you have some process for community "tasks"?

[rijkvanzanten]

Is it just a matter of creating a PR for it, or do you have some process for community "tasks"?

Nope! This is an open Issue so PRs are very welcome 🙂