Support multi-character CSV delimiter

[maxcutler]

Python's csv module only supports single-character delimiters, while CP often delivers CSV files with multi-character delimiters (e.g., ,|||,).

Fortunately, the csv reader class(es) can ingest any iterator as the input data. So, we should be able to write a generator that takes the raw file as input and morphs each line to a standard comma-separated format (performing any necessary escaping too) and fed to the csv reader.

Once that's in place, it would be nice to add a command-line parameter to the script to allow users to specify the delimiter (defaulting to ,) instead of modifying the script itself. Or, if you're really ambitious, you could implement a Sniffer class that auto-detects the delimiter.

[ghost]

I'm having this issue with my CP migration. Should I tweak the script somehow, or should I replace ",|||," with something?

[danielbachhuber]

Ask them to give you a proper csv with commas as the delimiter

On Dec 9, 2011, at 18:18, Christopher Keelereply@reply.github.com wrote:

I'm having this issue with my CP migration. Should I tweak the script somehow, or should I replace ",|||," with something?

---

Reply to this email directly or view it on GitHub: https://github.com/danielbachhuber/CoPress-Convert/issues/5#issuecomment-3083446

[ghost]

That will be fun. It took us 3 months to get these. I'm trying to fiddle with the file to get it working as well as requesting new ones. I've changed the delimiter, but the script fails with _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? on line 679.

I imagine that there's a rouge quotation mark somewhere, do you think?

[danielbachhuber]

Not sure. I'm out of town until Tuesday, but feel free to ping me on Skype if you're still stuck "danielbachhuber"

On Dec 9, 2011, at 21:11, Christopher Keelereply@reply.github.com wrote:

That will be fun. It took us 3 months to get these. I'm trying to fiddle with the file to get it working as well as requesting new ones. I've changed the delimiter, but the script fails with _csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode? on line 679.

---

Reply to this email directly or view it on GitHub: https://github.com/danielbachhuber/CoPress-Convert/issues/5#issuecomment-3085987

[ghost]

Turns out the file we got from CP has quite a few formatting errors (at least in relationship to your script). I'm hacking together a piece of Ruby that generates a properly-formatted version. It's working for everything except the images, which'll take more reading through the Python to figure out. Thanks for your responsiveness! I'd be more than happy to post the Ruby somewhere when I'm done, although while it's pretty general it may be too specific to my needs for others to benefit much from it.

[danielbachhuber]

Totally, that would be awesome. Best of luck

On Dec 11, 2011, at 5:20, Christopher Keelereply@reply.github.com wrote:

Turns out the file we got from CP has quite a few formatting errors (at least in relationship to your script). I'm hacking together a piece of Ruby that generates a properly-formatted version. It's working for everything except the images, which'll take more reading through the Python to figure out. Thanks for your responsiveness! I'd be more than happy to post the Ruby somewhere when I'm done, although while it's pretty general it may be too specific to my needs for others to benefit much from it.

---

Reply to this email directly or view it on GitHub: https://github.com/danielbachhuber/CoPress-Convert/issues/5#issuecomment-3095737

[ksong12]

Did anyone end up figuring this out? Going through the pain right now :(