Closed anjackson closed 1 year ago
As per RFC 4180:
Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes.
Examples how it should look (as per the CSV spec): "Anglo Nubian Goat Society, The" Bat Ecology and Bioacoustics Lab
Alternatively, escape function seems to work fine
Actually not sure if that's the same issue, but expectation was to fix the bug there the title (if it has comma) been splited and drifted into other column
testing: https://dev.webarchive.org.uk/act/targets/list, looking at "100 Best Sites" collection:
Original csv export:
<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
nid | title | field_url | author | field_crawl_frequency | created | | -- | -- | -- | -- | -- | -- | -- | -- 2902 | 38 Degrees | https://speakout.38degrees.org.uk/ | https://you.38degrees.org.uk/ | http://www.38degrees.org.uk/ | Nicola Bingham | QUARTERLY | 2013-07-22T15:02:52Z 2963 | Air BnB | https://www.airbnb.co.uk/ | https://www.airbnb.co.uk/sitemaps | Nicola Bingham | MONTHLY | 2013-07-24T13:30:41Z 2984 | Amazon | https://www.amazon.co.uk/ | https://www.amazon.co.uk/gp/site-directory?ie=UTF8&ref_=nav_shopall_fullstore | Nicola Bingham | DAILY | 2013-07-25T13:02:14Z 2904 | Anarchist Federation | http://www.afed.org.uk/ | Nicola Bingham | SIXMONTHLY | 2013-07-22T15:15:47Z 2983 | Argos.co.uk | http://www.argos.co.uk | Nicola Bingham | WEEKLY | 2013-07-25T12:57:13Z 2974 | Artsthread | http://www.artsthread.com/ | Nicola Bingham | QUARTERLY | 2013-07-25T09:30:46Z
The current export code writes out CSV, but does not handle escaping of commas at all! To fix this properly, a CSV writing library should be used, and all the CSV export code re-written.
Alternatively, while the export code is not well-written in general, it does at least appear to use a specific variable for the CSV SEPARATOR, rather than having commas everywhere:
https://github.com/ukwa/w3act/blob/6ad35fff95854edd847c279904534938ff73295d/app/uk/bl/Const.java#L358-L359
As an interim measure, we could just replace that with a
\t
tab. That should fix all CSV outputs immediately, as long as everyone who uses those outputs is aware of the change.