Windows line endings read as data

[nickynicolson]

Data are read OK, but as the line ending '\r' is included as a component of the data value no extensions are found from the rows of the core data-file.

Data: {'http://rs.tdwg.org/dwc/terms/taxonID': 'urn:ipni.org:name:77126806-1\r'}
----------------------------------------------------------------------------^

[niconoe]

Thanks for your report, @nickynicolson!

Do you have a failing archive a hand so I can test/fix from it?

[niconoe]

Ok, I received the test archive and it is clearly invalid:

The data file contains a single data column, followed by \r\n. Given it's a single-column file, that can either be interpreted as:

• A field terminated by \r, followed by `\n' to mark the end of the line.
• A field with no separator (since it's the last on its line), with \r\n as a line separator.

However, the metafile mentions fieldsTerminatedBy="\t" linesTerminatedBy="\n", so the \r character is not removed.

If we fix the metafile to match the scenarios above ( fieldsTerminatedBy="\r" linesTerminatedBy="\n" or fieldsTerminatedBy="\t" linesTerminatedBy="\r\n"), the data appears correctly.

I'd be tempted to say "wontfix", since I can't see a way to support such invalid archives without breaking other stuff (dropping \r at end of fields without the metafile doesn't seems wise).

@nickynicolson, do you agree?