Closed jondegenhardt closed 3 years ago
This has been partially addressed in PR #314. That PR takes care of most of the items described under approach 1, supporting Unix newlines. The longer term handling of Windows newlines is now being covered under issue #317.
PR #320 addresses the other significant issue to get to approach 1, detection of Window newlines.
This is a follow-on to issue #307, which identified issues with newline handling on Windows.
Note: Windows is not a currently supported platform. However, it is certainly desirable to have it run on Windows, making it worthwhile to identify issues needing to be addressed on Windows.
The general problem is that the different read and write mechanisms used in the different tsv-utils tools are not consistent with respect to handling of newlines on the Windows platform. In some cases files are read in binary mode, in other cases in text mode. Similar for writing. (Text mode engages newline translation on Windows in the low-level I/O routines.) At present the exact behavior of each tool is not known, this requires further evaluation.
There are a couple things that need to be done. One is to have complete test suite. The other is to choose newline policies. These would tested in the test suite.
Some possible newline handling policies:
The simplest policy to support would be to restrict the tools to Unix newlines. Require Unix newlines on input and output only Unix newlines. The existing test suite would largely support this. And, it would be the correct choice in many environments, especially in circumstances where a mix of Unix and Windows platforms are in use. That is, if data files are being shared, Unix newlines will normally be preferred.
Reading either form (option 3) might be easier done than expected, as most tools use
bufferedByLine
. In particular,bufferedByLine.front
handles newlines. However, a number of tools have their own reader functionality, so it would still be necessary to have a test suite for each tool.Full customization of newline handling has a material downside, in that it create additional user complexity in the form of additional command line arguments.