jckantor / TCLab

Software support for the Temperature Control Laboratory.
Apache License 2.0
13 stars 22 forks source link

Historian CSV improvements #113

Open llimeht opened 1 year ago

llimeht commented 1 year ago

This patch adds three simple improvements to the to_csv() method of the Historian:

alchemyst commented 1 year ago

I have never encountered problems with csv.writer inserting "lots of blank lines" when opening a file the normal way. I wonder if there is some other reason for these blank lines that is machine or OS specific? If there appears to be a newline every second line, it might be a problem with whatever you are using to open the files interpreting \r\n as two new lines.

[Edit] I stand corrected, this is the recommended way to use csv.writer, from the docs:

If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use \r\n linendings on write an extra \r will be added. It should always be safe to specify newline='', since the csv module does its own (universal) newline handling.

llimeht commented 1 year ago

@alchemyst the failure is indeed OS-specific. It's fine on linux (and presumably macos) and the output file is an utter mess on windows.

On Linux:

$ ipython3
Python 3.10.8 (main, Nov  4 2022, 09:21:25) [GCC 12.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import csv

In [2]: with open("test.csv", "w") as fh:
   ...:     writer = csv.writer(fh)
   ...:     writer.writerow(['a', 'b'])
   ...:     writer.writerows([[1, 2], [3, 4], [5, 6]])
   ...: 

In [3]: with open("test.csv", "rb") as fh:
   ...:     print(repr(fh.read()))
   ...: 
b'a,b\r\n1,2\r\n3,4\r\n5,6\r\n'

But on windows:

> ipython3
Python 3.10.4 | packaged by conda-forge | (main, Mar 30 2022, 08:38:02) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import csv

In [2]: with open("test.csv", "w") as fh:
   ...:     writer = csv.writer(fh)
   ...:     writer.writerow(['a', 'b'])
   ...:     writer.writerows([[1, 2], [3, 4], [5, 6]])
   ...:

In [3]: with open("test.csv", "rb") as fh:
   ...:     print(repr(fh.read()))
   ...:
b'a,b\r\r\n1,2\r\r\n3,4\r\r\n5,6\r\r\n'

Changing the above to use using newline="" produces the same CRLF-terminated output on both platforms.. (I assume it's CRLF-terminated even on Linux because the default for the csv module is dialect="excel").