Closed fbergmann closed 2 years ago
Thanks for reporting, Frank. Will fix that.
Hm, seems to work on my machine. According to the pandas docs, the default for writing is already utf-8 (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html). I do get the same error if I explicitly request cp1252.
Is the problem solved for you with df.to_csv(..., encoding="utf-8")
? Maybe the pandas documentation is wrong there...
the issue is that different operating systems use different default encodings. and there is a certain os that is widely used that defaults to cp1252. I'll try running with the different encoding to see what happens.
Actually just adding the encoding did not work ... turns out encoding is only supported if we specify filenames and write in binary mode, so:
df.to_csv(filename, sep='\t', index=True, encoding="utf-8")
works
with open(filename, 'w') as fh:
df.to_csv(fh, sep='\t', index=True, encoding="utf-8")
does not. But i dont see why we need to manually open the file, rather than just let to_csv does the work.
the issue is that different operating systems use different default encodings. and there is a certain os that is widely used that defaults to cp1252. I'll try running with the different encoding to see what happens.
Sure, we should fix that for windows. Only I am not sure what exactly is the problem. At least for the most recent pandas, the code seemed to agree with the docs in that the default encoding is utf-8, also on Windows. (Which pandas did you use?) I can't really test Windows things here, so we go for whatever you confirm me to solve the problem.
I think the problem is that we manually open the file in 'w' mode, and the encoding argument (which defaults to utf-8) only works on binary output. So this would work too:
df.to_csv(filename, sep='\t', index=True)
as would:
with open(filename, 'wb') as fh:
df.to_csv(fh, sep='\t', index=True)
If after reading the Sneyd_PNAS2002 model from the benchmark collection, i write it out again using the petab library, i get the following exception:
it turns out the unit is given in special characters. Since PEtab had no problems reading it, maybe the writers should pass a utf8 encoding attribute when writing out the tables. Rather than just: