InGen is a command line tool written on top of pandas and great_expectations to perform small scale data transformations and validations without writing code.
Describe the bug
I tried reading an XML file that is encoded in UTF-16 but ingen could not read it. It creashed with UnicodeDecodeError while trying to read using my mahcine's default encoding (utf-8).
To Reproduce
Steps to reproduce the behavior:
Create a XML file with UTF-16 encoding
Try to read it via InGen by adding it as a data source
See error:
Expected behavior
InGen should read UTF-16 files without crashing
Environment (please complete the following information):
Operating System: Mac
Additional context
We should add an option in the FileSource for users to declare the file encoding.
Describe the bug I tried reading an XML file that is encoded in UTF-16 but ingen could not read it. It creashed with UnicodeDecodeError while trying to read using my mahcine's default encoding (utf-8).
To Reproduce Steps to reproduce the behavior:
Expected behavior InGen should read UTF-16 files without crashing
Environment (please complete the following information):
Additional context We should add an option in the FileSource for users to declare the file encoding.