For now, there is no way for the user to manipulate the encoding when reading a file of a CSV and JSON DataNode.
Pandas and (almost) all excel-related libraries do not support explicit encoding types for parquet and excel files, so it is not possible in the near future.
Pickle Data node is not affected by the encoding, because Pickle is always binary file.
Description of the ideal solution
The user should be able to configure
What would that feature address
For now, there is no way for the user to manipulate the encoding when reading a file of a CSV and JSON DataNode.
Pandas and (almost) all excel-related libraries do not support explicit encoding types for parquet and excel files, so it is not possible in the near future. Pickle Data node is not affected by the encoding, because Pickle is always binary file.
Description of the ideal solution The user should be able to configure
and then, all reading/writing from the
json_dn_cfg
should use UTF-8 encoding.TODO: