Data importers fail to read UTF-8 BOM encoded files

USEPA / EPA_MOVES_Model

Estimating emissions for mobile sources

Other

80 stars 20 forks source link

Data importers fail to read UTF-8 BOM encoded files #39

Closed danielbizercox closed 2 years ago

danielbizercox commented 2 years ago

When importing files that are UTF-8 BOM encoded, MOVES silently fails. If using the GUI (e.g., County Data Manager) you will get an error about a missing column, even though the column is present in the file. If running on the command line (i.e., using the XML importer feature), there will be no error message.

In either case, the table is not imported into the input database.

danielbizercox commented 2 years ago

Modelers will encounter the UTF-8 BOM encoding issue if they save their input file in Excel using the "CSV UTF-8 (comma delimited) (*.csv)" file format.

This issue will be fixed in MOVES3.0.3. Until then, modelers can export to the "CSV (Comma delimited) (*.csv)" format or save using the .XLSX or .XLS format to avoid this issue.