CSV -> XLSX conversion fails with an error

hasadna / datacity-k8s

1 stars 0 forks source link

CSV -> XLSX conversion fails with an error #230

Open niryariv opened 3 years ago

niryariv commented 3 years ago

Uploaded CSV file, views correctly here
Attempting to upload to datastore via admin UI fails with a UnicodeEncodeError message:

OriHoch commented 3 years ago

duplicate of #213 explanation in this comment: https://github.com/hasadna/datacity-k8s/issues/213#issuecomment-830843900

niryariv commented 3 years ago

@OriHoch Since this happens when we do the conversion (users are instructed to avoid uploading XLS files) and it seems to break on specific characters like " (other files with Hebrew headers convert ok) is it possible to add some code that removes these characters from the header row before starting the XLSX conversion? How much work do you estimate?

OriHoch commented 3 years ago

I don't see how it is related to XLSX conversion, this is the bug as I understand it:

reproduction steps

upload a csv file which contains special characters (like quotes) in a field name
check resource view (example)
check datastore upload log (example)

expected

resource view page: shows green data api button and data dictionary displayed below the table
upload log shows status completed

actual

resource view page: no green data api button, data dictionary is not displayed below the table
upload log shows status failed

OriHoch commented 3 years ago

I estimate 5-10 hours to fix the bug I described in previous comment