search

WorldBank-Transport / ram-datapipeline

MIT License
3 stars 4 forks source link

Escape strings in the CSV file #6

Closed stvno closed 7 years ago

stvno commented 7 years ago

The generated CSV file doesn't escape the strings, which can produce weird results since the villages-file can contain commas, the current tool generates this: 三益村,BB80,CHN,China,Guizhou,Bijie,Weining Yi, Hui and Miao,21457,326,ChaHeXiang,66,26.62156404,103.80626496,3851.3,25.182612008186442

whereas it should escape records with commas eg: 三益村,BB80,CHN,China,Guizhou,Bijie,"Weining Yi, Hui and Miao",21457,326,ChaHeXiang,66,3825.6,3852.7,3838.6,9058.6,13182.4,0.0158746377848761,26.62156404,103.80626496

since Weining Yi, Hui and Miao is the name of the county

The problem is in the naive CSV creation here: https://github.com/WorldBank-Transport/rra-datapipeline/blob/develop/rra-analysis/app/index.js#L298

olafveerman commented 7 years ago

Thanks for reporting. That bit will probably change quite a bit over the coming weeks. Instead of storing it in CSV, we are thinking of storing the results in JSON, only to export them to CSV when the user wants to download results.

We'll keep this ticket open to have it on our radar.

danielfdsilva commented 7 years ago

Fixed in https://github.com/WorldBank-Transport/rra-datapipeline/blob/develop/rra-analysis/app/index.js#L332