Visualisation minion has problems with column headers

[t83714]

Visualisation Minion needs improvements:

Problem description

For this dataset: https://magda-dev.terria.io/dataset/ds-dga-26e2ebff-6cd5-4631-9653-18b56526e354/details?q=csv

https://dev.magda.io/dataset/ds-dga-26e2ebff-6cd5-4631-9653-18b56526e354/details?q=csv

Visualisation Sleuther parsed it as a one column CSV with only one column: id,resourceType,title,description,startDate,endDate,lat,lon,documentType,author,Evacuated,Homeless,Injuries,Deaths,Insured Cost,Train(s) damaged,Train(s) destroyed,Home(s) damaged,Home(s) destroyed,Building(s) damaged,Building(s) destroyed,Ind Premises destroyed,Com Premises damaged,Com Premises destroyed,Bridge(s) damaged,Bridge(s) destroyed,Aircraft damaged,Aircraft destroyed,Motor Vehicle(s) damaged,Motor Vehicle(s) destroyed,Water vessel(s) damaged,Water vessel(s) destroyed,Business(es) damaged,Business(es) destroyed,Farm(s) damaged,Farm(s) destroyed,Crop(s) destroyed,Livestock destroyed,Government assistance,regions,subjects,url,resourceUrl i.e. it failed to split columns by commas

The frontend also uses PapaParser for CSV parsing and can get correct result for this file.

Problem reproduction steps

Look at the file above.

Screenshot / Design / File reference

Solution

See #1714

• Make Visualization Minion more plug and play - in terms of input format
• Swap out papa parser with SheetJS
• Add tests

[aneesha09]

We need to look at visualisation sleuther and make it work better. Need to fix this properly

[aneesha09]

@AlexGilleran - Please add the known scenarios for improvement so we can pick this up to start improving it

[aneesha09]

This is a test case for #1714

[aneesha09]

Move away from papa parser for csv Had some success with SheetJS (xlsx in npm) with csv connector; It does most spreadsheet formats.