Detect, read and parse CSVs and save to Json

signal-noise / datasupply

Data Supply is an opinionated data processing, provision and presentation library of components for Javascript

MIT License

0 stars 0 forks source link

Detect, read and parse CSVs and save to Json #17

Closed chriscamplin closed 3 years ago

chriscamplin commented 3 years ago

Which issue, if any, is this issue related to?

The project directory is scanned for CSV/TSVs. If found these are parsed to a flat data structure, using headings as column names. A config file is used to define which targetDirectory to start recursively searching for CSVs. Any directories to exclude, any additional fileTypes and which column separator the parser should use, can also be defined. Currently data is saved as a Json file in a src/data Tests use Jest and mocks replace the fs API during. More testing should be implemented.

tomgp commented 3 years ago

Having given it some thought over the weekend I don't think mocking fs for tests makes much sense here. The integration with the filesystem is the thing we want to test. I think it'll be better to make a test directory and write tests against that.

chriscamplin commented 3 years ago

@tomgp I was actually thinking about rewriting the tests over the weekend too, I'll put some more time into this.

tomgp commented 3 years ago

@tomgp I was actually thinking about rewriting the tests over the weekend too, I'll put some more time into this.

Let's merge this. And sort testing out after a chat

chriscamplin commented 3 years ago

@marcelkornblum @tomgp If you are waiting for me, I don't have permission to merge:

marcelkornblum commented 3 years ago

Not sure my above comment was entirely intentional! Also will figure out permissions, you're not meant to be forbidden sorry Chris!