DigitalCommons / mykomap-monolith

A web application for mapping initiatives in the Solidarity Economy
0 stars 0 forks source link

30 dataset importer script #50

Closed rogup closed 3 days ago

rogup commented 2 weeks ago

What? Why?

Related to issue #30

Implement and test a script which can import a CSV of data and write a folder full of the files that the mykomap back-end can use.

This has been test-integrated with upcoming branches and will integrate with the following exceptions. I needed to:

Other conflicting changes upstream, specifically the searchable.json format had changed unexpectedly (to me), have been adjusted for.

Dependencies have been added for parsing CSV (udsv) and a CLI framework (clipanion).

What should we test?

The code should build and unit tests should all pass as ever.

Usage instructions are included in the documentation added in the command itself, which is defined in src/cli/dataset.ts. They are fairly brief for now, as more comprehensive documentation would need to explain the config schema, which is itself a moving target.

You can if you wish to test run the script against the dataset CSV included in the test, or even a larger one, although you will need an accompanying config for that.

A runscript has been added to aid with this:

npm run -- dataset import --help

This invokes using tsx, which is therefore a dependency needed in production mode. ts-node was tried originally, but appears to be problematic. I've not found a better way of running this so far.

Checklist

Deployment notes

rogup commented 3 days ago

This work has already made its way into main via nick/dev3 so will close this PR.