open-innovations / jrf-insight

JRF North England Insight Finder
https://open-innovations.github.io/jrf-insight/
MIT License
1 stars 0 forks source link

Add duckdb loader #69

Closed gilesdring closed 1 year ago

gilesdring commented 1 year ago

Resolves data from duckdb rather than the CSV loader. In time, this should mean optimised build times.

For now, this removes the need to use the CSV loader which speeds up the general data load, although the build is still rather slow.

Adds a provisioner, which can be run separately with the following command:

deno task vr provision

This downloads libduckdb.so into the ./lib folder. This is then referred to during the build by setting LD_LIBRARY_PATH. Currently only works on flavours of linux (tested on Ubuntu).