This PR adds support to directly export data from Postgres, which avoids Node.js memory limitations. The data is exported as standard CSV file, and compressed with gzip.
Since the data is exported from Postgres, it doesn't need to run through Node.js and JSON conversion first, which enables very large data sets.
To make this work in practice, extra configuration may be necessary when for instance Postgres runs in a container. Generally, it's assume that all data files end up in the same folder.
For this purpose, there are two new environment variables:
RDB_DATA_EXPORT_PATH the path as seen from Postgres
NODE_DATA_EXPORT_PATH the path as seen from Node.js, all files are assume to be accessible by Node.js or the web server to serve them to the client
This is now tested on github with a basic end-to-end test.
There are a few other minor commits here:
improve log output
avoid test race conditions by using separate databases
always resolve the path to be absolute when getting a "robust" one
This PR adds support to directly export data from Postgres, which avoids Node.js memory limitations. The data is exported as standard CSV file, and compressed with gzip. Since the data is exported from Postgres, it doesn't need to run through Node.js and JSON conversion first, which enables very large data sets.
To make this work in practice, extra configuration may be necessary when for instance Postgres runs in a container. Generally, it's assume that all data files end up in the same folder.
For this purpose, there are two new environment variables:
RDB_DATA_EXPORT_PATHthe path as seen from PostgresNODE_DATA_EXPORT_PATHthe path as seen from Node.js, all files are assume to be accessible by Node.js or the web server to serve them to the clientThis is now tested on github with a basic end-to-end test.
There are a few other minor commits here: