search

pouchdb-community / pouchdb-replication-stream

Replicate PouchDB/CouchDB databases with Node.js-style streams
Apache License 2.0
189 stars 58 forks source link

how to split file when using .dump #37

Closed barbalex closed 8 years ago

barbalex commented 9 years ago

My complete workflow is this:

  1. replicate main DB in to 5 different db's using filtered replication. Example code in powershell: curl -H "Content-Type: application/json" -X POST http://user:pwd@localhost:5984/_replicate -d " {\"source\": \"http://user:pwd@localhost:5984/artendb\", \"target\": \"ae_fauna\", \"create_target\": true, \"filter\":\"artendb/gruppe\", \"query_params\":{\"gruppe\":\"Fauna\"}}"
  2. dump these 5 db's in to text files while splitting them. Example code in powershell: pouchdb-dump http://localhost:5984/ae_fauna -o ae-fauna.txt -s 2000
  3. insert these files as attachments into the main DB (using node.js and nano)
  4. load these files in the web app using pouchdb-load to ensure quick first replication

The problem I am running in to is that curl and it's installation are a pain in the ass on Windows. So I am planning to do steps 1 to 3 all in a node.js module. That will also be a lot more comfortable for repeated usage.

I know how to do it all, except: is it possible to split files when using the dump method from pouchdb-replication-stream, like it is when using pouchdb-dump?

nolanlawson commented 9 years ago

Yes, in fact pouchdb-dump depends on pouchdb-replication-stream, so you can look at the source code to see how it does it: https://github.com/nolanlawson/pouchdb-dump-cli/blob/a13e0355a4c92062af2835ad9fb37f8ab56e45cb/index.js#L159-L174

Basically it requires a knowledge of Node.js streams, which can be a PITA for stuff like this, but definitely support slicing/dicing/transforming/splitting/etc.

barbalex commented 8 years ago

thanks for your great help