pouchdb-community / pouchdb-load

Load documents into CouchDB/PouchDB from a dumpfile
Apache License 2.0
120 stars 33 forks source link

add push_proxy option #26

Closed ddouglascarr closed 9 years ago

ddouglascarr commented 9 years ago

If I am setting up a database that will have a push and pull replication, I want pouchdb-load to register a checkpoint for push replication as well.

As it currently works, after using pouchdb-load with the proxy option, the pull replication can get going straight away, but the push replication needs to run the checkpointer across the whole database.

This isn't necessary when the database is new and completely populated by load.

foucdeg commented 9 years ago

I'm not sure that I understand the internals, but it would be nice to have that nice proxy speed boost for the push replication too.

nolanlawson commented 9 years ago

@ddouglascarr Hm, that seems kinda odd to me given that pouchdb-load only loads in one direction. :) Instead of using sync(), you could do two separate replicate() commands, and then each one can have their own load() and proxy. Does that make sense? Otherwise I'm not sure what you're asking for here.

BTW on a side note, bulkDocs() has been implemented and should arrive in PouchDB 4.1.0. The CouchDB 1.x implementation is forthcoming, and PouchDB Server will get it as soon as PouchDB does: https://github.com/pouchdb/pouchdb/pull/4169

ddouglascarr commented 9 years ago

@nolanlawson. Thanks for the response. I think I have a use case that might be not the target for the plugin. I'll explain what I'm doing and what I intend to do so that anyone else trying this can use as a reference.

TLDR: no, load() on the server is very inefficient. (would have to generate pouchdb dump on the client and send it back to the server, just to set the checkpoint)

I'll close the issue, as it looks like bulkGet() will replace this anyway.

Scenario:

Problem:

Solution:

If anyone else runs into this problem between now and when bulkGet is usable (unlikely), mention me here and I can help you do it.

foucdeg commented 9 years ago

@ddouglascarr this is my use case too and I'm interested in your solution.

foucdeg commented 9 years ago

@nolanlawson and I do think this is a common use case. I understand that you find it odd to make load() write a checkpoint on the remote, but once @ddouglascarr explains how to do it ourselves, his explanation might be relevant to mention in the readme.

foucdeg commented 9 years ago

Hi @ddouglascarr , sorry to dig this up but I still really need to know how to to manually create the checkpoint on the remote database. Can you please explain it or point me to any instructions ? Thanks