Improvements and bug fix of manual replicator

[GoogleCodeExporter]

Hi, I have to use manual replicator and found bug and some possible
improvements for it.

Just now `manaul_replication.py` uses old Session API and breaks when
trying to fetch list of available source server databases.

And it doesn't allow to replicate databases across one server.

In my fork [1] I've fixed the bug and added ability to replicate databases
on one server.

Also I think it will be very useful to allow usage of glob-syntax when
specifying databases names (helps if you operates several databases in your
application with some prefix on one server).

Please review my changes [2] and merge if possible.

Thanks.

[1]: https://daevaorn-couchdb-python.googlecode.com/hg/
[2]: http://code.google.com/r/daevaorn-couchdb-python/source/list

Original issue reported on code.google.com by daevaorn on 22 Apr 2010 at 2:35

[GoogleCodeExporter]

Okay, let's see where we want to go with the replicate.py script (formerly 
called
manual_replication.py). Here are some things I'd like to change:

- take source and target as arguments, not options
- interpret source and target as host + db by default
- allow --database option for specifying databases, plus globbing (so * for all 
dbs)
- flip the default for target compaction (don't compact by default)

I think the second point requires the assumption that the couchdb Server is at 
the
host level. Is that reasonable enough? I guess we could do a simple request to 
see
whether it's a database or a server, and then provide DWIM behavior.

Original comment by djc.ochtman on 6 Jun 2010 at 5:19

[GoogleCodeExporter]

Ok. Just now I've updated my branch to current upstream state and will try to 
work on
your remarks soon.

Original comment by daevaorn on 6 Jun 2010 at 7:17

[GoogleCodeExporter]

You might want to wait until we have reached some consensus. :)

BTW, we should probably also have some tests for this. Your branch had some 
typos
that didn't look very good.

Original comment by djc.ochtman on 6 Jun 2010 at 7:21

[GoogleCodeExporter]

Matt, can you review my proposed changes here?

Original comment by djc.ochtman on 16 Jun 2010 at 6:33

[GoogleCodeExporter]

Yeah, I think that list is reasonable. I came at it from the point of view of 
the user and came up with basically the same list, just with a bit more detail 
on the implementation.

Firstly, I think the source and target should be required args (not as bad as 
it sounds, see glob-like examples below).

Here's how I think various use cases should look:

Replicate one database to remote server:

* `couchdb-replicate $src/foo $tgt/` (implicit remote db name)
* `couchdb-replicate $src/foo $tgt/foo`
* `couchdb-replicate $src/foo $tgt/bar`

Replicate all databases:

* `couchdb-replicate $src $tgt`

Note: I'm tempted to ban this, it seems a little dangerous ;-) and is easily 
handled by a glob-like URL.

Simple glob-like URLs:

* `couchdb-replicate $src/* $tgt`
* `couchdb-replicate $src/user/* $tgt`
* `couchdb-replicate $src/user/? $tgt`
* `couchdb-replicate * $tgt`
* `couchdb-replicate user/* $tgt`
* `couchdb-replicate user/? $tgt`

Probably just support '*' and '?'. Use --database for more complex matching.

Of the options:

* Yes, --compact (False by default) is better.
* Drop --push, just flip the src and tgt if that's what's wanted.
* --continuous, fine as it is

There are various combinations that don't work, e.g. matching src URL and 
--database, --compact and --continuous, but that's just coding.

The only thing I can think of that's not nice is replicating all remote 
databases to the local machine because there's no nice way of saying "here". 
Best I can come up with is using ".", e.g. `couchdb-replicate $tgt .`.

Original comment by matt.goo...@gmail.com on 16 Jun 2010 at 9:24

[GoogleCodeExporter]

I just pushed changes to implement these (leading up to r81d6ba4427d9). Matt, 
can you review this? I'd like to release 0.8 before or in this weekend.

Original comment by djc.ochtman on 5 Aug 2010 at 5:49

[GoogleCodeExporter]

Hi, just had a really quick play and fixed a few issues. I've pushed the 
changes.

It's actually using a conditional expression (aka ternary operator) right now 
which means Python 2.5+ support only.

Also, I notice it always starts the replication task on the target. I wonder if 
there should be a "where=(source|target)" option to control that.

Just a thought ... we could probably make use of the fnmatch module to build 
the list of source databases but it's no big deal.

Other than that it looks good to me. Certainly worked here (apart from fixes 
I've pushed).

Original comment by matt.goo...@gmail.com on 6 Aug 2010 at 10:45

[GoogleCodeExporter]

Thanks for the fixes!

I'll remove the ternary operator. About starting the replication task on the 
target, you yourself said in comment 5 that people could just switch source and 
target?

Yeah, I wondered about something like fnmatch. Will take a whack at using that 
instead.

Original comment by djc.ochtman on 7 Aug 2010 at 8:43

[GoogleCodeExporter]

Er, yeah , I was wrong about switching the source and target, sorry! Push vs 
pull is something completely different. Tell you what, get a release out and 
I'll put --push back again to make up for being dumb ;-).

Original comment by matt.goo...@gmail.com on 7 Aug 2010 at 2:27

[GoogleCodeExporter]

I think this ticket can be closed. All needed changes have been applied to the 
replicatior.

Original comment by daevaorn on 23 Sep 2010 at 8:06

[GoogleCodeExporter]

Yup.

Original comment by djc.ochtman on 23 Sep 2010 at 8:13

• Changed state: Fixed