camtool subcommand for converting index to another format

[GoogleCodeExporter]

In my tests, MySQL indexing is much faster than using a sqlite index, but it 
requires running a mysqld. The time consumed by importing many GBs of photos 
and videos differs by multiple hours between the two indexers for me.

I do not want keep a mysqld running all the time, for various reasons. 

For initial mass imports I could 
a) use a fast index temporarily and import everything, 
b) migrate the simple tables of the index to another format,
c) switch the configuration over to the new format,
d) restart camlistored and shutdown/remove the previous index. 

camtool could help out at least on b) and c)

Original issue reported on code.google.com by dsimfab...@gmail.com on 23 Aug 2014 at 9:52

[GoogleCodeExporter]

According to issue #492 , Bill already has the KV to postgres part done.

Out of curiosity, why do you not want mysqld running all the time?

Original comment by mathieu....@gmail.com on 23 Aug 2014 at 1:38

[GoogleCodeExporter]

I agree that we should have a camtool command for this.

Mathieu, the benefits of having a standalone binary should be obvious. We 
should be able to come up with a pure Go solution that's as fast as MySQL for 
how simply we use MySQL.

Original comment by bradfitz on 23 Aug 2014 at 3:15

• Changed state: Accepted

[GoogleCodeExporter]

I agree. Further on, for me it's a matter of maximum parsimony and me being a 
lazy dog: I simply do not want to care for yet another daemon and the 
ressources and admin effort it requires. 

Especially since I plan to run Camlistore on my weak ARM5-based NAS. Go 
programs like camlistored are delightfully easy to cross-compile and deploy on 
such a system. Getting a fully-blown RDBMS ann all its dependencies to run on 
it ... -- no thanks. :-)

Original comment by dsimfab...@gmail.com on 23 Aug 2014 at 6:34

[GoogleCodeExporter]

Here's the small program I hacked together to migrate KV -> Postgres

https://gist.github.com/wathiede/92f9f0f63bda23a2eaef

Not sure if it is bug free, but it seemed to work for me.  Most of the code is 
around creating the sorted.KeyValue based on user input.  The rest is pretty 
simple.

Original comment by couchmo...@gmail.com on 25 Aug 2014 at 3:46

[GoogleCodeExporter]

Looks good.

Original comment by bradfitz on 25 Aug 2014 at 3:47

[GoogleCodeExporter]

Original comment by bradfitz on 21 Oct 2014 at 3:04

• Changed state: HelpWanted
• Added labels: Component-Tools

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/497

Original comment by bradfitz on 14 Dec 2014 at 11:37

• Changed state: Moved
• Added labels: IssueMoved, Restrict-AddIssueComment-Commit