search

csirtgadgets / massive-octo-spice

DEPRECATED - USE v3 (bearded-avenger)
https://github.com/csirtgadgets/bearded-avenger-deploymentkit/wiki
GNU Lesser General Public License v3.0
227 stars 60 forks source link

v1 -> v2 migration #363

Closed villain closed 8 years ago

villain commented 9 years ago

unsure if anyone else is experiencing this, however i do not appear to be able to migrate data from a v1 to v2 instance. the script is being run with the following options:

perl -I../src/lib -Ilib bin/migrate-data.pl --threads 8 --psql-host v1.host --es-token -d --es-host es.host

the script starts to migrate what looks like 1000 docs x thread count (so 8,000 using the above command line then stops. i've left this running for varying amounts of time (1 day, up to a week) with the same results. below is the debug output:

[2015-10-22T10:31:07,530Z][14932][INFO][main:137]: staring up.. [2015-10-22T10:31:07,530Z][14932][INFO][main:150]: starting up ES connection... [2015-10-22T10:31:07,530Z][14932][INFO][main:156]: checking journal: /tmp/cif-migrate.journal [2015-10-22T10:31:07,530Z][14932][INFO][main:159]: creating threads... [2015-10-22T10:31:07,752Z][14932][INFO][main:214]: starting workers [2015-10-22T10:31:07,754Z][14932][DEBUG][main:227]: connecting to archive.. [2015-10-22T10:31:07,756Z][14932][DEBUG][main:246]: starting at id: 151020 [2015-10-22T11:07:38,523Z][14932][DEBUG][main:256]: total count: 271463617 [2015-10-22T11:07:38,523Z][14932][DEBUG][main:257]: pages: 54293 [2015-10-22T11:07:38,530Z][14932][DEBUG][main:264]: sending ctrl warm-up msg... [2015-10-22T11:07:38,737Z][14932][INFO][main:418]: starting writer thread... [2015-10-22T11:07:38,737Z][14932][DEBUG][main:270]: creating 8 worker threads... Subroutine CIF::Legacy::Archive::db_Main redefined at /usr/local/share/perl/5.18.2/Ima/DBI.pm line 278. [2015-10-22T11:07:38,859Z][14932][DEBUG][main:325]: starting worker: 3 [2015-10-22T11:07:39,042Z][14932][DEBUG][main:325]: starting worker: 4 [2015-10-22T11:07:39,218Z][14932][DEBUG][main:325]: starting worker: 5 [2015-10-22T11:07:39,354Z][14932][DEBUG][main:325]: starting worker: 6 [2015-10-22T11:07:39,579Z][14932][DEBUG][main:325]: starting worker: 7 [2015-10-22T11:07:39,743Z][14932][DEBUG][main:325]: starting worker: 8 [2015-10-22T11:07:39,853Z][14932][DEBUG][main:325]: starting worker: 9 [2015-10-22T11:07:39,976Z][14932][DEBUG][main:325]: starting worker: 10 [2015-10-22T11:07:39,976Z][14932][DEBUG][main:277]: executing sql... [2015-10-22T11:07:40,228Z][14932][DEBUG][main:283]: sending next pages to workers... [2015-10-22T11:07:40,290Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:07:40,303Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 [2015-10-22T11:07:40,584Z][14932][DEBUG][main:288]: waiting on workers to finish up... .[2015-10-22T11:07:40,706Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:07:40,707Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 .[2015-10-22T11:07:40,740Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:07:40,742Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 .[2015-10-22T11:07:40,842Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:07:40,843Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 .[2015-10-22T11:07:40,893Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:07:40,895Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22

< snip >

[2015-10-22T11:23:20,436Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 .[2015-10-22T11:23:20,536Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:23:20,537Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22 .[2015-10-22T11:23:20,548Z][14932][DEBUG][main:442]: found message... [2015-10-22T11:23:20,549Z][14932][DEBUG][CIF::Storage::ElasticSearch:463]: submitting to index: cif.observables-2013.07.22

and it just sits there

wesyoung commented 9 years ago

can you verify the es.host is accessible from this box?

villain commented 9 years ago

yep, its accessible. i can see the documents that it processes successfully in the elasticsearch index

wesyoung commented 8 years ago

fwiw; i think there was an unused zmq pipe (from an older version of the script) clogging this up. you're probably way past this; but i figured i'd try and follow through.