appbaseio / abc

Power of appbase.io via CLI, with nifty imports from your favorite data sources
https://medium.appbase.io/open-sourcing-abc-import-import-mongodb-sql-json-csv-data-into-elasticsearch-8090965c4340
Apache License 2.0
472 stars 46 forks source link

Feature request: Resuming tail after abc is restarted #91

Open masumsoft opened 6 years ago

masumsoft commented 6 years ago

Question: Does this resume from where it left off if abc is stopped and started later on?

siddharthlatest commented 6 years ago

@masumsoft Currently, no - but we will look into adding support for it.

I am changing this issue to a feature request.

siddharthlatest commented 6 years ago

The support for this has been added in 0.6.2 release.

hobbesuk commented 5 years ago

Are there any examples on how the resume should work? Adding --tail or --tail=true to my abc command to import mongo does tail but on each restart it imports the entire collection again. Is this expected behaviour?

masumsoft commented 5 years ago

The support for this has been added in 0.6.2 release.

@siddharthlatest Is there any additional params needed for the resume functionality to work? Also I failed to find any changes relevant to the resume functionality in 0.6.2.

jeet-parekh commented 5 years ago

@hobbesuk, your user will need to have oplog access. Check out the notes section in the mongodb docs.

jeet-parekh commented 5 years ago

@masumsoft, postgres and mongo have tailing support. Check out the postgres and mongodb docs.

masumsoft commented 5 years ago

@jeet-parekh the issue is not about tailing support, we already use that for mongodb, it's about resuming the tailing from where it left off if abc is restarted. For large databases, it takes a long time to reindex the whole db again with elastic search which is a waste of time and makes abc useless if the server needs to restart for some reason.

siddharthlatest commented 5 years ago

Update: Resume tailing works with MongoDB, but doesn't work with Postgres.