rwynn / monstache

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
https://rwynn.github.io/monstache-site/
MIT License
1.28k stars 180 forks source link

direct-read-namespaces sync all databases from mongo cluster #431

Open fayas-mdosl opened 3 years ago

fayas-mdosl commented 3 years ago

I have configured as below. but monstache sync entire databases! As per your documentation tI configured to copy collections from MongoDB to Elasticsearch. it syncs other than I mentioned. Please kindly reply how it's done

mongo-url ="xyz" elasticsearch-urls=["tyu"] elasticsearch-max-conns = 10 replay = false resume = true resume-name = "default"

namespace-regex ='mydb.documents'

direct-read-namespaces = ["mydb.documents"] #if I enable this, it sync all databases from cluster (every databse a,b,c)

change-stream-namespaces = ["mydb.documents"] index-as-update = true dropped-collections = true dropped-databases = true verbose = true direct-read-split-max = -1 exit-after-direct-reads = false [logs] info = "/etc/monstache-build/logs/monstache_info.log" warn = "/etc/monstache-build/logs/monstache_warn.log" error = "/etc/monstache-build/logs/monstache_error.log" trace = "/etc/monstache-build/logs/monstache_trace.log" stats = "/etc/monstache-build/logs/monstache_stats.log"

rwynn commented 3 years ago

I have not experienced this issue. Can you check the output of monstache -f your_config.toml -print-config? Also, ensure you don't have any monstache environment variables set.

fayas-mdosl commented 3 years ago

I did check the config of mine. it seems good can you explain how to do index mapping while some index exists on my elastic search?

I found out in my trace log that there's the same index that exists in elastic search and therefore it rejects my sync! this issue came due to synced all databases.

Please help me with the index mapping

my trace log "reason":"mapper [metaData.value] of different type, current_type [date], merged_type [text]"}}}]}

Bulk response item: {"_index":"mydb.documents","_type":"_doc","_id":"5ec2e34vdfbvr557u768","status":400,"error":{"type":"illegal_argument_exception","reason":"mapper [metaData.value] of different type, current_type [date], merged_type [text]"}}

my mapping [[mapping]] namespace = "test.test" index = "index1"

also, can you explain how "resumes" works