search

rwynn / monstache-site

documentation for the monstache project
https://rwynn.github.io/monstache-site/
MIT License
22 stars 30 forks source link

I want to express all the collections in a database。How should I write direct-read-namespaces #2

Closed snoopy83101 closed 6 years ago

snoopy83101 commented 6 years ago

At the first synchronization I want to express all the collections in a database。 How should I write direct-read-namespaces

rwynn commented 6 years ago

Currently you have to list each one explicitly.

richiarora commented 6 years ago

I had a question : Can new collections added to database be automatically sent to elasticsearch ? considering i use wild card like : namespace-regex = '^mydb.(*\$cmd)$' so ideally it will search through all the collections. will it cater to new collections that i add?

2- Sometimes i have observed collections are not getting synced at all.

rwynn commented 6 years ago

Hi it doesn’t actually maintain a list of collections but rather listens for all changes in Mongodb and then tests the namespace of the change against your regex. So yes it would pick up new collections dynamically.

Simply use ^mydb. as your regex.

If a collection is not getting synced make sure it’s db.col is a match against your configured regexes. Or remove those regexes and sync all changes.

rwynn commented 6 years ago

There should be a backslash before the . In the regex. In my previous comment. It got removed when I posted.

richiarora commented 6 years ago

hi thanks for quick reply so i used namespace-regex = '^my-database\.' or namespace-regex = ^my-database\.([A-Za-z0-9\-]*\$cmd)$ Collections in my database are test1 and test2 and also tried removing regex completely just to push all data to elasticsearch but in logs i only see details related to monstache_stats index.

PS there are no indexes defined in the elasticsearch i was hoping for exception in logs that index not found etc

rwynn commented 6 years ago

Have you also put all the namespaces that you currently have in the direct-read-namespaces array? If you are syncing for the first time you need to do direct reads. Or replay the oplog. By default monstache just tails the oplog for new changes.

rwynn commented 6 years ago

Also if you have a dash in your db name then you need to escape it in the regex.

richiarora commented 6 years ago

hi thanks for pointing out the error the replay=true did the trick. Also wanted to know that will this be capable of working in production on sharded environment ?

rwynn commented 6 years ago

It does have support for sharded clusters. See the docs for mongo-url and mongo-config-url.

https://rwynn.github.io/monstache-site/config/

You would need to evaluate if this would work in your production setup. I don’t have any references at this time.