rwynn / monstache

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
https://rwynn.github.io/monstache-site/
MIT License
1.29k stars 181 forks source link

Reduce cost of network traffic out operations #652

Open taihoa opened 2 years ago

taihoa commented 2 years ago

When I run the monstache server on aws, I notice that the network out traffic is steadily increasing day by day. And what's more strange is that on Saturdays and Sundays the traffic is still high, even though these days there is no new data to sync to elasticsearch.

Could you please help me with the cause and solution for this problem?

image

config.toml



gzip = false

stats = true

index-stats = true

elasticsearch-max-conns = 10

elasticsearch-max-seconds = 5

elasticsearch-max-bytes = 8000000

dropped-collections = true

dropped-databases = true

replay = false

resume = true

resume-write-unsafe = false

resume-name = "default"

resume-strategy = 0

index-files = true

file-highlighting = true

file-namespaces = ["rocketchat.rocketchat_uploads.files"]

verbose = false

exit-after-direct-reads = false

prune-invalid-json = true

cluster-name = ''

index-as-update = true

enable-oplog = true

index-oplog-time = true
rwynn commented 2 years ago

Is there any reason you enabled enable-oplog = true? That is an option to support legacy versions of MongoDB before change streams were introduced in 3.6. It issues queries directly against the system oplog collections when that is enabled.