Dynomite Log File Grows Indefinitely

[guruprasadb80]

I am seeing an issue in our production setup where Dynomite Log file grows indefinitely while printing the logs and system eventually reaches disk full condition and services are impacted.

[2019-06-19 09:51:01.905] dnode_accept:157 Accepting client connection from 10.237.85.33:40460 on sd 1023 [2019-06-19 09:51:01.905] dnode_accept:145 accept on <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> failed: Too many open files [2019-06-19 09:51:01.905] dnode_recv:214 <CONN_PEER_PROXY 0xd54260 9 listening on '10.237.92.98:5002'> Failed to accept a connection. Continuing

This happened once in the past on 19th June and same issue was seen again yesterday.

I have seen an earlier post 424. However, this issue seems to have been fixed and i am using a build v0.6.9.

Dynomite Config:

dyn_o_mite: datacenter: dc-cak rack: rack2 dyn_listen: 10.227.92.98:5002 dyn_seeds:

Distribution to dynomite instances on CAK-Redis1 server

• 10.227.92.97:5001:rack1:dc-cak:0
• 10.227.92.97:5002:rack1:dc-cak:1073741824
• 10.227.92.97:5003:rack1:dc-cak:2147483648
• 10.227.92.97:5004:rack1:dc-cak:3221225472

  Distribution to dynomite instances on cak-Redis2 server (other than self)

• 10.227.92.98:5001:rack2:dc-cak:1073741824
• 10.227.92.98:5003:rack2:dc-cak:2147483648
• 10.227.92.98:5004:rack2:dc-cak:3221225472

  Distribution to dynomite instances on rch-Redis1 server

• 10.227.85.33:5001:rack1:dc-rch:0
• 10.227.85.33:5002:rack1:dc-rch:1073741824
• 10.227.85.33:5003:rack1:dc-rch:2147483648
• 10.227.85.33:5004:rack1:dc-rch:3221225472

  Distribution to dynomite instances on rch-Redis2 server

• 10.227.85.34:5001:rack2:dc-rch:0
• 10.227.85.34:5002:rack2:dc-rch:1073741824
• 10.237.85.34:5003:rack2:dc-rch:2147483648
• 10.227.85.34:5004:rack2:dc-rch:3221225472 listen: 10.237.92.98:6312 servers:
• 127.0.0.1:6302:1 tokens: '0' data_store: 0 stats_listen: 127.0.0.1:6322 read_consistency : DC_ONE write_consistency : DC_ONE

Any clues on what could have gone wrong?

[guruprasadb80]

Below logs are seen continuously.

[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'> [2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>

[smukil]

@guruprasadb80 We use internal tooling at Netflix to do log rotation, and that's why it's not implemented as part of Dynomite. It's fairly easy to set up yourself however.

Eg: https://linux.die.net/man/8/logrotate

[Carson220]

Below logs are seen continuously.

[2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'> [2019-08-05 16:48:53.637] client_unref_internal_try_put:94 <CONN_CLIENT 0x229a330 -1 from '10.227.92.111:40286'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>

Does your problem solve? I've been in this situation, too. What does this log entry mean?

client_unref_internal_try_put:94 <CONN_CLIENT 0x2299b20 -1 from '10.227.92.111:40284'> unref owner <POOL 0x222b0e0 'dyn_o_mite'>