search

apache / openwhisk-devtools

Development tools for building and deploying Apache OpenWhisk
https://openwhisk.apache.org/
Apache License 2.0
180 stars 126 forks source link

CouchDB is restarting #316

Closed pankaj-dahiya-devops closed 3 years ago

pankaj-dahiya-devops commented 4 years ago

CouchDB is restarting at any point of time, with average load.

Logs:

ubuntu@ip-192-168-2-172:~$ docker logs --tail 100 -f openwhisk_db_1

[info] 2020-03-06T06:29:06.017872Z nonode@nohost <0.206.0> -------- Apache CouchDB has started on http://any:5986/ [info] 2020-03-06T06:29:06.018025Z nonode@nohost <0.9.0> -------- Application couch started on node nonode@nohost [info] 2020-03-06T06:29:06.018117Z nonode@nohost <0.9.0> -------- Application ets_lru started on node nonode@nohost [error] 2020-03-06T06:29:06.032765Z nonode@nohost <0.256.0> -------- Could not get design docs for <<"shards/e0000000-ffffffff/local_subjects.1583327006">> error:{badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [error] 2020-03-06T06:29:06.032768Z nonode@nohost emulator -------- Error in process <0.264.0> with exit value: {badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]}

[notice] 2020-03-06T06:29:06.035915Z nonode@nohost <0.275.0> -------- rexi_server : started servers [notice] 2020-03-06T06:29:06.037558Z nonode@nohost <0.279.0> -------- rexi_buffer : started servers [info] 2020-03-06T06:29:06.037660Z nonode@nohost <0.9.0> -------- Application rexi started on node nonode@nohost [info] 2020-03-06T06:29:06.054675Z nonode@nohost <0.9.0> -------- Application mem3 started on node nonode@nohost [info] 2020-03-06T06:29:06.054794Z nonode@nohost <0.9.0> -------- Application fabric started on node nonode@nohost [info] 2020-03-06T06:29:06.066158Z nonode@nohost <0.9.0> -------- Application chttpd started on node nonode@nohost [info] 2020-03-06T06:29:06.073771Z nonode@nohost <0.9.0> -------- Application couch_index started on node nonode@nohost [info] 2020-03-06T06:29:06.073841Z nonode@nohost <0.9.0> -------- Application couch_mrview started on node nonode@nohost [info] 2020-03-06T06:29:06.073894Z nonode@nohost <0.9.0> -------- Application couch_plugins started on node nonode@nohost [notice] 2020-03-06T06:29:06.104516Z nonode@nohost <0.90.0> -------- config: [features] scheduler set to true for reason nil [info] 2020-03-06T06:29:06.120000Z nonode@nohost <0.9.0> -------- Application couch_replicator started on node nonode@nohost [info] 2020-03-06T06:29:06.125201Z nonode@nohost <0.9.0> -------- Application couch_peruser started on node nonode@nohost [info] 2020-03-06T06:29:06.132083Z nonode@nohost <0.9.0> -------- Application ddoc_cache started on node nonode@nohost [info] 2020-03-06T06:29:06.141197Z nonode@nohost <0.9.0> -------- Application global_changes started on node nonode@nohost [info] 2020-03-06T06:29:06.141222Z nonode@nohost <0.9.0> -------- Application jiffy started on node nonode@nohost [info] 2020-03-06T06:29:06.145054Z nonode@nohost <0.9.0> -------- Application mango started on node nonode@nohost [info] 2020-03-06T06:29:06.148923Z nonode@nohost <0.9.0> -------- Application setup started on node nonode@nohost [info] 2020-03-06T06:29:06.148962Z nonode@nohost <0.9.0> -------- Application snappy started on node nonode@nohost [info] 2020-03-06T06:29:06.832398Z nonode@nohost <0.513.0> -------- Starting index update for db: shards/80000000-9fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832447Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832468Z nonode@nohost <0.512.0> -------- Starting index update for db: shards/60000000-7fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832461Z nonode@nohost <0.516.0> -------- Starting index update for db: shards/e0000000-ffffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832541Z nonode@nohost <0.515.0> -------- Starting index update for db: shards/00000000-1fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832558Z nonode@nohost <0.514.0> -------- Starting index update for db: shards/c0000000-dfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832602Z nonode@nohost <0.517.0> -------- Starting index update for db: shards/40000000-5fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:06.832643Z nonode@nohost <0.518.0> -------- Starting index update for db: shards/20000000-3fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:07.087833Z nonode@nohost <0.313.0> 040e83f052 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 311 [notice] 2020-03-06T06:29:09.317652Z nonode@nohost <0.593.0> 16f6381196 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 129 [notice] 2020-03-06T06:29:11.085734Z nonode@nohost <0.357.0> -------- couch_replicator_clustering : cluster stable [notice] 2020-03-06T06:29:11.403482Z nonode@nohost <0.593.0> 351f4fa8d8 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 37 [notice] 2020-03-06T06:29:11.590263Z nonode@nohost <0.405.0> -------- Started replicator db changes listener <0.735.0> [notice] 2020-03-06T06:29:13.511453Z nonode@nohost <0.593.0> d67f188579 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 18 [notice] 2020-03-06T06:29:15.610703Z nonode@nohost <0.593.0> 5a85946595 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 22 [notice] 2020-03-06T06:29:18.004197Z nonode@nohost <0.593.0> 9abc4ebad2 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 17 [info] 2020-03-06T06:29:20.093836Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:20.119636Z nonode@nohost <0.593.0> 618c92dcda db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 27 [notice] 2020-03-06T06:29:21.992352Z nonode@nohost <0.279.0> -------- rexi_buffer : cluster stable [notice] 2020-03-06T06:29:21.992456Z nonode@nohost <0.275.0> -------- rexi_server : cluster stable [notice] 2020-03-06T06:29:22.201283Z nonode@nohost <0.593.0> 67e91ba1d6 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 15 [notice] 2020-03-06T06:29:24.589630Z nonode@nohost <0.593.0> 1542d318b3 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 303 [os_mon] cpu supervisor port (cpu_sup): Erlang has closed [os_mon] memory supervisor port (memsup): Erlang has closed [info] 2020-03-06T06:29:27.886118Z nonode@nohost <0.9.0> -------- Application couch_log started on node nonode@nohost [info] 2020-03-06T06:29:27.889343Z nonode@nohost <0.9.0> -------- Application folsom started on node nonode@nohost [info] 2020-03-06T06:29:27.917527Z nonode@nohost <0.9.0> -------- Application couch_stats started on node nonode@nohost [info] 2020-03-06T06:29:27.917672Z nonode@nohost <0.9.0> -------- Application khash started on node nonode@nohost [info] 2020-03-06T06:29:27.922342Z nonode@nohost <0.9.0> -------- Application couch_event started on node nonode@nohost [info] 2020-03-06T06:29:27.922469Z nonode@nohost <0.9.0> -------- Application hyper started on node nonode@nohost [info] 2020-03-06T06:29:27.926485Z nonode@nohost <0.9.0> -------- Application ibrowse started on node nonode@nohost [info] 2020-03-06T06:29:27.930253Z nonode@nohost <0.9.0> -------- Application ioq started on node nonode@nohost [info] 2020-03-06T06:29:27.930381Z nonode@nohost <0.9.0> -------- Application mochiweb started on node nonode@nohost [info] 2020-03-06T06:29:27.936093Z nonode@nohost <0.206.0> -------- Apache CouchDB 2.3.1 is starting.

[info] 2020-03-06T06:29:27.936155Z nonode@nohost <0.207.0> -------- Starting couch_sup [notice] 2020-03-06T06:29:27.942717Z nonode@nohost <0.90.0> -------- config: [features] pluggable-storage-engines set to true for reason nil [info] 2020-03-06T06:29:28.011774Z nonode@nohost <0.206.0> -------- Apache CouchDB has started. Time to relax.

[info] 2020-03-06T06:29:28.011841Z nonode@nohost <0.206.0> -------- Apache CouchDB has started on http://any:5986/ [info] 2020-03-06T06:29:28.011999Z nonode@nohost <0.9.0> -------- Application couch started on node nonode@nohost [info] 2020-03-06T06:29:28.012088Z nonode@nohost <0.9.0> -------- Application ets_lru started on node nonode@nohost [error] 2020-03-06T06:29:28.028307Z nonode@nohost emulator -------- Error in process <0.264.0> with exit value: {badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]}

[error] 2020-03-06T06:29:28.028329Z nonode@nohost <0.256.0> -------- Could not get design docs for <<"shards/e0000000-ffffffff/local_subjects.1583327006">> error:{badarg,[{ets,member,[mem3_openers,<<"local_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{fabric_view_all_docs,go,5,[{file,"src/fabric_view_all_docs.erl"},{line,24}]},{couch_db,'-get_design_docs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,627}]}]} [notice] 2020-03-06T06:29:28.030082Z nonode@nohost <0.275.0> -------- rexi_server : started servers [notice] 2020-03-06T06:29:28.031924Z nonode@nohost <0.279.0> -------- rexi_buffer : started servers [info] 2020-03-06T06:29:28.032017Z nonode@nohost <0.9.0> -------- Application rexi started on node nonode@nohost [info] 2020-03-06T06:29:28.048755Z nonode@nohost <0.9.0> -------- Application mem3 started on node nonode@nohost [info] 2020-03-06T06:29:28.048783Z nonode@nohost <0.9.0> -------- Application fabric started on node nonode@nohost [info] 2020-03-06T06:29:28.059825Z nonode@nohost <0.9.0> -------- Application chttpd started on node nonode@nohost [info] 2020-03-06T06:29:28.067676Z nonode@nohost <0.9.0> -------- Application couch_index started on node nonode@nohost [info] 2020-03-06T06:29:28.067742Z nonode@nohost <0.9.0> -------- Application couch_mrview started on node nonode@nohost [info] 2020-03-06T06:29:28.067762Z nonode@nohost <0.9.0> -------- Application couch_plugins started on node nonode@nohost [notice] 2020-03-06T06:29:28.098336Z nonode@nohost <0.90.0> -------- config: [features] scheduler set to true for reason nil [info] 2020-03-06T06:29:28.114139Z nonode@nohost <0.9.0> -------- Application couch_replicator started on node nonode@nohost [info] 2020-03-06T06:29:28.119777Z nonode@nohost <0.9.0> -------- Application couch_peruser started on node nonode@nohost [info] 2020-03-06T06:29:28.127034Z nonode@nohost <0.9.0> -------- Application ddoc_cache started on node nonode@nohost [info] 2020-03-06T06:29:28.136469Z nonode@nohost <0.9.0> -------- Application global_changes started on node nonode@nohost [info] 2020-03-06T06:29:28.136509Z nonode@nohost <0.9.0> -------- Application jiffy started on node nonode@nohost [info] 2020-03-06T06:29:28.140539Z nonode@nohost <0.9.0> -------- Application mango started on node nonode@nohost [info] 2020-03-06T06:29:28.144531Z nonode@nohost <0.9.0> -------- Application setup started on node nonode@nohost [info] 2020-03-06T06:29:28.144553Z nonode@nohost <0.9.0> -------- Application snappy started on node nonode@nohost [info] 2020-03-06T06:29:28.458348Z nonode@nohost <0.513.0> -------- Starting index update for db: shards/80000000-9fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458363Z nonode@nohost <0.515.0> -------- Starting index update for db: shards/20000000-3fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458363Z nonode@nohost <0.512.0> -------- Starting index update for db: shards/00000000-1fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458417Z nonode@nohost <0.516.0> -------- Starting index update for db: shards/60000000-7fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458430Z nonode@nohost <0.519.0> -------- Starting index update for db: shards/40000000-5fffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458528Z nonode@nohost <0.514.0> -------- Starting index update for db: shards/e0000000-ffffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458544Z nonode@nohost <0.517.0> -------- Starting index update for db: shards/c0000000-dfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [info] 2020-03-06T06:29:28.458578Z nonode@nohost <0.518.0> -------- Starting index update for db: shards/a0000000-bfffffff/local_activations.1583327014 idx: _design/whisks.v2.1.0 [notice] 2020-03-06T06:29:29.196043Z nonode@nohost <0.313.0> 0860ed2981 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 791 [notice] 2020-03-06T06:29:31.299075Z nonode@nohost <0.593.0> ea308a5967 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 28 [notice] 2020-03-06T06:29:33.080007Z nonode@nohost <0.357.0> -------- couch_replicator_clustering : cluster stable [notice] 2020-03-06T06:29:33.407720Z nonode@nohost <0.593.0> c1f76fdb37 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 16 [notice] 2020-03-06T06:29:33.910140Z nonode@nohost <0.405.0> -------- Started replicator db changes listener <0.735.0> [notice] 2020-03-06T06:29:35.512161Z nonode@nohost <0.593.0> 27d2ecef6f db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 25 [notice] 2020-03-06T06:29:37.613436Z nonode@nohost <0.593.0> c5cdf30014 db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 16 [notice] 2020-03-06T06:29:39.793311Z nonode@nohost <0.593.0> a585d7298a db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 92 [notice] 2020-03-06T06:29:41.899248Z nonode@nohost <0.593.0> e8bd3e076e db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 58 [notice] 2020-03-06T06:29:43.031047Z nonode@nohost <0.275.0> -------- rexi_server : cluster stable [notice] 2020-03-06T06:29:43.031109Z nonode@nohost <0.279.0> -------- rexi_buffer : cluster stable [notice] 2020-03-06T06:29:44.005069Z nonode@nohost <0.593.0> c550ea3b9e db.docker:5984 172.18.0.11 whisk_admin GET /local_activations/_design/whisks.v2.1.0/_view/activations?descending=true&endkey=%5B%22guest%22,0%5D&reduce=false&startkey=%5B%22guest%22,%22%EF%BF%B0%22,%22%EF%BF%B0%22%5D&limit=200&stale=update_after 200 ok 17 erl_child_setup closed

Crash dump is being written to: erl_crash.dump...[os_mon] cpu supervisor port (cpu_sup): Erlang has closed [os_mon] memory supervisor port (memsup): Erlang has closed

rabbah commented 4 years ago

This suggests your process ran out of memory. How much ram are you allocating to couch/can you bump it?

 [os_mon] memory supervisor port (memsup): Erlang has closed
pankaj-dahiya-devops commented 4 years ago

Hi, we are using 6GB RAM with very minimal load!

rabbah commented 3 years ago

Closing as stale.