Validator account got auto killed.

[nexusfallout]

Node was running for almost 15 days 1GB RAM/25GB SDD/1CPU on digital ocean. It got auto killed at 2019/03/16

posting here parts of logs i think is usefull, if you need the full log let me know its 4 files actually.

2019/03/16-17:20:48.160926 7fdfe8a21e00 Options.write_buffer_size: 67108864 2019/03/16-17:20:48.160928 7fdfe8a21e00 Options.max_write_buffer_number: 2 2019/03/16-17:20:48.160929 7fdfe8a21e00 Options.compression: Snappy 2019/03/16-17:20:48.160931 7fdfe8a21e00 Options.bottommost_compression: Disabled 2019/03/16-17:20:48.160932 7fdfe8a21e00 Options.prefix_extractor: nullptr 2019/03/16-17:20:48.160933 7fdfe8a21e00 Options.memtable_insert_with_hint_prefix_extractor: nullptr 2019/03/16-17:20:48.160934 7fdfe8a21e00 Options.num_levels: 7 2019/03/16-17:20:48.160935 7fdfe8a21e00 Options.min_write_buffer_number_to_merge: 1 2019/03/16-17:20:48.160936 7fdfe8a21e00 Options.max_write_buffer_number_to_maintain: 0 2019/03/16-17:20:48.160938 7fdfe8a21e00 Options.compression_opts.window_bits: -14 2019/03/16-17:20:48.160939 7fdfe8a21e00 Options.compression_opts.level: -1 2019/03/16-17:20:48.160940 7fdfe8a21e00 Options.compression_opts.strategy: 0 2019/03/16-17:20:48.160941 7fdfe8a21e00 Options.compression_opts.max_dict_bytes: 0 2019/03/16-17:20:48.160943 7fdfe8a21e00 Options.compression_opts.zstd_max_train_bytes: 0 2019/03/16-17:20:48.160944 7fdfe8a21e00 Options.level0_file_num_compaction_trigger: 4 2019/03/16-17:20:48.160945 7fdfe8a21e00 Options.level0_slowdown_writes_trigger: 20 2019/03/16-17:20:48.160946 7fdfe8a21e00 Options.level0_stop_writes_trigger: 36 2019/03/16-17:20:48.160947 7fdfe8a21e00 Options.target_file_size_base: 67108864 2019/03/16-17:20:48.160948 7fdfe8a21e00 Options.target_file_size_multiplier: 1 2019/03/16-17:20:48.160949 7fdfe8a21e00 Options.max_bytes_for_level_base: 268435456 2019/03/16-17:20:48.160951 7fdfe8a21e00 Options.level_compaction_dynamic_level_bytes: 0 2019/03/16-17:20:48.160952 7fdfe8a21e00 Options.max_bytes_for_level_multiplier: 10.000000 2019/03/16-17:20:48.160954 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[0]: 1 2019/03/16-17:20:48.160955 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[1]: 1 2019/03/16-17:20:48.160956 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[2]: 1 2019/03/16-17:20:48.160958 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[3]: 1 2019/03/16-17:20:48.160959 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[4]: 1 2019/03/16-17:20:48.160960 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[5]: 1 2019/03/16-17:20:48.160961 7fdfe8a21e00 Options.max_bytes_for_level_multiplier_addtl[6]: 1 2019/03/16-17:20:48.160963 7fdfe8a21e00 Options.max_sequential_skip_in_iterations: 8 2019/03/16-17:20:48.160964 7fdfe8a21e00 Options.max_compaction_bytes: 1677721600 2019/03/16-17:20:48.160965 7fdfe8a21e00 Options.arena_block_size: 8388608 2019/03/16-17:20:48.160966 7fdfe8a21e00 Options.soft_pending_compaction_bytes_limit: 68719476736 2019/03/16-17:20:48.160967 7fdfe8a21e00 Options.hard_pending_compaction_bytes_limit: 274877906944 2019/03/16-17:20:48.160968 7fdfe8a21e00 Options.rate_limit_delay_max_milliseconds: 100 2019/03/16-17:20:48.160975 7fdfe8a21e00 Options.disable_auto_compactions: 0 2019/03/16-17:20:48.160977 7fdfe8a21e00 Options.compaction_style: kCompactionStyleLevel 2019/03/16-17:20:48.160978 7fdfe8a21e00 Options.compaction_pri: kByCompensatedSize 2019/03/16-17:20:48.160979 7fdfe8a21e00 Options.compaction_options_universal.size_ratio: 1 2019/03/16-17:20:48.160980 7fdfe8a21e00 Options.compaction_options_universal.min_merge_width: 2 2019/03/16-17:20:48.160982 7fdfe8a21e00 Options.compaction_options_universal.max_merge_width: 4294967295 2019/03/16-17:20:48.160983 7fdfe8a21e00 Options.compaction_options_universal.max_size_amplification_percent: 200 2019/03/16-17:20:48.160984 7fdfe8a21e00 Options.compaction_options_universal.compression_size_percent: -1 2019/03/16-17:20:48.160985 7fdfe8a21e00 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2019/03/16-17:20:48.160987 7fdfe8a21e00 Options.compaction_options_fifo.max_table_files_size: 1073741824 2019/03/16-17:20:48.160988 7fdfe8a21e00 Options.compaction_options_fifo.allow_compaction: 0 2019/03/16-17:20:48.160989 7fdfe8a21e00 Options.compaction_options_fifo.ttl: 0 2019/03/16-17:20:48.160990 7fdfe8a21e00 Options.table_properties_collectors: 2019/03/16-17:20:48.160992 7fdfe8a21e00 Options.inplace_update_support: 0 2019/03/16-17:20:48.160993 7fdfe8a21e00 Options.inplace_update_num_locks: 10000 2019/03/16-17:20:48.160994 7fdfe8a21e00 Options.memtable_prefix_bloom_size_ratio: 0.000000 2019/03/16-17:20:48.160996 7fdfe8a21e00 Options.memtable_huge_page_size: 0 2019/03/16-17:20:48.160998 7fdfe8a21e00 Options.bloom_locality: 0 2019/03/16-17:20:48.160999 7fdfe8a21e00 Options.max_successive_merges: 0 2019/03/16-17:20:48.161000 7fdfe8a21e00 Options.optimize_filters_for_hits: 0 2019/03/16-17:20:48.161002 7fdfe8a21e00 Options.paranoid_file_checks: 0 2019/03/16-17:20:48.161003 7fdfe8a21e00 Options.force_consistency_checks: 0 2019/03/16-17:20:48.161004 7fdfe8a21e00 Options.report_bg_io_stats: 0 2019/03/16-17:20:48.161006 7fdfe8a21e00 Options.ttl: 0 2019/03/16-17:20:48.164637 7fdfe8a21e00 [rocksdb/db/version_set.cc:3361] Recovered from manifest file:/root/.local/share/joystream-node/chains/joy_testnet_1/db/MANIFEST-000153 succeeded,manifest_file_number is 153, next_file_number is 289, last_sequence is 9126191, log_number is 0,prev_log_number is 0,max_column_family is 9,min_log_number_to_keep is 0 2019/03/16-17:20:48.164644 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [default] (ID 0), log number is 150 2019/03/16-17:20:48.164647 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col0] (ID 1), log number is 150 2019/03/16-17:20:48.164648 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col1] (ID 2), log number is 286 2019/03/16-17:20:48.164650 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col2] (ID 3), log number is 282 2019/03/16-17:20:48.164651 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col3] (ID 4), log number is 150 2019/03/16-17:20:48.164653 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col4] (ID 5), log number is 150 2019/03/16-17:20:48.164654 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col5] (ID 6), log number is 150 2019/03/16-17:20:48.164656 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col6] (ID 7), log number is 150 2019/03/16-17:20:48.164657 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col7] (ID 8), log number is 150 2019/03/16-17:20:48.164658 7fdfe8a21e00 [rocksdb/db/version_set.cc:3369] Column family [col8] (ID 9), log number is 150 2019/03/16-17:20:48.164980 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756848164966, "job": 1, "event": "recovery_started", "log_files": [160, 164, 166, 169, 173, 175, 177, 179, 183, 185, 187, 190, 194, 196, 198, 200, 204, 206, 208, 211, 215, 217, 219, 221, 225, 227, 229, 233, 236, 238, 240, 244, 246, 248, 250, 254, 257, 259, 261, 265, 267, 269, 271, 275, 278, 280, 282, 286]} 2019/03/16-17:20:48.165005 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #160 mode 2 2019/03/16-17:20:48.747218 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #164 mode 2 2019/03/16-17:20:48.823670 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #166 mode 2 2019/03/16-17:20:49.350923 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #169 mode 2 2019/03/16-17:20:49.457136 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #173 mode 2 2019/03/16-17:20:49.951027 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #175 mode 2 2019/03/16-17:20:50.116323 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #177 mode 2 2019/03/16-17:20:50.552241 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #179 mode 2 2019/03/16-17:20:50.761009 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #183 mode 2 2019/03/16-17:20:51.171247 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #185 mode 2 2019/03/16-17:20:51.431592 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #187 mode 2 2019/03/16-17:20:51.762700 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #190 mode 2 2019/03/16-17:20:52.089674 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #194 mode 2 2019/03/16-17:20:52.514158 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #196 mode 2 2019/03/16-17:20:53.261151 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #198 mode 2 2019/03/16-17:20:53.675175 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #200 mode 2 2019/03/16-17:20:54.473578 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #204 mode 2 2019/03/16-17:20:54.771434 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #206 mode 2 2019/03/16-17:20:55.448738 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #208 mode 2 2019/03/16-17:20:55.631899 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #211 mode 2 2019/03/16-17:20:56.212854 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #215 mode 2 2019/03/16-17:20:56.321921 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #217 mode 2 2019/03/16-17:20:57.096872 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #219 mode 2 2019/03/16-17:20:57.166899 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #221 mode 2 2019/03/16-17:20:57.903040 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #225 mode 2 2019/03/16-17:20:57.914215 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #227 mode 2 2019/03/16-17:20:58.869598 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #229 mode 2 2019/03/16-17:20:58.929339 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #233 mode 2 2019/03/16-17:20:59.818413 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #236 mode 2 2019/03/16-17:20:59.902353 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #238 mode 2 2019/03/16-17:21:00.805709 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #240 mode 2 2019/03/16-17:21:00.941272 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #244 mode 2 2019/03/16-17:21:01.669072 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #246 mode 2 2019/03/16-17:21:02.093630 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #248 mode 2 2019/03/16-17:21:02.986178 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #250 mode 2 2019/03/16-17:21:03.302178 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #254 mode 2 2019/03/16-17:21:03.843163 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #257 mode 2 2019/03/16-17:21:04.221869 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #259 mode 2 2019/03/16-17:21:04.746598 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #261 mode 2 2019/03/16-17:21:05.169535 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #265 mode 2 2019/03/16-17:21:05.582502 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #267 mode 2 2019/03/16-17:21:06.180393 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #269 mode 2 2019/03/16-17:21:06.608382 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #271 mode 2 2019/03/16-17:21:07.252003 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #275 mode 2 2019/03/16-17:21:07.595997 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #278 mode 2 2019/03/16-17:21:08.188591 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #280 mode 2 2019/03/16-17:21:08.457049 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #282 mode 2 2019/03/16-17:21:09.292141 7fdfe8a21e00 [rocksdb/db/db_impl_open.cc:551] Recovering log #286 mode 2 2019/03/16-17:21:09.903527 7fdfe8a21e00 [rocksdb/db/version_set.cc:2847] Creating manifest 289 2019/03/16-17:21:10.078405 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870078374, "cf_name": "col0", "job": 1, "event": "table_file_creation", "file_number": 290, "file_size": 8783593, "table_properties": {"data_size": 8748939, "index_size": 58813, "filter_size": 0, "raw_key_size": 9144444, "raw_average_key_size": 43, "raw_value_size": 1316, "raw_average_value_size": 0, "num_data_blocks": 2143, "num_entries": 207829, "filter_policy_name": "", "kDeletedKeys": "207668", "kMergeOperands": "0"}} 2019/03/16-17:21:10.111710 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870111682, "cf_name": "col1", "job": 1, "event": "table_file_creation", "file_number": 291, "file_size": 2734827, "table_properties": {"data_size": 2723014, "index_size": 15321, "filter_size": 0, "raw_key_size": 711840, "raw_average_key_size": 40, "raw_value_size": 1976595, "raw_average_value_size": 111, "num_data_blocks": 686, "num_entries": 17796, "filter_policy_name": "", "kDeletedKeys": "11611", "kMergeOperands": "0"}} 2019/03/16-17:21:10.244191 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870244155, "cf_name": "col2", "job": 1, "event": "table_file_creation", "file_number": 292, "file_size": 29962241, "table_properties": {"data_size": 29902844, "index_size": 94089, "filter_size": 0, "raw_key_size": 686916, "raw_average_key_size": 38, "raw_value_size": 29515206, "raw_average_value_size": 1672, "num_data_blocks": 4109, "num_entries": 17647, "filter_policy_name": "", "kDeletedKeys": "13283", "kMergeOperands": "0"}} 2019/03/16-17:21:10.464379 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870464356, "cf_name": "col3", "job": 1, "event": "table_file_creation", "file_number": 293, "file_size": 20358667, "table_properties": {"data_size": 20250765, "index_size": 156004, "filter_size": 0, "raw_key_size": 10804988, "raw_average_key_size": 26, "raw_value_size": 14957496, "raw_average_value_size": 36, "num_data_blocks": 6469, "num_entries": 415486, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:21:10.730760 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870730736, "cf_name": "col4", "job": 1, "event": "table_file_creation", "file_number": 294, "file_size": 45921033, "table_properties": {"data_size": 45771961, "index_size": 288305, "filter_size": 0, "raw_key_size": 9144388, "raw_average_key_size": 44, "raw_value_size": 36206002, "raw_average_value_size": 174, "num_data_blocks": 11555, "num_entries": 207827, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:21:10.841064 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870841046, "cf_name": "col5", "job": 1, "event": "table_file_creation", "file_number": 295, "file_size": 9705381, "table_properties": {"data_size": 9662110, "index_size": 69528, "filter_size": 0, "raw_key_size": 9144388, "raw_average_key_size": 44, "raw_value_size": 2984394, "raw_average_value_size": 14, "num_data_blocks": 2805, "num_entries": 207827, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:21:10.850952 7fdfe8a21e00 EVENT_LOG_v1 {"time_micros": 1552756870850948, "job": 1, "event": "recovery_finished"} 2019/03/16-17:21:10.963089 7fdfe8a1d700 [WARN] [rocksdb/db/db_impl.cc:491] ------- DUMPING STATS ------- 2019/03/16-17:21:10.963351 7fdfe8a1d700 [WARN] [rocksdb/db/db_impl.cc:492] ** DB Stats ** Uptime(secs): 22.8 total, 22.8 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

2019/03/16-17:51:45.073139 7fb13d6fde00 Options.write_buffer_size: 67108864 2019/03/16-17:51:45.073140 7fb13d6fde00 Options.max_write_buffer_number: 2 2019/03/16-17:51:45.073141 7fb13d6fde00 Options.compression: Snappy 2019/03/16-17:51:45.073142 7fb13d6fde00 Options.bottommost_compression: Disabled 2019/03/16-17:51:45.073143 7fb13d6fde00 Options.prefix_extractor: nullptr 2019/03/16-17:51:45.073144 7fb13d6fde00 Options.memtable_insert_with_hint_prefix_extractor: nullptr 2019/03/16-17:51:45.073152 7fb13d6fde00 Options.num_levels: 7 2019/03/16-17:51:45.073153 7fb13d6fde00 Options.min_write_buffer_number_to_merge: 1 2019/03/16-17:51:45.073154 7fb13d6fde00 Options.max_write_buffer_number_to_maintain: 0 2019/03/16-17:51:45.073155 7fb13d6fde00 Options.compression_opts.window_bits: -14 2019/03/16-17:51:45.073156 7fb13d6fde00 Options.compression_opts.level: -1 2019/03/16-17:51:45.073157 7fb13d6fde00 Options.compression_opts.strategy: 0 2019/03/16-17:51:45.073158 7fb13d6fde00 Options.compression_opts.max_dict_bytes: 0 2019/03/16-17:51:45.073159 7fb13d6fde00 Options.compression_opts.zstd_max_train_bytes: 0 2019/03/16-17:51:45.073160 7fb13d6fde00 Options.level0_file_num_compaction_trigger: 4 2019/03/16-17:51:45.073161 7fb13d6fde00 Options.level0_slowdown_writes_trigger: 20 2019/03/16-17:51:45.073162 7fb13d6fde00 Options.level0_stop_writes_trigger: 36 2019/03/16-17:51:45.073163 7fb13d6fde00 Options.target_file_size_base: 67108864 2019/03/16-17:51:45.073164 7fb13d6fde00 Options.target_file_size_multiplier: 1 2019/03/16-17:51:45.073164 7fb13d6fde00 Options.max_bytes_for_level_base: 268435456 2019/03/16-17:51:45.073165 7fb13d6fde00 Options.level_compaction_dynamic_level_bytes: 0 2019/03/16-17:51:45.073166 7fb13d6fde00 Options.max_bytes_for_level_multiplier: 10.000000 2019/03/16-17:51:45.073168 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[0]: 1 2019/03/16-17:51:45.073169 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[1]: 1 2019/03/16-17:51:45.073170 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[2]: 1 2019/03/16-17:51:45.073171 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[3]: 1 2019/03/16-17:51:45.073172 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[4]: 1 2019/03/16-17:51:45.073173 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[5]: 1 2019/03/16-17:51:45.073174 7fb13d6fde00 Options.max_bytes_for_level_multiplier_addtl[6]: 1 2019/03/16-17:51:45.073175 7fb13d6fde00 Options.max_sequential_skip_in_iterations: 8 2019/03/16-17:51:45.073176 7fb13d6fde00 Options.max_compaction_bytes: 1677721600 2019/03/16-17:51:45.073176 7fb13d6fde00 Options.arena_block_size: 8388608 2019/03/16-17:51:45.073177 7fb13d6fde00 Options.soft_pending_compaction_bytes_limit: 68719476736 2019/03/16-17:51:45.073178 7fb13d6fde00 Options.hard_pending_compaction_bytes_limit: 274877906944 2019/03/16-17:51:45.073179 7fb13d6fde00 Options.rate_limit_delay_max_milliseconds: 100 2019/03/16-17:51:45.073180 7fb13d6fde00 Options.disable_auto_compactions: 0 2019/03/16-17:51:45.073181 7fb13d6fde00 Options.compaction_style: kCompactionStyleLevel 2019/03/16-17:51:45.073182 7fb13d6fde00 Options.compaction_pri: kByCompensatedSize 2019/03/16-17:51:45.073183 7fb13d6fde00 Options.compaction_options_universal.size_ratio: 1 2019/03/16-17:51:45.073184 7fb13d6fde00 Options.compaction_options_universal.min_merge_width: 2 2019/03/16-17:51:45.073185 7fb13d6fde00 Options.compaction_options_universal.max_merge_width: 4294967295 2019/03/16-17:51:45.073186 7fb13d6fde00 Options.compaction_options_universal.max_size_amplification_percent: 200 2019/03/16-17:51:45.073187 7fb13d6fde00 Options.compaction_options_universal.compression_size_percent: -1 2019/03/16-17:51:45.073188 7fb13d6fde00 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2019/03/16-17:51:45.073189 7fb13d6fde00 Options.compaction_options_fifo.max_table_files_size: 1073741824 2019/03/16-17:51:45.073190 7fb13d6fde00 Options.compaction_options_fifo.allow_compaction: 0 2019/03/16-17:51:45.073191 7fb13d6fde00 Options.compaction_options_fifo.ttl: 0 2019/03/16-17:51:45.073192 7fb13d6fde00 Options.table_properties_collectors: 2019/03/16-17:51:45.073193 7fb13d6fde00 Options.inplace_update_support: 0 2019/03/16-17:51:45.073198 7fb13d6fde00 Options.inplace_update_num_locks: 10000 2019/03/16-17:51:45.073200 7fb13d6fde00 Options.memtable_prefix_bloom_size_ratio: 0.000000 2019/03/16-17:51:45.073201 7fb13d6fde00 Options.memtable_huge_page_size: 0 2019/03/16-17:51:45.073202 7fb13d6fde00 Options.bloom_locality: 0 2019/03/16-17:51:45.073203 7fb13d6fde00 Options.max_successive_merges: 0 2019/03/16-17:51:45.073204 7fb13d6fde00 Options.optimize_filters_for_hits: 0 2019/03/16-17:51:45.073204 7fb13d6fde00 Options.paranoid_file_checks: 0 2019/03/16-17:51:45.073205 7fb13d6fde00 Options.force_consistency_checks: 0 2019/03/16-17:51:45.073206 7fb13d6fde00 Options.report_bg_io_stats: 0 2019/03/16-17:51:45.073207 7fb13d6fde00 Options.ttl: 0 2019/03/16-17:51:45.074843 7fb13d6fde00 [rocksdb/db/version_set.cc:3361] Recovered from manifest file:/root/.local/share/joystream-node/chains/joy_testnet_1/db/MANIFEST-000289 succeeded,manifest_file_number is 289, next_file_number is 302, last_sequence is 9158877, log_number is 0,prev_log_number is 0,max_column_family is 9,min_log_number_to_keep is 0 2019/03/16-17:51:45.074848 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [default] (ID 0), log number is 287 2019/03/16-17:51:45.074850 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col0] (ID 1), log number is 287 2019/03/16-17:51:45.074851 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col1] (ID 2), log number is 287 2019/03/16-17:51:45.074852 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col2] (ID 3), log number is 287 2019/03/16-17:51:45.074853 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col3] (ID 4), log number is 287 2019/03/16-17:51:45.074854 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col4] (ID 5), log number is 287 2019/03/16-17:51:45.074855 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col5] (ID 6), log number is 287 2019/03/16-17:51:45.074856 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col6] (ID 7), log number is 287 2019/03/16-17:51:45.074857 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col7] (ID 8), log number is 287 2019/03/16-17:51:45.074858 7fb13d6fde00 [rocksdb/db/version_set.cc:3369] Column family [col8] (ID 9), log number is 287 2019/03/16-17:51:45.075036 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705075030, "job": 1, "event": "recovery_started", "log_files": [296]} 2019/03/16-17:51:45.075040 7fb13d6fde00 [rocksdb/db/db_impl_open.cc:551] Recovering log #296 mode 2 2019/03/16-17:51:45.664438 7fb13d6fde00 [rocksdb/db/version_set.cc:2847] Creating manifest 302 2019/03/16-17:51:45.679334 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705679322, "cf_name": "col0", "job": 1, "event": "table_file_creation", "file_number": 303, "file_size": 241968, "table_properties": {"data_size": 240124, "index_size": 1559, "filter_size": 0, "raw_key_size": 247556, "raw_average_key_size": 43, "raw_value_size": 44, "raw_average_value_size": 0, "num_data_blocks": 60, "num_entries": 5627, "filter_policy_name": "", "kDeletedKeys": "5625", "kMergeOperands": "0"}} 2019/03/16-17:51:45.770842 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705770814, "cf_name": "col1", "job": 1, "event": "table_file_creation", "file_number": 304, "file_size": 8980789, "table_properties": {"data_size": 8942981, "index_size": 51770, "filter_size": 0, "raw_key_size": 4367800, "raw_average_key_size": 40, "raw_value_size": 4350788, "raw_average_value_size": 39, "num_data_blocks": 2224, "num_entries": 109195, "filter_policy_name": "", "kDeletedKeys": "88522", "kMergeOperands": "0"}} 2019/03/16-17:51:45.922311 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705922261, "cf_name": "col2", "job": 1, "event": "table_file_creation", "file_number": 305, "file_size": 29759615, "table_properties": {"data_size": 29700462, "index_size": 93893, "filter_size": 0, "raw_key_size": 610516, "raw_average_key_size": 39, "raw_value_size": 29282003, "raw_average_value_size": 1875, "num_data_blocks": 4104, "num_entries": 15614, "filter_policy_name": "", "kDeletedKeys": "11260", "kMergeOperands": "0"}} 2019/03/16-17:51:45.944870 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705944854, "cf_name": "col3", "job": 1, "event": "table_file_creation", "file_number": 306, "file_size": 544305, "table_properties": {"data_size": 540573, "index_size": 4003, "filter_size": 0, "raw_key_size": 292500, "raw_average_key_size": 26, "raw_value_size": 405000, "raw_average_value_size": 36, "num_data_blocks": 176, "num_entries": 11250, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:51:45.966348 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705966332, "cf_name": "col4", "job": 1, "event": "table_file_creation", "file_number": 307, "file_size": 1241560, "table_properties": {"data_size": 1236446, "index_size": 7510, "filter_size": 0, "raw_key_size": 247500, "raw_average_key_size": 44, "raw_value_size": 980104, "raw_average_value_size": 174, "num_data_blocks": 313, "num_entries": 5625, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:51:45.979598 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758705979585, "cf_name": "col5", "job": 1, "event": "table_file_creation", "file_number": 308, "file_size": 252166, "table_properties": {"data_size": 250082, "index_size": 1824, "filter_size": 0, "raw_key_size": 247500, "raw_average_key_size": 44, "raw_value_size": 58105, "raw_average_value_size": 10, "num_data_blocks": 76, "num_entries": 5625, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}} 2019/03/16-17:51:46.005301 7fb13d6fde00 EVENT_LOG_v1 {"time_micros": 1552758706005294, "job": 1, "event": "recovery_finished"} 2019/03/16-17:51:46.033335 7fb13d6f9700 [WARN] [rocksdb/db/db_impl.cc:491] ------- DUMPING STATS ------- 2019/03/16-17:51:46.033496 7fb13d6f9700 [WARN] [rocksdb/db/db_impl.cc:492] ** DB Stats ** Uptime(secs): 1.0 total, 1.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent

posting parts of the logs i think which is useful, if you need full log let me know.

[bwhm]

Thanks for posting this @feardax. Can you have a look @mnaamani?

[mnaamani]

@feardax Thanks for sharing. I will try to see if I can make sense of the logs. Can you explain exactly what happened? Did your node just crash and did it work normally when you restarted it?

By your validator account being auto-killed I assume you mean the account was no longer in the validator list? (Which is expected if the validator node is detected to be offline)

[mnaamani]

The only thing I can gather from the logs, is there was some issue with the database. Perhaps the node crashed due to low RAM, and it left the database in an inconstant state. While the validator is running maybe you can monitor memory usage to find out if perhaps increasing your RAM to 2GB might help.

[nexusfallout]

@mnaamani Edited

Okay got it. i will check for that info.

[nexusfallout]

@mnaamani okay so you are right i see a peak in Memory usage to 90 percentage and suddenly dropping which i think when the script had got killed.

[mnaamani]

Glad you found the source of the problem. Will close for now, please reopen if you still have this issue in future.