strapdata / elassandra

Elassandra = Elasticsearch + Apache Cassandra
http://www.elassandra.io
Apache License 2.0
1.71k stars 199 forks source link

Index not found on most indexes #188

Closed cevalogistics closed 6 years ago

cevalogistics commented 6 years ago

Elasticsearch version: 5.5.0.13

Plugins installed: [] Tried to and failed

JVM version: 1.8

OS version: ubuntu 16.04

We were trying to install a plugin and it went into jarhell. We stopped elassandra (docker image) and re-downloaded the image and re-installed. Since then we are getting 404 error on most indexes.

Here is an example: We created then referenced index and got back an acknowledged. It took some time though.

Here is the post

curl http://inception.cevalogistics.com:9200/_cat/indices?v

-XPUT 'inception.cevalogistics.com:9200/driverperf_v1/driverperf_v1/1' -d ' {"Pkey":"1","STOP_ID":"383733289","TRIP_ID":"106729811","EVENT_TYPE":"ARRIVAL","SCHED_EVENT_DATE":"2017-02-13 11:59:00Z","ACTUAL_EVENT_DATE":"2017-02-13 14:00:00","LATE_STATUS":"Late","ONTIME_STATUS":"Late","EVENT_REASON":"No Status","LEG_TYPE":"Pickup","ROUTE_NAME":"Demo-Gen","DRIVER":"Wattana Arsanok","BUSINESS_UNIT":"271 CEVA Ground Thailand","SVC_PROVIDER":"CGTO","STOP_NBR":"1","SCHED_VS_ACTUAL_MINS":121,"TRIP_STATUS":"Complete","SERVICE_TIME_MINS":2,"LOCATION_TYPE":"Warehouse Facility","FINAL_DLVY_STATUS":"Completed","STOP_STATE":"BKK","STOP_COUNTRY":"THAILAND","GEO":"0,0"} '

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"IhC82nztR76akRo-uV0PRA","index":"driverperf_v1"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"IhC82nztR76akRo-uV0PRA","index":"driverperf_v1"},"status":404}

MacBook-Pro-2:~ liottar$ curl http://inception.logistics.corp:9200/_cat/indices green open poc_training PvumvsazTgyJnPisTHuRFA 6 0 2 0 13.4kb 13.4kb red open driverperf_v1 IhC82nztR76akRo-uV0PRA
green open .kibana 60q4nNzrTqSqTd5ztckASw 6 0 9 0 50.9kb 50.9kb red
open wmsskuet QHR0roXvQ66LICG3aZltZA
red open gtnx_purchase_order 0A5Tb-GXQrSpmN1oeBDhQQ
red
open prdedi Mw5gBIs8Q-2ZRp1REikJCA
red open gtnx_shipment_v1 ntyLLHQwQmqAy0Kxd3dZfg
green open gtnx_shipment_v2 zUNifXgSRKyiSv_49_FWJg 6 0 0 0 618b 618b red
open driverperf ZoF29x-5RJ-v4WGyudFa1g

I tried to re-index the ones in red but they all come back 404...

Any ideas?

vroyer commented 6 years ago

Could mention the plugin you tried to install and add your cluster state and CQL schema ? In order to re-index, did your run nodetool rebuild_index ? (You should not the elasticsearch reindex API)

Le 25 avr. 2018 à 16:32, rsliotta notifications@github.com a écrit :

Elasticsearch version: 5.5.0.13

Plugins installed: [] Tried to and failed

JVM version: 1.8

OS version: ubuntu 16.04

We were trying to install a plugin and it went into jarhell. We stopped elassandra (docker image) and re-downloaded the image and re-installed. Since then we are getting 404 error on most indexes.

Here is an example: We created then referenced index and got back an acknowledged. It took some time though.

Here is the post

curl http://inception.cevalogistics.com:9200/_cat/indices?v http://inception.cevalogistics.com:9200/_cat/indices?v -XPUT 'inception.cevalogistics.com:9200/driverperf_v1/driverperf_v1/1' -d ' {"Pkey":"1","STOP_ID":"383733289","TRIP_ID":"106729811","EVENT_TYPE":"ARRIVAL","SCHED_EVENT_DATE":"2017-02-13 11:59:00Z","ACTUAL_EVENT_DATE":"2017-02-13 14:00:00","LATE_STATUS":"Late","ONTIME_STATUS":"Late","EVENT_REASON":"No Status","LEG_TYPE":"Pickup","ROUTE_NAME":"Demo-Gen","DRIVER":"Wattana Arsanok","BUSINESS_UNIT":"271 CEVA Ground Thailand","SVC_PROVIDER":"CGTO","STOP_NBR":"1","SCHED_VS_ACTUAL_MINS":121,"TRIP_STATUS":"Complete","SERVICE_TIME_MINS":2,"LOCATION_TYPE":"Warehouse Facility","FINAL_DLVY_STATUS":"Completed","STOP_STATE":"BKK","STOP_COUNTRY":"THAILAND","GEO":"0,0"} '

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"IhC82nztR76akRo-uV0PRA","index":"driverperf_v1"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"IhC82nztR76akRo-uV0PRA","index":"driverperf_v1"},"status":404}

MacBook-Pro-2:~ liottar$ curl http://inception.logistics.corp:9200/_cat/indices http://inception.logistics.corp:9200/_cat/indices green open poc_training PvumvsazTgyJnPisTHuRFA 6 0 2 0 13.4kb 13.4kb red open driverperf_v1 IhC82nztR76akRo-uV0PRA green open .kibana 60q4nNzrTqSqTd5ztckASw 6 0 9 0 50.9kb 50.9kb red open wmsskuet QHR0roXvQ66LICG3aZltZA red open gtnx_purchase_order 0A5Tb-GXQrSpmN1oeBDhQQ red open prdedi Mw5gBIs8Q-2ZRp1REikJCA red open gtnx_shipment_v1 ntyLLHQwQmqAy0Kxd3dZfg green open gtnx_shipment_v2 zUNifXgSRKyiSv_49_FWJg 6 0 0 0 618b 618b red open driverperf ZoF29x-5RJ-v4WGyudFa1g

I tried to re-index the ones in red but they all come back 404...

Any ideas?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/strapdata/elassandra/issues/188, or mute the thread https://github.com/notifications/unsubscribe-auth/AJzHmYVlhFFsjpRYm1-MnqUtLg-Ux0Fkks5tsIjhgaJpZM4Tji1V.

cevalogistics commented 6 years ago

Vincent,

https://github.com/NLPchina/elasticsearch-sql

I DID use nodetool rebuild_index.

I believe I saw some type of indexmissing error there are well. I will try to to reproduce.

The real concern at the moment is I cannot create a new index and load data though. The create works but I see the index is immediately red.

cevalogistics commented 6 years ago

Vincent here is a log. It looks like one of the nodes is broken. Can you direct me with to the documention do I can fix it? I could not find anything about this.

Starting cassandra with org.apache.cassandra.service.ElassandraDaemon 2018-04-25 19:20:03,317 INFO [main] org.apache.cassandra.config.YamlConfigurationLoader.getStorageConfigURL(YamlConfigurationLoader.java:89) Configuration location: file:/opt/elassandra-5.5.0.13/conf/cassandra.yaml 2018-04-25 19:20:04,370 INFO [main] org.apache.cassandra.config.Config.log(Config.java:495) Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=null; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=10.233.122.153; broadcast_rpc_address=10.233.122.153; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=; cluster_name=Cassandra; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=null; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@353d0772; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_materialized_views=true; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=GossipingPropertyFileSnitch; file_cache_round_up=null; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=null; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=all; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=10.233.122.153; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=8; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=null; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=elassandra-0.elassandra.default.svc.cluster.local}; server_encryption_options=; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@2667f029; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=5000] 2018-04-25 19:20:04,373 INFO [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:367) DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap 2018-04-25 19:20:04,374 INFO [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:425) Global memtable on-heap threshold is enabled at 2043MB 2018-04-25 19:20:04,375 INFO [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:429) Global memtable off-heap threshold is enabled at 2043MB 2018-04-25 19:20:04,739 INFO [main] org.apache.cassandra.net.RateBasedBackPressure.(RateBasedBackPressure.java:123) Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 5000. 2018-04-25 19:20:04,740 INFO [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:729) Back-pressure is disabled with strategy null. 2018-04-25 19:20:05,195 INFO [main] org.apache.cassandra.locator.GossipingPropertyFileSnitch.(GossipingPropertyFileSnitch.java:64) Loaded cassandra-topology.properties for compatibility 2018-04-25 19:20:10,300 INFO [ScheduledTasks:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:498) Updating topology for all endpoints that have changed 2018-04-25 19:20:14,127 INFO [main] org.apache.cassandra.cql3.QueryProcessor.(QueryProcessor.java:116) Initialized prepared statement caches with 31 MB (native) and 31 MB (Thrift) 2018-04-25 19:20:16,641 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.IndexInfo 2018-04-25 19:20:18,228 INFO [SSTableBatchOpen:1] org.apache.cassandra.utils.memory.BufferPool$GlobalPool.(BufferPool.java:230) Global buffer pool is enabled, when pool is exhausted (max is 512.000MiB) it will allocate on heap 2018-04-25 19:20:18,682 INFO [main] org.apache.cassandra.service.CacheService.initKeyCache(CacheService.java:112) Initializing key cache with capacity of 100 MBs. 2018-04-25 19:20:18,698 INFO [main] org.apache.cassandra.service.CacheService.initRowCache(CacheService.java:134) Initializing row cache with capacity of 0 MBs 2018-04-25 19:20:18,702 INFO [main] org.apache.cassandra.service.CacheService.initCounterCache(CacheService.java:163) Initializing counter cache with capacity of 50 MBs 2018-04-25 19:20:18,705 INFO [main] org.apache.cassandra.service.CacheService.initCounterCache(CacheService.java:174) Scheduling counter cache save to every 7200 seconds (going to save all keys). 2018-04-25 19:20:18,761 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.batches 2018-04-25 19:20:18,775 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.paxos 2018-04-25 19:20:18,819 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.local 2018-04-25 19:20:18,867 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.peers 2018-04-25 19:20:18,914 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.peer_events 2018-04-25 19:20:18,929 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.range_xfers 2018-04-25 19:20:18,947 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.compaction_history 2018-04-25 19:20:18,991 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.sstable_activity 2018-04-25 19:20:19,054 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.size_estimates 2018-04-25 19:20:19,112 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.available_ranges 2018-04-25 19:20:19,137 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.transferred_ranges 2018-04-25 19:20:19,155 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.views_builds_in_progress 2018-04-25 19:20:19,179 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.built_views 2018-04-25 19:20:19,192 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.hints 2018-04-25 19:20:19,212 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.batchlog 2018-04-25 19:20:19,226 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.prepared_statements 2018-04-25 19:20:19,260 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_keyspaces 2018-04-25 19:20:19,272 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_columnfamilies 2018-04-25 19:20:19,285 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_columns 2018-04-25 19:20:19,450 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_triggers 2018-04-25 19:20:19,464 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_usertypes 2018-04-25 19:20:19,474 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_functions 2018-04-25 19:20:19,492 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system.schema_aggregates 2018-04-25 19:20:19,498 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace system as storage service is not initialized 2018-04-25 19:20:19,685 INFO [main] org.apache.cassandra.db.monitoring.ApproximateTime.(ApproximateTime.java:44) Scheduling approximate time-check task with a precision of 10 milliseconds 2018-04-25 19:20:26,796 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/QHR0roXvQ66LICG3aZltZA] no index state found - ignoring 2018-04-25 19:20:26,801 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/Mw5gBIs8Q-2ZRp1REikJCA] no index state found - ignoring 2018-04-25 19:20:26,803 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/ntyLLHQwQmqAy0Kxd3dZfg] no index state found - ignoring 2018-04-25 19:20:26,805 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/8qbvVZlGR_6fo7pmJqtwaA] no index state found - ignoring 2018-04-25 19:20:26,806 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/zUNifXgSRKyiSv_49_FWJg] no index state found - ignoring 2018-04-25 19:20:26,807 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/0A5Tb-GXQrSpmN1oeBDhQQ] no index state found - ignoring 2018-04-25 19:20:26,809 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/IhC82nztR76akRo-uV0PRA] no index state found - ignoring 2018-04-25 19:20:26,812 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/ZoF29x-5RJ-v4WGyudFa1g] no index state found - ignoring 2018-04-25 19:20:26,815 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/60q4nNzrTqSqTd5ztckASw] no index state found - ignoring 2018-04-25 19:20:26,821 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/PvumvsazTgyJnPisTHuRFA] no index state found - ignoring 2018-04-25 19:20:26,824 WARN [main] org.elasticsearch.common.util.IndexFolderUpgrader.upgrade(IndexFolderUpgrader.java:117) [/opt/elassandra-5.5.0.13/data/elasticsearch.data/nodes/0/indices/_a1JvbgAR3-NPZdRUzywAg] no index state found - ignoring 2018-04-25 19:20:27,279 INFO [main] org.apache.cassandra.utils.JMXServerUtils.logJmxServiceUrl(JMXServerUtils.java:246) Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi 2018-04-25 19:20:27,298 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:481) Hostname: elassandra-0.elassandra.default.svc.cluster.local 2018-04-25 19:20:27,299 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:488) JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_162 2018-04-25 19:20:27,300 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:489) Heap size: 7.980GiB/7.980GiB 2018-04-25 19:20:27,302 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) Code Cache Non-heap memory: init = 2555904(2496K) used = 11403072(11135K) committed = 11599872(11328K) max = 251658240(245760K) 2018-04-25 19:20:27,307 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) Metaspace Non-heap memory: init = 0(0K) used = 59403232(58010K) committed = 62656512(61188K) max = -1(-1K) 2018-04-25 19:20:27,307 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) Compressed Class Space Non-heap memory: init = 0(0K) used = 8445912(8247K) committed = 9617408(9392K) max = 1073741824(1048576K) 2018-04-25 19:20:27,308 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) Par Eden Space Heap memory: init = 167772160(163840K) used = 167772160(163840K) committed = 167772160(163840K) max = 167772160(163840K) 2018-04-25 19:20:27,308 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) Par Survivor Space Heap memory: init = 20971520(20480K) used = 20971520(20480K) committed = 20971520(20480K) max = 20971520(20480K) 2018-04-25 19:20:27,310 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:494) CMS Old Gen Heap memory: init = 8380219392(8183808K) used = 21590432(21084K) committed = 8380219392(8183808K) max = 8380219392(8183808K) 2018-04-25 19:20:27,313 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:496) Classpath: /opt/elassandra-5.5.0.13/conf:/opt/elassandra-5.5.0.13/lib/HdrHistogram-2.1.9.jar:/opt/elassandra-5.5.0.13/lib/ST4-4.0.8.jar:/opt/elassandra-5.5.0.13/lib/airline-0.6.jar:/opt/elassandra-5.5.0.13/lib/ant-1.7.0.jar:/opt/elassandra-5.5.0.13/lib/ant-launcher-1.7.0.jar:/opt/elassandra-5.5.0.13/lib/antlr-3.5.2.jar:/opt/elassandra-5.5.0.13/lib/antlr-runtime-3.5.2.jar:/opt/elassandra-5.5.0.13/lib/asm-5.0.4.jar:/opt/elassandra-5.5.0.13/lib/caffeine-2.2.6.jar:/opt/elassandra-5.5.0.13/lib/cassandra-all-3.11.2.jar:/opt/elassandra-5.5.0.13/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/elassandra-5.5.0.13/lib/cassandra-thrift-3.11.2.jar:/opt/elassandra-5.5.0.13/lib/commons-cli-1.3.1.jar:/opt/elassandra-5.5.0.13/lib/commons-codec-1.10.jar:/opt/elassandra-5.5.0.13/lib/commons-lang3-3.3.1.jar:/opt/elassandra-5.5.0.13/lib/commons-logging-1.2.jar:/opt/elassandra-5.5.0.13/lib/commons-math3-3.2.jar:/opt/elassandra-5.5.0.13/lib/compress-lzf-0.8.4.jar:/opt/elassandra-5.5.0.13/lib/concurrent-trees-2.4.0.jar:/opt/elassandra-5.5.0.13/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/elassandra-5.5.0.13/lib/disruptor-3.0.1.jar:/opt/elassandra-5.5.0.13/lib/ecj-4.4.2.jar:/opt/elassandra-5.5.0.13/lib/elasticsearch-5.5.0.jar:/opt/elassandra-5.5.0.13/lib/fastutil-6.5.7.jar:/opt/elassandra-5.5.0.13/lib/guava-19.0.jar:/opt/elassandra-5.5.0.13/lib/hibernate-validator-4.3.0.Final.jar:/opt/elassandra-5.5.0.13/lib/high-scale-lib-1.0.6.jar:/opt/elassandra-5.5.0.13/lib/hppc-0.7.1.jar:/opt/elassandra-5.5.0.13/lib/httpclient-4.5.2.jar:/opt/elassandra-5.5.0.13/lib/httpcore-4.4.5.jar:/opt/elassandra-5.5.0.13/lib/jackson-core-2.8.6.jar:/opt/elassandra-5.5.0.13/lib/jackson-core-asl-1.9.13.jar:/opt/elassandra-5.5.0.13/lib/jackson-dataformat-cbor-2.8.6.jar:/opt/elassandra-5.5.0.13/lib/jackson-dataformat-smile-2.8.6.jar:/opt/elassandra-5.5.0.13/lib/jackson-dataformat-yaml-2.8.6.jar:/opt/elassandra-5.5.0.13/lib/jackson-mapper-asl-1.9.13.jar:/opt/elassandra-5.5.0.13/lib/java-version-checker-5.5.0.jar:/opt/elassandra-5.5.0.13/lib/javassist-3.20.0-GA.jar:/opt/elassandra-5.5.0.13/lib/javax.inject-1.jar:/opt/elassandra-5.5.0.13/lib/jbcrypt-0.3m.jar:/opt/elassandra-5.5.0.13/lib/jboss-logging-3.1.0.CR2.jar:/opt/elassandra-5.5.0.13/lib/jcl-over-slf4j-1.7.7.jar:/opt/elassandra-5.5.0.13/lib/jctools-core-1.2.1.jar:/opt/elassandra-5.5.0.13/lib/jflex-1.6.0.jar:/opt/elassandra-5.5.0.13/lib/jna-4.4.0.jar:/opt/elassandra-5.5.0.13/lib/joda-time-2.9.5.jar:/opt/elassandra-5.5.0.13/lib/jopt-simple-5.0.2.jar:/opt/elassandra-5.5.0.13/lib/json-simple-1.1.jar:/opt/elassandra-5.5.0.13/lib/jts-1.13.jar:/opt/elassandra-5.5.0.13/lib/libthrift-0.9.2.jar:/opt/elassandra-5.5.0.13/lib/log4j-api-2.8.2.jar:/opt/elassandra-5.5.0.13/lib/log4j-core-2.8.2.jar:/opt/elassandra-5.5.0.13/lib/log4j-over-slf4j-1.7.7.jar:/opt/elassandra-5.5.0.13/lib/log4j-to-slf4j-2.8.2.jar:/opt/elassandra-5.5.0.13/lib/logback-classic-1.1.8.jar:/opt/elassandra-5.5.0.13/lib/logback-core-1.1.8.jar:/opt/elassandra-5.5.0.13/lib/lucene-analyzers-common-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-backward-codecs-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-core-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-grouping-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-highlighter-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-join-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-memory-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-misc-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-queries-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-queryparser-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-sandbox-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-spatial-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-spatial-extras-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-spatial3d-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lucene-suggest-6.6.0.jar:/opt/elassandra-5.5.0.13/lib/lz4-1.3.0.jar:/opt/elassandra-5.5.0.13/lib/metrics-core-3.1.0.jar:/opt/elassandra-5.5.0.13/lib/metrics-jvm-3.1.0.jar:/opt/elassandra-5.5.0.13/lib/netty-all-4.1.12.Final.jar:/opt/elassandra-5.5.0.13/lib/ohc-core-0.4.4.jar:/opt/elassandra-5.5.0.13/lib/plugin-cli-5.5.0.jar:/opt/elassandra-5.5.0.13/lib/reporter-config-base-3.0.3.jar:/opt/elassandra-5.5.0.13/lib/reporter-config3-3.0.3.jar:/opt/elassandra-5.5.0.13/lib/securesm-1.1.jar:/opt/elassandra-5.5.0.13/lib/sigar-1.6.4.jar:/opt/elassandra-5.5.0.13/lib/slf4j-api-1.7.25.jar:/opt/elassandra-5.5.0.13/lib/snakeyaml-1.15.jar:/opt/elassandra-5.5.0.13/lib/snappy-java-1.1.1.7.jar:/opt/elassandra-5.5.0.13/lib/spatial4j-0.6.jar:/opt/elassandra-5.5.0.13/lib/stream-2.5.2.jar:/opt/elassandra-5.5.0.13/lib/t-digest-3.0.jar:/opt/elassandra-5.5.0.13/lib/thrift-server-0.3.7.jar:/opt/elassandra-5.5.0.13/lib/validation-api-1.0.0.GA.jar:/opt/elassandra-5.5.0.13/lib/jamm-0.3.0.jar 2018-04-25 19:20:27,314 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:498) JVM Arguments: [-Djava.library.path=/opt/elassandra-5.5.0.13/lib/sigar-bin, -Xloggc:/opt/elassandra-5.5.0.13/logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms8192M, -Xmx8192M, -Xmn200M, -XX:CompileCommandFile=/opt/elassandra-5.5.0.13/conf/hotspot_compiler, -javaagent:/opt/elassandra-5.5.0.13/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/elassandra-5.5.0.13/lib/sigar-bin, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -Djava.awt.headless=true, -Dlogback.configurationFile=/opt/elassandra-5.5.0.13/conf/logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/opt/elassandra-5.5.0.13/data, -Dcassandra-pidfile=/opt/elassandra-5.5.0.13/cassandra.pid, -Dcassandra-foreground=yes] 2018-04-25 19:20:27,775 INFO [main] org.apache.cassandra.utils.NativeLibrary.tryMlockall(NativeLibrary.java:174) JNA mlockall successful 2018-04-25 19:20:27,777 INFO [main] org.apache.cassandra.service.StartupChecks$1.execute(StartupChecks.java:140) jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 2018-04-25 19:20:27,778 WARN [main] org.apache.cassandra.service.StartupChecks$3.execute(StartupChecks.java:169) JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. 2018-04-25 19:20:27,782 WARN [main] org.apache.cassandra.service.StartupChecks$5.checkOutOfMemoryHandling(StartupChecks.java:220) The JVM is not configured to stop on OutOfMemoryError which can cause data corruption. Use one of the following JVM options to configure the behavior on OutOfMemoryError: -XX:+ExitOnOutOfMemoryError, -XX:+CrashOnOutOfMemoryError, or -XX:OnOutOfMemoryError=";" 2018-04-25 19:20:27,788 INFO [main] org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:44) Initializing SIGAR library OpenJDK 64-Bit Server VM warning: You have loaded library /opt/elassandra-5.5.0.13/lib/sigar-bin/libsigar-amd64-linux.so which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. 2018-04-25 19:20:27,838 INFO [main] org.apache.cassandra.utils.SigarLibrary.(SigarLibrary.java:57) Could not initialize SIGAR library org.hyperic.sigar.Sigar.getFileSystemListNative()[Lorg/hyperic/sigar/FileSystem; 2018-04-25 19:20:27,838 INFO [main] org.apache.cassandra.utils.SigarLibrary.warnIfRunningInDegradedMode(SigarLibrary.java:185) Sigar could not be initialized, test for checking degraded mode omitted. 2018-04-25 19:20:27,840 WARN [main] org.apache.cassandra.service.StartupChecks$8.execute(StartupChecks.java:311) Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl. 2018-04-25 19:20:28,236 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.keyspaces 2018-04-25 19:20:28,329 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.tables 2018-04-25 19:20:28,411 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.columns 2018-04-25 19:20:28,468 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.triggers 2018-04-25 19:20:28,551 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.dropped_columns 2018-04-25 19:20:28,615 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.views 2018-04-25 19:20:28,691 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.types 2018-04-25 19:20:28,743 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.functions 2018-04-25 19:20:28,896 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.aggregates 2018-04-25 19:20:28,937 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_schema.indexes 2018-04-25 19:20:28,953 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace system_schema as storage service is not initialized 2018-04-25 19:20:29,942 INFO [main] org.apache.cassandra.service.StorageService.populateTokenMetadata(StorageService.java:600) Populating token metadata from system tables 2018-04-25 19:20:30,012 INFO [main] org.apache.cassandra.service.StorageService.populateTokenMetadata(StorageService.java:607) Token metadata: Normal Tokens: /10.233.90.50:[-4455400665633705009, -3307979720416438909, -2177627868341832852, -1794129631622032748, 624939019991921312, 5488838871775427570, 5915936481258352663, 6228162807192115475] /10.233.97.184:[-8248027387514132260, -7318731352128861107, -861916047897630515, 3049595437829094990, 3681862537357852614, 7941763563148319339, 8234545346034199838, 9073971229396420877] /10.233.114.136:[-9048183706652428987, -5086691515524827064, -4260497491636981485, -4164481151685738517, -1054867843544454931, 4437679225950754397, 5141005615383236012, 6372063045572782995] /10.233.117.250:[-7227302454714817399, -5939375512383729793, -2206250258799490034, 1732568509439143634, 2600887383460939089, 4748189529193459068, 8796692000955143727, 9083020370715658495] /10.233.121.107:[-3299701738014063480, -2539197605680580109, -1839508776458638598, -118864622025183534, 1111019062840381676, 6132333345085630748, 6774441214942613853, 8907766337491345741] /10.233.122.153:[-8592977458636693389, -6090579284089889517, -362226123124935217, 4451716466158730928, 4631534558835767927, 6275689832852561075, 8401438299379969757, 8914019774812791023]

2018-04-25 19:20:31,490 INFO [main] org.apache.cassandra.service.ElassandraDaemon.activateAndWaitShards(ElassandraDaemon.java:366) Activating Elasticsearch, shards starting before opening user keyspaces 2018-04-25 19:20:32,618 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.wmsskuet' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:32,620 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.wmsskuet' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,560 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.prdedi' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,563 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.prdedi' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,644 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query..kibana' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,646 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch..kibana' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,866 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.driverperf_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:33,871 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.driverperf_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,008 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.bob_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,011 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.bob_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,088 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.gtnx_shipment_v2' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,092 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.gtnx_shipment_v2' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,160 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.gtnx_purchase_order' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,161 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.gtnx_purchase_order' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,351 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.poc_training' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,352 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.poc_training' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,462 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.gtnx_shipment_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,463 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.gtnx_shipment_v1' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,604 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.query.driverperf' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:34,605 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.apache.cassandra.service.StorageService.setLoggingLevel(StorageService.java:3846) set log level to TRACE for classes under 'index.search.slowlog.fetch.driverperf' (if the level doesn't look like 'TRACE' then the logger couldn't parse 'TRACE') 2018-04-25 19:20:35,541 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elassandra.gateway.CassandraGatewayService$GatewayRecoveryListener$1.clusterStateProcessed(CassandraGatewayService.java:134) Recovered [10] indices into cluster_state 2018-04-25 19:20:35,545 INFO [main] org.apache.cassandra.service.ElassandraDaemon.activateAndWaitShards(ElassandraDaemon.java:370) Elasticsearch shards started, ready to go on. 2018-04-25 19:20:36,043 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing driverperf.driverperf 2018-04-25 19:20:36,066 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace driverperf as storage service is not initialized 2018-04-25 19:20:36,092 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing wmsskuet.wmsskuet 2018-04-25 19:20:36,389 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace wmsskuet as storage service is not initialized 2018-04-25 19:20:36,406 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_distributed.parent_repair_history 2018-04-25 19:20:36,462 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_distributed.repair_history 2018-04-25 19:20:36,502 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_distributed.view_build_status 2018-04-25 19:20:36,519 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized 2018-04-25 19:20:36,547 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing prdedi.prdedi 2018-04-25 19:20:36,870 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace prdedi as storage service is not initialized 2018-04-25 19:20:36,883 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing gtnx_shipment_v2.gtnx_shipment_v2 2018-04-25 19:20:36,898 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace gtnx_shipment_v2 as storage service is not initialized 2018-04-25 19:20:36,918 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing gtnx_purchase_order.gtnx_purchase_order 2018-04-25 19:20:36,952 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace gtnx_purchase_order as storage service is not initialized 2018-04-25 19:20:36,981 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing gtnx_shipment_v1.gtnx_shipment_v1 2018-04-25 19:20:37,017 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace gtnx_shipment_v1 as storage service is not initialized 2018-04-25 19:20:37,037 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.config 2018-04-25 19:20:37,055 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.dashboard 2018-04-25 19:20:37,077 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.index_pattern 2018-04-25 19:20:37,099 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.search 2018-04-25 19:20:37,120 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.server 2018-04-25 19:20:37,139 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.timelion_sheet 2018-04-25 19:20:37,153 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.url 2018-04-25 19:20:37,183 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing _kibana.visualization 2018-04-25 19:20:37,188 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace _kibana as storage service is not initialized 2018-04-25 19:20:37,213 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing elastic_admin.metadata 2018-04-25 19:20:37,283 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace elastic_admin as storage service is not initialized 2018-04-25 19:20:37,298 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing driverperf_v1.driverperf_v1 2018-04-25 19:20:37,308 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace driverperf_v1 as storage service is not initialized 2018-04-25 19:20:37,320 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing bob_v1.bob_v1 2018-04-25 19:20:37,333 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace bob_v1 as storage service is not initialized 2018-04-25 19:20:37,347 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_auth.resource_role_permissons_index 2018-04-25 19:20:37,364 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_auth.role_members 2018-04-25 19:20:37,379 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_auth.role_permissions 2018-04-25 19:20:37,401 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_auth.roles 2018-04-25 19:20:37,410 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace system_auth as storage service is not initialized 2018-04-25 19:20:37,429 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing poc_training.data 2018-04-25 19:20:37,443 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing poc_training.poc_training 2018-04-25 19:20:37,452 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace poc_training as storage service is not initialized 2018-04-25 19:20:37,474 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_traces.events 2018-04-25 19:20:37,489 INFO [main] org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:411) Initializing system_traces.sessions 2018-04-25 19:20:37,493 INFO [main] org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:137) Not submitting build tasks for views in keyspace system_traces as storage service is not initialized 2018-04-25 19:20:37,619 INFO [pool-5-thread-1] org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:197) reading saved cache /opt/elassandra-5.5.0.13/data/saved_caches/KeyCache-e.db 2018-04-25 19:20:37,664 INFO [pool-5-thread-1] org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:262) Harmless error reading saved cache /opt/elassandra-5.5.0.13/data/saved_caches/KeyCache-e.db java.lang.RuntimeException: Cache schema version 0db054b9-5d37-38bd-9793-59eed856ef27 does not match current schema version 98a7b463-83da-373a-a5c8-36a1adeb9e48 at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:206) at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:164) at org.apache.cassandra.cache.AutoSavingCache$3.call(AutoSavingCache.java:160) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-04-25 19:20:37,670 INFO [pool-5-thread-1] org.apache.cassandra.cache.AutoSavingCache$4.run(AutoSavingCache.java:173) Completed loading (111 ms; 88 keys) KeyCache cache 2018-04-25 19:20:37,711 INFO [main] org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:157) Replaying /opt/elassandra-5.5.0.13/data/commitlog/CommitLog-6-1524606157180.log 2018-04-25 19:20:37,768 INFO [main] org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:159) Log replay complete, 0 replayed mutations 2018-04-25 19:20:37,775 INFO [main] org.apache.cassandra.service.StorageService.populateTokenMetadata(StorageService.java:600) Populating token metadata from system tables 2018-04-25 19:20:37,794 INFO [main] org.apache.cassandra.service.StorageService.populateTokenMetadata(StorageService.java:607) Token metadata: Normal Tokens: /10.233.90.50:[-4455400665633705009, -3307979720416438909, -2177627868341832852, -1794129631622032748, 624939019991921312, 5488838871775427570, 5915936481258352663, 6228162807192115475] /10.233.97.184:[-8248027387514132260, -7318731352128861107, -861916047897630515, 3049595437829094990, 3681862537357852614, 7941763563148319339, 8234545346034199838, 9073971229396420877] /10.233.114.136:[-9048183706652428987, -5086691515524827064, -4260497491636981485, -4164481151685738517, -1054867843544454931, 4437679225950754397, 5141005615383236012, 6372063045572782995] /10.233.117.250:[-7227302454714817399, -5939375512383729793, -2206250258799490034, 1732568509439143634, 2600887383460939089, 4748189529193459068, 8796692000955143727, 9083020370715658495] /10.233.121.107:[-3299701738014063480, -2539197605680580109, -1839508776458638598, -118864622025183534, 1111019062840381676, 6132333345085630748, 6774441214942613853, 8907766337491345741] /10.233.122.153:[-8592977458636693389, -6090579284089889517, -362226123124935217, 4451716466158730928, 4631534558835767927, 6275689832852561075, 8401438299379969757, 8914019774812791023]

2018-04-25 19:20:38,078 INFO [Service Thread] org.apache.cassandra.service.GCInspector.handleNotification(GCInspector.java:284) ParNew GC in 219ms. CMS Old Gen: 24747656 -> 38169280; Par Eden Space: 167362360 -> 0; 2018-04-25 19:20:38,358 INFO [main] org.apache.cassandra.cql3.QueryProcessor.preloadPreparedStatement(QueryProcessor.java:163) Preloaded 8 prepared statements 2018-04-25 19:20:38,360 INFO [main] org.apache.cassandra.service.StorageService.initServer(StorageService.java:618) Cassandra version: 3.11.2 2018-04-25 19:20:38,361 INFO [main] org.apache.cassandra.service.StorageService.initServer(StorageService.java:619) Thrift API version: 20.1.0 2018-04-25 19:20:38,361 INFO [main] org.apache.cassandra.service.StorageService.initServer(StorageService.java:620) CQL supported versions: 3.4.4 (default: 3.4.4) 2018-04-25 19:20:38,362 INFO [main] org.apache.cassandra.service.StorageService.initServer(StorageService.java:622) Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4) 2018-04-25 19:20:38,468 INFO [main] org.apache.cassandra.io.sstable.IndexSummaryManager.(IndexSummaryManager.java:85) Initializing index summary manager with a memory pool size of 408 MB and a resize interval of 60 minutes 2018-04-25 19:20:38,483 INFO [main] org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:753) Starting Messaging Service on /10.233.122.153:7000 (eth0) 2018-04-25 19:20:38,559 INFO [main] org.apache.cassandra.service.StorageService.loadRingState(StorageService.java:707) Loading persisted ring state 2018-04-25 19:20:38,586 INFO [main] org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:825) Starting up server gossip 2018-04-25 19:20:38,757 INFO [main] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.122.153 2018-04-25 19:20:38,761 INFO [main] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.122.153 2018-04-25 19:20:38,881 INFO [main] org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:994) Using saved tokens [-362226123124935217, -6090579284089889517, -8592977458636693389, 4451716466158730928, 4631534558835767927, 6275689832852561075, 8401438299379969757, 8914019774812791023] 2018-04-25 19:20:38,910 INFO [main] org.apache.cassandra.service.StorageService.setMode(StorageService.java:1455) JOINING: Finish joining ring 2018-04-25 19:20:38,975 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='driverperf', ColumnFamily='driverperf') 2018-04-25 19:20:38,980 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='wmsskuet', ColumnFamily='wmsskuet') 2018-04-25 19:20:38,987 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='prdedi', ColumnFamily='prdedi') 2018-04-25 19:20:38,988 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='gtnx_shipment_v2', ColumnFamily='gtnx_shipment_v2') 2018-04-25 19:20:38,991 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='gtnx_purchase_order', ColumnFamily='gtnx_purchase_order') 2018-04-25 19:20:38,992 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='gtnx_shipment_v1', ColumnFamily='gtnx_shipment_v1') 2018-04-25 19:20:38,993 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='config') 2018-04-25 19:20:38,993 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='dashboard') 2018-04-25 19:20:38,996 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='index_pattern') 2018-04-25 19:20:38,996 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='search') 2018-04-25 19:20:38,997 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='server') 2018-04-25 19:20:38,997 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='url') 2018-04-25 19:20:38,998 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='visualization') 2018-04-25 19:20:38,998 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='_kibana', ColumnFamily='timelion_sheet') 2018-04-25 19:20:38,999 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='elastic_admin', ColumnFamily='metadata') 2018-04-25 19:20:39,001 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='driverperf_v1', ColumnFamily='driverperf_v1') 2018-04-25 19:20:39,002 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='bob_v1', ColumnFamily='bob_v1') 2018-04-25 19:20:39,002 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='poc_training', ColumnFamily='data') 2018-04-25 19:20:39,003 INFO [main] org.apache.cassandra.index.SecondaryIndexManager.executePreJoinTasksBlocking(SecondaryIndexManager.java:518) Executing pre-join tasks for: CFS(Keyspace='poc_training', ColumnFamily='poc_training') 2018-04-25 19:20:39,081 INFO [main] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.122.153 state jump to NORMAL 2018-04-25 19:20:39,109 INFO [main] org.apache.cassandra.gms.Gossiper.waitToSettle(Gossiper.java:1670) Waiting for gossip to settle... 2018-04-25 19:20:39,784 INFO [GossipTasks:1] org.apache.cassandra.net.OutboundTcpConnection.(OutboundTcpConnection.java:108) OutboundTcpConnection using coalescing strategy DISABLED 2018-04-25 19:20:39,840 INFO [HANDSHAKE-/10.233.97.184] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.97.184 2018-04-25 19:20:39,897 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1053) Node /10.233.114.136 has restarted, now UP 2018-04-25 19:20:39,917 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.114.136 state jump to NORMAL 2018-04-25 19:20:39,920 INFO [HANDSHAKE-/10.233.114.136] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.114.136 2018-04-25 19:20:39,931 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.114.136 2018-04-25 19:20:39,932 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.114.136 2018-04-25 19:20:39,934 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1053) Node /10.233.97.184 has restarted, now UP 2018-04-25 19:20:39,938 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.97.184 state jump to NORMAL 2018-04-25 19:20:39,940 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.97.184 2018-04-25 19:20:39,941 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.97.184 2018-04-25 19:20:39,943 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1053) Node /10.233.121.107 has restarted, now UP 2018-04-25 19:20:39,997 INFO [RequestResponseStage-2] org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:1019) InetAddress /10.233.114.136 is now UP 2018-04-25 19:20:40,006 INFO [RequestResponseStage-1] org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:1019) InetAddress /10.233.97.184 is now UP 2018-04-25 19:20:40,016 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.121.107 state jump to NORMAL 2018-04-25 19:20:40,024 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.121.107 2018-04-25 19:20:40,025 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.121.107 2018-04-25 19:20:40,028 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1053) Node /10.233.117.250 has restarted, now UP 2018-04-25 19:20:40,067 INFO [HANDSHAKE-/10.233.121.107] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.121.107 2018-04-25 19:20:40,072 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.117.250 state jump to NORMAL 2018-04-25 19:20:40,077 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.117.250 2018-04-25 19:20:40,094 INFO [HANDSHAKE-/10.233.117.250] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.117.250 2018-04-25 19:20:40,099 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.117.250 2018-04-25 19:20:40,107 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1053) Node /10.233.90.50 has restarted, now UP 2018-04-25 19:20:40,121 INFO [RequestResponseStage-4] org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:1019) InetAddress /10.233.121.107 is now UP 2018-04-25 19:20:40,155 INFO [RequestResponseStage-1] org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:1019) InetAddress /10.233.117.250 is now UP 2018-04-25 19:20:40,160 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2303) Node /10.233.90.50 state jump to NORMAL 2018-04-25 19:20:40,171 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.90.50 2018-04-25 19:20:40,179 INFO [GossipStage:1] org.apache.cassandra.locator.TokenMetadata.updateTopology(TokenMetadata.java:479) Updating topology for /10.233.90.50 2018-04-25 19:20:40,198 INFO [HANDSHAKE-/10.233.90.50] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.90.50 2018-04-25 19:20:40,228 INFO [RequestResponseStage-1] org.apache.cassandra.gms.Gossiper.realMarkAlive(Gossiper.java:1019) InetAddress /10.233.90.50 is now UP 2018-04-25 19:20:40,249 INFO [HANDSHAKE-/10.233.97.184] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.97.184 2018-04-25 19:20:40,445 INFO [HANDSHAKE-/10.233.90.50] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.90.50 2018-04-25 19:20:40,814 WARN [GossipTasks:1] org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:288) Not marking nodes down due to local pause of 30595253465 > 5000000000 2018-04-25 19:20:41,614 INFO [HANDSHAKE-/10.233.121.107] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.121.107 2018-04-25 19:20:41,914 INFO [HANDSHAKE-/10.233.117.250] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.117.250 2018-04-25 19:20:42,268 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1055) Node /10.233.122.152 is now part of the cluster 2018-04-25 19:20:42,295 WARN [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2342) Not updating host ID 62891ec9-7de6-4d55-b2cb-7eae97ed2957 for /10.233.122.152 because it's mine 2018-04-25 19:20:42,304 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token -362226123124935217. Ignoring /10.233.122.152 2018-04-25 19:20:42,307 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token -6090579284089889517. Ignoring /10.233.122.152 2018-04-25 19:20:42,311 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token -8592977458636693389. Ignoring /10.233.122.152 2018-04-25 19:20:42,314 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token 4451716466158730928. Ignoring /10.233.122.152 2018-04-25 19:20:42,322 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token 4631534558835767927. Ignoring /10.233.122.152 2018-04-25 19:20:42,331 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token 6275689832852561075. Ignoring /10.233.122.152 2018-04-25 19:20:42,332 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token 8401438299379969757. Ignoring /10.233.122.152 2018-04-25 19:20:42,333 INFO [GossipStage:1] org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:2400) Nodes /10.233.122.152 and /10.233.122.153 have the same token 8914019774812791023. Ignoring /10.233.122.152 2018-04-25 19:20:42,407 INFO [GossipStage:1] org.apache.cassandra.gms.Gossiper.markDead(Gossiper.java:1034) InetAddress /10.233.122.152 is now DOWN 2018-04-25 19:20:42,633 INFO [HANDSHAKE-/10.233.114.136] org.apache.cassandra.net.OutboundTcpConnection.lambda$handshakeVersion$1(OutboundTcpConnection.java:560) Handshaking version with /10.233.114.136 2018-04-25 19:20:47,111 INFO [main] org.apache.cassandra.gms.Gossiper.waitToSettle(Gossiper.java:1701) No gossip backlog; proceeding 2018-04-25 19:20:47,455 INFO [main] org.apache.cassandra.service.NativeTransportService.initialize(NativeTransportService.java:75) Netty using Java NIO event loop 2018-04-25 19:20:47,665 INFO [main] org.apache.cassandra.transport.Server.start(Server.java:155) Using Netty Version: [netty-buffer=netty-buffer-4.1.12.Final.3acd5c68ea, netty-codec=netty-codec-4.1.12.Final.3acd5c68ea, netty-codec-dns=netty-codec-dns-4.1.12.Final.3acd5c68ea, netty-codec-haproxy=netty-codec-haproxy-4.1.12.Final.3acd5c68ea, netty-codec-http=netty-codec-http-4.1.12.Final.3acd5c68ea, netty-codec-http2=netty-codec-http2-4.1.12.Final.3acd5c68ea, netty-codec-memcache=netty-codec-memcache-4.1.12.Final.3acd5c68ea, netty-codec-mqtt=netty-codec-mqtt-4.1.12.Final.3acd5c68ea, netty-codec-redis=netty-codec-redis-4.1.12.Final.3acd5c68ea, netty-codec-smtp=netty-codec-smtp-4.1.12.Final.3acd5c68ea, netty-codec-socks=netty-codec-socks-4.1.12.Final.3acd5c68ea, netty-codec-stomp=netty-codec-stomp-4.1.12.Final.3acd5c68ea, netty-codec-xml=netty-codec-xml-4.1.12.Final.3acd5c68ea, netty-common=netty-common-4.1.12.Final.3acd5c68ea, netty-handler=netty-handler-4.1.12.Final.3acd5c68ea, netty-handler-proxy=netty-handler-proxy-4.1.12.Final.3acd5c68ea, netty-resolver=netty-resolver-4.1.12.Final.3acd5c68ea, netty-resolver-dns=netty-resolver-dns-4.1.12.Final.3acd5c68ea, netty-tcnative=netty-tcnative-2.0.3.Final.c3e60ce, netty-transport=netty-transport-4.1.12.Final.3acd5c68ea, netty-transport-native-epoll=netty-transport-native-epoll-4.1.12.Final.3acd5c6, netty-transport-native-kqueue=netty-transport-native-kqueue-4.1.12.Final.3acd5c68ea, netty-transport-native-unix-common=netty-transport-native-unix-common-4.1.12.Final.3acd5c68ea, netty-transport-rxtx=netty-transport-rxtx-4.1.12.Final.3acd5c68ea, netty-transport-sctp=netty-transport-sctp-4.1.12.Final.3acd5c68ea, netty-transport-udt=netty-transport-udt-4.1.12.Final.3acd5c68ea] 2018-04-25 19:20:47,672 INFO [main] org.apache.cassandra.transport.Server.start(Server.java:156) Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)... 2018-04-25 19:20:47,696 INFO [main] org.apache.cassandra.service.CassandraDaemon.start(CassandraDaemon.java:537) Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it 2018-04-25 19:20:47,705 INFO [main] org.elassandra.discovery.CassandraDiscovery.doStart(CassandraDiscovery.java:152) localNode name=10.233.122.153 id=62891ec9-7de6-4d55-b2cb-7eae97ed2957 localAddress=/10.233.122.153 publish_host=10.233.122.153:9300 2018-04-25 19:20:47,886 INFO [main] org.elassandra.discovery.CassandraDiscovery.setSearchEnabled(CassandraDiscovery.java:723) searchEnabled set to [true] Elassandra started 2018-04-25 19:20:48,800 INFO [Service Thread] org.apache.cassandra.service.GCInspector.handleNotification(GCInspector.java:284) ParNew GC in 242ms. CMS Old Gen: 38169280 -> 124063144; Par Eden Space: 167772160 -> 0; 2018-04-25 19:20:48,857 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,871 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,883 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,923 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,935 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,953 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:48,960 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,002 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,018 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,024 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,069 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,081 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,097 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,106 WARN [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elasticsearch.cluster.routing.IndexRoutingTable$Builder.(IndexRoutingTable.java:380) Keyspace not available: java.lang.AssertionError: recovery source only available on unassigned or initializing shard but was STARTED 2018-04-25 19:20:49,107 INFO [elasticsearch[10.233.122.153][clusterService#updateTask][T#1]] org.elassandra.gateway.CassandraGatewayService$1.clusterStateProcessed(CassandraGatewayService.java:77) cassandra ring block released 2018-04-25 19:21:43,882 INFO [GossipTasks:1] org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:812) FatClient /10.233.122.152 has been silent for 30000ms, removing from gossip 2018-04-25 19:25:24,531 WARN [elasticsearch[10.233.122.153][http_server_worker][T#2]] org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:290) Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header. 2018-04-25 19:25:24,576 WARN [elasticsearch[10.233.122.153][http_server_worker][T#2]] org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishWithUnexpectedFailure(TransportReplicationAction.java:859) unexpected error during the primary phase for action [indices:data/write/bulk[s]], request [BulkShardRequest [[driverperf_v1][0]] containing [index {[driverperf_v1][driverperf_v1][1], source[ {"Pkey":"1","STOP_ID":"383733289","TRIP_ID":"106729811","EVENT_TYPE":"ARRIVAL","SCHED_EVENT_DATE":"2017-02-13 11:59:00Z","ACTUAL_EVENT_DATE":"2017-02-13 14:00:00","LATE_STATUS":"Late","ONTIME_STATUS":"Late","EVENT_REASON":"No Status","LEG_TYPE":"Pickup","ROUTE_NAME":"Demo-Gen","DRIVER":"Wattana Arsanok","BUSINESS_UNIT":"271 CEVA Ground Thailand","SVC_PROVIDER":"CGTO","STOP_NBR":"1","SCHED_VS_ACTUAL_MINS":121,"TRIP_STATUS":"Complete","SERVICE_TIME_MINS":2,"LOCATION_TYPE":"Warehouse Facility","FINAL_DLVY_STATUS":"Completed","STOP_STATE":"BKK","STOP_COUNTRY":"THAILAND","GEO":"0,0"} ]}]] org.elasticsearch.index.IndexNotFoundException: no such index at org.elasticsearch.cluster.routing.RoutingTable.shardRoutingTable(RoutingTable.java:141) at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.primary(TransportReplicationAction.java:738) at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:667) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:147) at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:92) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:363) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:478) at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:171) at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:85) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:69) at org.elasticsearch.action.bulk.TransportSingleItemBulkWriteAction.doExecute(TransportSingleItemBulkWriteAction.java:44) at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:420) at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:432) at org.elasticsearch.rest.action.document.RestIndexAction.lambda$prepareRequest$1(RestIndexAction.java:90) at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:262) at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:200) at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:506) at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:72) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:63) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at java.lang.Thread.run(Thread.java:748)

vroyer commented 6 years ago

Sorry, but please post your cluster state : curl -XGET http://$NODE:9200/_cluster/state/?pretty=true

Moreover, it seems that you have some infrastructure glitch (a freezed node or network issue) according to the message: org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:812) FatClient /10.233.122.152 has been silent for 30000ms, removing from gossip

cevalogistics commented 6 years ago

Vincent,

So what is the typical recovery for this? Rebuild the node? In any event here is the status.

MacBook-Pro-2:~ liottar$ curl http://inception.cevalogistics.com:9200/_cluster/state?pretty=true { "cluster_name" : "Cassandra", "version" : 210, "state_uuid" : "icI4SxpQThCJjp3sioIPpA", "master_node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "blocks" : { }, "nodes" : { "c4366c6f-7cdc-4dc2-8a02-75511c393d30" : { "name" : "10.233.97.184", "status" : "ALIVE", "ephemeral_id" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "transport_address" : "10.233.97.184:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } }, "d76a5242-4d12-41e8-a526-7724b57dbc31" : { "name" : "10.233.114.136", "status" : "ALIVE", "ephemeral_id" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "transport_address" : "10.233.114.136:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } }, "9796b736-e0d2-4a11-88cd-62bb9bc50ebe" : { "name" : "10.233.121.107", "status" : "ALIVE", "ephemeral_id" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "transport_address" : "10.233.121.107:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } }, "e271fb65-2836-4421-8292-77fcc3890012" : { "name" : "10.233.90.50", "status" : "ALIVE", "ephemeral_id" : "e271fb65-2836-4421-8292-77fcc3890012", "transport_address" : "10.233.90.50:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } }, "62891ec9-7de6-4d55-b2cb-7eae97ed2957" : { "name" : "10.233.122.153", "status" : "DEAD", "ephemeral_id" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "transport_address" : "10.233.122.153:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } }, "ab90840a-5ed7-45d7-b816-6545f5103ac5" : { "name" : "10.233.117.250", "status" : "ALIVE", "ephemeral_id" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "transport_address" : "10.233.117.250:9300", "attributes" : { "rack" : "Rack1", "dc" : "DC1" } } }, "metadata" : { "version" : 71, "cluster_uuid" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "templates" : { }, "indices" : { "wmsskuet" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1522877640989", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "QHR0roXvQ66LICG3aZltZA", "version" : { "created" : "5050099" }, "provided_name" : "wmsskuet" } }, "mappings" : { "wmsskuet" : { "properties" : { "FromLocId" : { "type" : "keyword" }, "SiteId" : { "type" : "keyword" }, "ElapsedMin" : { "type" : "double" }, "Consignment" : { "type" : "keyword" }, "ElapsedTime" : { "type" : "long" }, "PKey" : { "type" : "keyword" }, "CustomerId" : { "type" : "keyword" }, "Code" : { "type" : "keyword" }, "StationId" : { "type" : "keyword" }, "CompleteDateTime" : { "format" : "yyyy-MM-dd HH:mm:ssZ", "type" : "date" }, "GroupId" : { "type" : "keyword" }, "SupplierId" : { "type" : "keyword" }, "Weekday" : { "type" : "keyword" }, "ContainerId" : { "type" : "keyword" }, "IsSummary" : { "type" : "keyword" }, "UserId" : { "type" : "keyword" }, "FinalLocId" : { "type" : "keyword" }, "ClientId" : { "type" : "keyword" }, "SkuId" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "gtnx_shipment_v1" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1523990432282", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "ntyLLHQwQmqAy0Kxd3dZfg", "version" : { "created" : "5050099" }, "provided_name" : "gtnx_shipment_v1" } }, "mappings" : { "gtnx_shipment_v1" : { "properties" : { "DEST_Days" : { "type" : "long" }, "Delivered at DEST" : { "type" : "keyword" }, "Cargo Ready Date" : { "type" : "keyword" }, "PTP_Days" : { "type" : "long" }, "Port of Loading" : { "type" : "keyword" }, "Product Code" : { "type" : "keyword" }, "Purchase Order" : { "type" : "keyword" }, "Arrived at POD" : { "type" : "keyword" }, "Load Type" : { "type" : "keyword" }, "Arrived at POL" : { "type" : "keyword" }, "Shipping Order Num" : { "type" : "keyword" }, "ShipmentStatus" : { "type" : "keyword" }, "Supplier" : { "type" : "keyword" }, "Carrier" : { "type" : "keyword" }, "id" : { "type" : "keyword" }, "Create Date" : { "format" : "yyyy-MM-dd", "type" : "date" }, "Line Item Id" : { "type" : "keyword" }, "ETD Port Of Load Date" : { "type" : "keyword" }, "Final Destination" : { "type" : "keyword" }, "Mode" : { "type" : "keyword" }, "Cargo_Days" : { "type" : "long" }, "ShipmentRow" : { "type" : "long" }, "Shipping Order Status" : { "type" : "keyword" }, "Last Updated" : { "format" : "yyyy-MM-dd", "type" : "date" }, "POL_Days" : { "type" : "long" }, "SHIPMENT ID" : { "type" : "keyword" }, "Departed from POL" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "gtnx_purchase_order" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1523027214574", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "0A5Tb-GXQrSpmN1oeBDhQQ", "version" : { "created" : "5050099" }, "provided_name" : "gtnx_purchase_order" } }, "mappings" : { "gtnx_purchase_order" : { "properties" : { "DEST_Days" : { "type" : "long" }, "Delivered at DEST" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Cargo Ready Date" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "PTP_Days" : { "type" : "long" }, "Business Segment" : { "type" : "keyword" }, "Ship Windows Duration" : { "type" : "long" }, "Accept_Yr" : { "type" : "keyword" }, "Port of Loading" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Product Code" : { "type" : "keyword" }, "Purchase Order" : { "type" : "keyword" }, "Arrived at POD" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Load Type" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Yr_Wkno" : { "type" : "keyword" }, "Wk_no" : { "type" : "keyword" }, "Arrived at POL" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Container" : { "type" : "keyword" }, "Shipping Order Num" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "ShipmentStatus" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Acceptance Status Date" : { "format" : "yyyy-MM-dd", "type" : "date" }, "Supplier" : { "type" : "keyword" }, "Carrier" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "id" : { "type" : "keyword" }, "Last Modified Date Order" : { "format" : "yyyy-MM-dd", "type" : "date" }, "Create Date" : { "type" : "date" }, "Line Item Id" : { "type" : "keyword" }, "ETD Port Of Load Date" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Final Destination" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Mode" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "BL AWB" : { "type" : "keyword" }, "Cargo_Days" : { "type" : "long" }, "Acceptance Status" : { "type" : "keyword" }, "ShipmentRow" : { "type" : "long" }, "Shipping Order Status" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Last Updated" : { "type" : "date" }, "POL_Days" : { "type" : "long" }, "SHIPMENT ID" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "Departed from POL" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "bob_v1" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1524692982712", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "nUIEpqJHR8eqweFbrp8hhA", "version" : { "created" : "5050099" }, "provided_name" : "bob_v1" } }, "mappings" : { "bob_v1" : { "properties" : { "SCHED_VS_ACTUAL_MINS" : { "type" : "long" }, "STOP_ID" : { "type" : "keyword" }, "SVC_PROVIDER" : { "type" : "keyword" }, "ACTUAL_EVENT_DATE" : { "type" : "keyword" }, "EVENT_TYPE" : { "type" : "keyword" }, "DRIVER" : { "type" : "keyword" }, "ONTIME_STATUS" : { "type" : "keyword" }, "LATE_STATUS" : { "type" : "keyword" }, "Pkey" : { "type" : "keyword" }, "BUSINESS_UNIT" : { "type" : "keyword" }, "STOP_NBR" : { "type" : "keyword" }, "EVENT_REASON" : { "type" : "keyword" }, "LOCATION_TYPE" : { "type" : "keyword" }, "TRIP_STATUS" : { "type" : "keyword" }, "GEO" : { "type" : "geo_point" }, "ROUTE_NAME" : { "type" : "keyword" }, "SCHED_EVENT_DATE" : { "format" : "yyyy-MM-dd HH:mm:ssZ", "type" : "date" }, "FINAL_DLVY_STATUS" : { "type" : "keyword" }, "TRIP_ID" : { "type" : "keyword" }, "LEG_TYPE" : { "type" : "keyword" }, "STOP_COUNTRY" : { "type" : "keyword" }, "SERVICE_TIME_MINS" : { "type" : "long" }, "STOP_STATE" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "gtnx_shipment_v2" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1523986277307", "number_of_shards" : "6", "number_of_replicas" : "0", "uuid" : "zUNifXgSRKyiSv_49_FWJg", "version" : { "created" : "5050099" }, "provided_name" : "gtnx_shipment_v2" } }, "mappings" : { "gtnx_shipment_v2" : { "properties" : { "DEST_Days" : { "type" : "long" }, "Delivered at DEST" : { "type" : "keyword" }, "Cargo Ready Date" : { "type" : "keyword" }, "PTP_Days" : { "type" : "long" }, "Port of Loading" : { "type" : "keyword" }, "Product Code" : { "type" : "keyword" }, "Purchase Order" : { "type" : "keyword" }, "Arrived at POD" : { "type" : "keyword" }, "Load Type" : { "type" : "keyword" }, "Arrived at POL" : { "type" : "keyword" }, "Shipping Order Num" : { "type" : "keyword" }, "ShipmentStatus" : { "type" : "keyword" }, "Supplier" : { "type" : "keyword" }, "Carrier" : { "type" : "keyword" }, "id" : { "type" : "keyword" }, "Create Date" : { "format" : "yyyy-MM-dd", "type" : "date" }, "Line Item Id" : { "type" : "keyword" }, "ETD Port Of Load Date" : { "type" : "keyword" }, "Final Destination" : { "type" : "keyword" }, "Mode" : { "type" : "keyword" }, "Cargo_Days" : { "type" : "long" }, "ShipmentRow" : { "type" : "long" }, "Shipping Order Status" : { "type" : "keyword" }, "Last Updated" : { "format" : "yyyy-MM-dd", "type" : "date" }, "POL_Days" : { "type" : "long" }, "SHIPMENT ID" : { "type" : "keyword" }, "Departed from POL" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, ".kibana" : { "state" : "open", "settings" : { "index" : { "keyspace" : "_kibana", "number_of_shards" : "6", "provided_name" : ".kibana", "mapper" : { "dynamic" : "false" }, "creation_date" : "1522694684445", "number_of_replicas" : "0", "uuid" : "60q4nNzrTqSqTd5ztckASw", "version" : { "created" : "5050099" } } }, "mappings" : { "server" : { "dynamic" : "strict", "properties" : { "uuid" : { "type" : "keyword" } } }, "visualization" : { "dynamic" : "strict", "properties" : { "savedSearchId" : { "type" : "keyword" }, "description" : { "type" : "text" }, "uiStateJSON" : { "type" : "text" }, "title" : { "type" : "text" }, "version" : { "type" : "integer" }, "kibanaSavedObjectMeta" : { "properties" : { "searchSourceJSON" : { "type" : "text" } } }, "visState" : { "type" : "text" } } }, "dashboard" : { "dynamic" : "strict", "properties" : { "hits" : { "type" : "integer" }, "timeFrom" : { "type" : "keyword" }, "timeTo" : { "type" : "keyword" }, "refreshInterval" : { "properties" : { "display" : { "type" : "keyword" }, "section" : { "type" : "integer" }, "value" : { "type" : "integer" }, "pause" : { "type" : "boolean" } } }, "description" : { "type" : "text" }, "uiStateJSON" : { "type" : "text" }, "timeRestore" : { "type" : "boolean" }, "title" : { "type" : "text" }, "version" : { "type" : "integer" }, "kibanaSavedObjectMeta" : { "properties" : { "searchSourceJSON" : { "type" : "text" } } }, "optionsJSON" : { "type" : "text" }, "panelsJSON" : { "type" : "text" } } }, "index-pattern" : { "dynamic" : "strict", "properties" : { "notExpandable" : { "type" : "boolean" }, "fieldFormatMap" : { "type" : "text" }, "sourceFilters" : { "type" : "text" }, "timeFieldName" : { "type" : "keyword" }, "intervalName" : { "type" : "keyword" }, "fields" : { "type" : "text" }, "title" : { "type" : "text" } } }, "config" : { "dynamic" : "true", "properties" : { "buildNum" : { "type" : "keyword" }, "defaultIndex" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } } } }, "default" : { "dynamic" : "strict" }, "url" : { "dynamic" : "strict", "properties" : { "accessCount" : { "type" : "long" }, "accessDate" : { "type" : "date" }, "url" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 2048, "type" : "keyword" } } }, "createDate" : { "type" : "date" } } }, "timelion-sheet" : { "dynamic" : "strict", "properties" : { "hits" : { "type" : "integer" }, "timelion_sheet" : { "type" : "text" }, "timelion_interval" : { "type" : "keyword" }, "timelion_columns" : { "type" : "integer" }, "timelion_other_interval" : { "type" : "keyword" }, "timelion_rows" : { "type" : "integer" }, "description" : { "type" : "text" }, "title" : { "type" : "text" }, "version" : { "type" : "integer" }, "kibanaSavedObjectMeta" : { "properties" : { "searchSourceJSON" : { "type" : "text" } } }, "timelion_chart_height" : { "type" : "integer" } } }, "search" : { "dynamic" : "strict", "properties" : { "hits" : { "type" : "integer" }, "columns" : { "type" : "keyword" }, "description" : { "type" : "text" }, "sort" : { "type" : "keyword" }, "title" : { "type" : "text" }, "version" : { "type" : "integer" }, "kibanaSavedObjectMeta" : { "properties" : { "searchSourceJSON" : { "type" : "text" } } } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "driverperf" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1524663125928", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "ZoF29x-5RJ-v4WGyudFa1g", "version" : { "created" : "5050099" }, "provided_name" : "driverperf" } }, "mappings" : { "driverperf" : { "properties" : { "SCHED_VS_ACTUAL_MINS" : { "type" : "long" }, "STOP_ID" : { "type" : "keyword" }, "SVC_PROVIDER" : { "type" : "keyword" }, "ACTUAL_EVENT_DATE" : { "type" : "keyword" }, "EVENT_TYPE" : { "type" : "keyword" }, "DRIVER" : { "type" : "keyword" }, "ONTIME_STATUS" : { "type" : "keyword" }, "LATE_STATUS" : { "type" : "keyword" }, "Pkey" : { "type" : "keyword" }, "BUSINESS_UNIT" : { "type" : "keyword" }, "STOP_NBR" : { "type" : "keyword" }, "EVENT_REASON" : { "type" : "keyword" }, "LOCATION_TYPE" : { "type" : "keyword" }, "TRIP_STATUS" : { "type" : "keyword" }, "GEO" : { "type" : "geo_point" }, "ROUTE_NAME" : { "type" : "keyword" }, "SCHED_EVENT_DATE" : { "format" : "yyyy-MM-dd HH:mm:ssZ", "type" : "date" }, "FINAL_DLVY_STATUS" : { "type" : "keyword" }, "TRIP_ID" : { "type" : "keyword" }, "LEG_TYPE" : { "type" : "keyword" }, "STOP_COUNTRY" : { "type" : "keyword" }, "SERVICE_TIME_MINS" : { "type" : "long" }, "STOP_STATE" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "twitter" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1524691938307", "number_of_shards" : "6", "number_of_replicas" : "0", "uuid" : "cRLgf28HRH-ygrOUywGCtQ", "version" : { "created" : "5050099" }, "provided_name" : "twitter" } }, "mappings" : { "user" : { "properties" : { "name" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } } } }, "tweet" : { "properties" : { "postDate" : { "type" : "date" }, "message" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "user" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "driverperf_v1" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1524663591070", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "IhC82nztR76akRo-uV0PRA", "version" : { "created" : "5050099" }, "provided_name" : "driverperf_v1" } }, "mappings" : { "driverperf_v1" : { "properties" : { "SCHED_VS_ACTUAL_MINS" : { "type" : "long" }, "STOP_ID" : { "type" : "keyword" }, "SVC_PROVIDER" : { "type" : "keyword" }, "ACTUAL_EVENT_DATE" : { "type" : "keyword" }, "EVENT_TYPE" : { "type" : "keyword" }, "DRIVER" : { "type" : "keyword" }, "ONTIME_STATUS" : { "type" : "keyword" }, "LATE_STATUS" : { "type" : "keyword" }, "Pkey" : { "type" : "keyword" }, "BUSINESS_UNIT" : { "type" : "keyword" }, "STOP_NBR" : { "type" : "keyword" }, "EVENT_REASON" : { "type" : "keyword" }, "LOCATION_TYPE" : { "type" : "keyword" }, "TRIP_STATUS" : { "type" : "keyword" }, "GEO" : { "type" : "geo_point" }, "ROUTE_NAME" : { "type" : "keyword" }, "SCHED_EVENT_DATE" : { "format" : "yyyy-MM-dd HH:mm:ssZ", "type" : "date" }, "FINAL_DLVY_STATUS" : { "type" : "keyword" }, "TRIP_ID" : { "type" : "keyword" }, "LEG_TYPE" : { "type" : "keyword" }, "STOP_COUNTRY" : { "type" : "keyword" }, "SERVICE_TIME_MINS" : { "type" : "long" }, "STOP_STATE" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "poc_training" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1523966913762", "number_of_shards" : "6", "number_of_replicas" : "0", "uuid" : "PvumvsazTgyJnPisTHuRFA", "version" : { "created" : "5050099" }, "provided_name" : "poc_training" } }, "mappings" : { "poc_training" : { "properties" : { "max_possible_score" : { "type" : "double" }, "shipper" : { "type" : "text" }, "dangerous_goods" : { "type" : "text" }, "risk_assessment_id" : { "type" : "double" }, "current_mitigated_score" : { "type" : "double" }, "origin_airport" : { "type" : "text" }, "customer" : { "type" : "text" } } }, "data" : { "properties" : { "max_possible_score" : { "type" : "double" }, "shipper" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "consignee" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "dangerous_goods" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "risk_assessment_id" : { "type" : "double" }, "current_mitigated_score" : { "type" : "double" }, "origin_airport" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } }, "customer" : { "type" : "text", "fields" : { "keyword" : { "ignore_above" : 256, "type" : "keyword" } } } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } }, "prdedi" : { "state" : "open", "settings" : { "index" : { "creation_date" : "1522701473519", "number_of_shards" : "6", "number_of_replicas" : "2", "uuid" : "Mw5gBIs8Q-2ZRp1REikJCA", "version" : { "created" : "5050099" }, "provided_name" : "prdedi" } }, "mappings" : { "prdedi" : { "properties" : { "transaction_status" : { "type" : "keyword" }, "date_updated" : { "format" : "yyyy-MM-dd HH:mm:ssZ", "type" : "date" }, "to_app" : { "type" : "keyword" }, "interface_name" : { "type" : "keyword" }, "weekday" : { "type" : "keyword" }, "schema_name" : { "type" : "keyword" }, "file_size" : { "type" : "double" }, "business_unit_id" : { "type" : "keyword" }, "hour_updated" : { "type" : "long" }, "from_app" : { "type" : "keyword" }, "correlation_id" : { "type" : "keyword" }, "customer_name" : { "type" : "keyword" }, "id" : { "type" : "keyword" } } } }, "aliases" : [ ], "primary_terms" : { "0" : 0, "1" : 0, "2" : 0, "3" : 0, "4" : 0, "5" : 0 }, "in_sync_allocations" : { "1" : [ ], "2" : [ ], "5" : [ ], "3" : [ ], "4" : [ ], "0" : [ ] } } }, "index-graveyard" : { "tombstones" : [ { "index" : { "index_name" : "gtnx_purchase_order", "index_uuid" : "J8SH5YpAQqW8mCNaSOseLw" }, "delete_date_in_millis" : 1523027203794 }, { "index" : { "index_name" : "gtnx_shipment_v1", "index_uuid" : "cFhACbNGQx-_UY3C2ZUacQ" }, "delete_date_in_millis" : 1523030048063 }, { "index" : { "index_name" : "poc_training", "index_uuid" : "2e7OqG_JSoycDtZRiw5fnQ" }, "delete_date_in_millis" : 1523965085592 }, { "index" : { "index_name" : "poc_training", "index_uuid" : "mLb7jZOGQfiIyOIEnu70VA" }, "delete_date_in_millis" : 1523965151787 }, { "index" : { "index_name" : "poc_training", "index_uuid" : "W0LIUcWLTR2vCHLZp0o7-A" }, "delete_date_in_millis" : 1523966805006 }, { "index" : { "index_name" : "poc_training", "index_uuid" : "3apP2arFT5aR_Zt2c6DQHg" }, "delete_date_in_millis" : 1523966909854 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "881t-hihT7SBtj2ujC2VEg" }, "delete_date_in_millis" : 1523973129762 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "u54YyFjATr26T2-jLa29eA" }, "delete_date_in_millis" : 1523974496216 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "ZwawlWzfR0imLxj8MvN3xw" }, "delete_date_in_millis" : 1523974794313 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "WzDGFbivS1utVLLFruzEaA" }, "delete_date_in_millis" : 1523986035784 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "XkP9tcU-R7Oi2K-6gg45Ew" }, "delete_date_in_millis" : 1523986131448 }, { "index" : { "index_name" : "gtnx_shipment_v2", "index_uuid" : "QzSbJtKPQ9G9Pxy7MuzPnQ" }, "delete_date_in_millis" : 1523986248741 }, { "index" : { "index_name" : "gtnx_shipment_v1", "index_uuid" : "_a1JvbgAR3-NPZdRUzywAg" }, "delete_date_in_millis" : 1523990379789 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "lQlRYZ0AQLyy6AxMbqnGCg" }, "delete_date_in_millis" : 1524517034015 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "A8av0rBDTzCOmIVym1wWaQ" }, "delete_date_in_millis" : 1524597283890 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "tlvYAcl9Rti2dAUq60bLEw" }, "delete_date_in_millis" : 1524598885805 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "2sclRWBlSv2ju3f7XRfAxw" }, "delete_date_in_millis" : 1524601944967 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "OkJHpHH5QVef9kzMhX4bCw" }, "delete_date_in_millis" : 1524609938301 }, { "index" : { "index_name" : "driverperf", "index_uuid" : "I2GuZcJSS9m1JFqGWfbQCg" }, "delete_date_in_millis" : 1524663060798 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "8qbvVZlGR_6fo7pmJqtwaA" }, "delete_date_in_millis" : 1524690487751 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "VaQe-vdKR0mLzauATdR4FA" }, "delete_date_in_millis" : 1524691177564 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "Us8FFBAFQhqTT91wsWrSPQ" }, "delete_date_in_millis" : 1524691323288 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "zZZE_lg7QhSrXR9t3nc9zQ" }, "delete_date_in_millis" : 1524692669848 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "nAlEPtmORROSlJyLlMpUzQ" }, "delete_date_in_millis" : 1524692779043 }, { "index" : { "index_name" : "bob_v1", "index_uuid" : "E4aVJ-6sTmOAh1lfSmxt-g" }, "delete_date_in_millis" : 1524692931319 } ] } }, "routing_table" : { "indices" : { ".kibana" : { "shards" : { "1" : [ { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : ".kibana", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "2" : [ { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : ".kibana", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "3" : [ { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : ".kibana", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "4" : [ { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : ".kibana", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "5" : [ { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : ".kibana", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "0" : [ { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : ".kibana", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ] } }, "twitter" : { "shards" : { "4" : [ { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "twitter", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "2" : [ { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "twitter", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "1" : [ { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "twitter", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "3" : [ { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "twitter", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "5" : [ { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "twitter", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "0" : [ { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "twitter", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ] } }, "poc_training" : { "shards" : { "2" : [ { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "poc_training", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "1" : [ { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "poc_training", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "5" : [ { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "poc_training", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "4" : [ { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "poc_training", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "3" : [ { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "poc_training", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "0" : [ { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "poc_training", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ] } }, "gtnx_shipment_v2" : { "shards" : { "1" : [ { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "2" : [ { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "5" : [ { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "4" : [ { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "3" : [ { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "0" : [ { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ] } } } }, "routing_nodes" : { "unassigned" : [ ], "nodes" : { "e271fb65-2836-4421-8292-77fcc3890012" : [ { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : ".kibana", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "twitter", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "poc_training", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "e271fb65-2836-4421-8292-77fcc3890012", "relocating_node" : null, "shard" : 1, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-5086691515524827064,-4455400665633705009]", "(-4164481151685738517,-3307979720416438909]", "(-2206250258799490034,-2177627868341832852]", "(-1839508776458638598,-1794129631622032748]", "(-118864622025183534,624939019991921312]", "(5141005615383236012,5915936481258352663]", "(6132333345085630748,6228162807192115475]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "62891ec9-7de6-4d55-b2cb-7eae97ed2957" : [ { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : ".kibana", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "twitter", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "poc_training", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "62891ec9-7de6-4d55-b2cb-7eae97ed2957", "relocating_node" : null, "shard" : 0, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-9048183706652428987,-8592977458636693389]", "(-7227302454714817399,-6090579284089889517]", "(-861916047897630515,-362226123124935217]", "(4437679225950754397,4631534558835767927]", "(6228162807192115475,6275689832852561075]", "(8234545346034199838,8401438299379969757]", "(8907766337491345741,8914019774812791023]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "d76a5242-4d12-41e8-a526-7724b57dbc31" : [ { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : ".kibana", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "twitter", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "poc_training", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "d76a5242-4d12-41e8-a526-7724b57dbc31", "relocating_node" : null, "shard" : 2, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-9223372036854775808,-9048183706652428987]", "(-5939375512383729793,-5086691515524827064]", "(-4455400665633705009,-4164481151685738517]", "(-1794129631622032748,-1054867843544454931]", "(3681862537357852614,4437679225950754397]", "(4748189529193459068,5141005615383236012]", "(6275689832852561075,6372063045572782995]", "(9083020370715658495,9223372036854775807]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "ab90840a-5ed7-45d7-b816-6545f5103ac5" : [ { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : ".kibana", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "twitter", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "poc_training", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "ab90840a-5ed7-45d7-b816-6545f5103ac5", "relocating_node" : null, "shard" : 3, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-7318731352128861107,-7227302454714817399]", "(-6090579284089889517,-5939375512383729793]", "(-2539197605680580109,-2206250258799490034]", "(1111019062840381676,2600887383460939089]", "(4631534558835767927,4748189529193459068]", "(8401438299379969757,8796692000955143727]", "(9073971229396420877,9083020370715658495]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "9796b736-e0d2-4a11-88cd-62bb9bc50ebe" : [ { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : ".kibana", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "twitter", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "poc_training", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "9796b736-e0d2-4a11-88cd-62bb9bc50ebe", "relocating_node" : null, "shard" : 4, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-3307979720416438909,-2539197605680580109]", "(-2177627868341832852,-1839508776458638598]", "(-362226123124935217,-118864622025183534]", "(624939019991921312,1111019062840381676]", "(5915936481258352663,6132333345085630748]", "(6372063045572782995,6774441214942613853]", "(8796692000955143727,8907766337491345741]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ], "c4366c6f-7cdc-4dc2-8a02-75511c393d30" : [ { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : ".kibana", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "twitter", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "poc_training", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } }, { "state" : "STARTED", "primary" : true, "node" : "c4366c6f-7cdc-4dc2-8a02-75511c393d30", "relocating_node" : null, "shard" : 5, "index" : "gtnx_shipment_v2", "token_ranges" : [ "(-8592977458636693389,-7318731352128861107]", "(-1054867843544454931,-861916047897630515]", "(2600887383460939089,3681862537357852614]", "(6774441214942613853,8234545346034199838]", "(8914019774812791023,9073971229396420877]" ], "allocation_id" : { "id" : "dummy_alloc_id" } } ] } } } MacBook-Pro-2:~ liottar$

cevalogistics commented 6 years ago

Vincent. I found the root cause. It looks like port 9300 was stuck on the Kubernetes host. I restarted the node and it looks good now.