search

cortexproject / cortex

A horizontally scalable, highly available, multi-tenant, long term Prometheus.
https://cortexmetrics.io/
Apache License 2.0
5.48k stars 801 forks source link

Flaky TestDeleteSeriesAllIndexBackends #3420

Closed pracucci closed 3 years ago

pracucci commented 4 years ago

Describe the bug The integration test TestDeleteSeriesAllIndexBackends looks flaky:

=== RUN   TestDeleteSeriesAllIndexBackends
17:26:51 Starting cassandra
17:26:52 Ports for container: e2e-cortex-test-cassandra Mapping: map[9042:32936]
17:26:52 Starting dynamodb
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
17:26:53 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
17:26:53 cassandra: CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
17:26:54 Ports for container: e2e-cortex-test-dynamodb Mapping: map[8000:32937]
17:26:54 Starting bigtable
17:26:54 dynamodb: Initializing DynamoDB Local with the following configuration:
17:26:54 dynamodb: Port:    8000
17:26:54 dynamodb: InMemory:    true
17:26:54 dynamodb: DbPath:  null
17:26:54 dynamodb: SharedDb:    true
17:26:54 dynamodb: shouldDelayTransientStatuses:    false
17:26:54 dynamodb: CorsParams:  *
17:26:55 cassandra: INFO  [main] 2020-10-28 17:26:55,542 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
17:26:55 bigtable: Bigtable emulator running on [::]:9035
17:26:56 Ports for container: e2e-cortex-test-bigtable Mapping: map[9035:32938]
17:26:56 Starting consul
17:26:56 cassandra: INFO  [main] 2020-10-28 17:26:56,870 Config.java:481 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=192.168.48.2; broadcast_rpc_address=192.168.48.2; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@37e547da; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=0; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=192.168.48.2; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=1; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=192.168.48.2}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@2b6856dd; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
17:26:56 cassandra: INFO  [main] 2020-10-28 17:26:56,871 DatabaseDescriptor.java:366 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
17:26:56 cassandra: INFO  [main] 2020-10-28 17:26:56,872 DatabaseDescriptor.java:420 - Global memtable on-heap threshold is enabled at 462MB
17:26:56 cassandra: INFO  [main] 2020-10-28 17:26:56,872 DatabaseDescriptor.java:424 - Global memtable off-heap threshold is enabled at 462MB
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,151 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,152 DatabaseDescriptor.java:710 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,373 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,382 CassandraDaemon.java:471 - Hostname: cassandra
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,382 CassandraDaemon.java:478 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_131
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,384 CassandraDaemon.java:479 - Heap size: 1.805GiB/1.805GiB
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,385 CassandraDaemon.java:484 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4501952(4396K) committed = 4521984(4416K) max = 251658240(245760K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,386 CassandraDaemon.java:484 - Metaspace Non-heap memory: init = 0(0K) used = 17549904(17138K) committed = 18219008(17792K) max = -1(-1K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,387 CassandraDaemon.java:484 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2090424(2041K) committed = 2228224(2176K) max = 1073741824(1048576K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,388 CassandraDaemon.java:484 - Par Eden Space Heap memory: init = 167772160(163840K) used = 94023544(91819K) committed = 167772160(163840K) max = 167772160(163840K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,389 CassandraDaemon.java:484 - Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,390 CassandraDaemon.java:484 - CMS Old Gen Heap memory: init = 1749024768(1708032K) used = 0(0K) committed = 1749024768(1708032K) max = 1749024768(1708032K)
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,391 CassandraDaemon.java:486 - Classpath: /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.4.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.11.0.jar:/usr/share/cassandra/apache-cassandra-thrift-3.11.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,392 CassandraDaemon.java:488 - JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms1868M, -Xmx1868M, -Xmn200M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.initial_token=0, -Dcassandra.skip_wait_for_gossip_to_settle=0, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,578 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,579 StartupChecks.java:131 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,580 StartupChecks.java:160 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,580 StartupChecks.java:197 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,582 SigarLibrary.java:44 - Initializing SIGAR library
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,620 SigarLibrary.java:180 - Checked OS settings and found them configured for optimal performance.
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,632 StartupChecks.java:265 - Maximum number of memory map areas per process (vm.max_map_count) 65530 is too low, recommended value: 1048575, you can change it with sysctl.
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,687 StartupChecks.java:286 - Directory /var/lib/cassandra/data doesn't exist
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,701 StartupChecks.java:286 - Directory /var/lib/cassandra/commitlog doesn't exist
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,702 StartupChecks.java:286 - Directory /var/lib/cassandra/saved_caches doesn't exist
17:26:57 cassandra: WARN  [main] 2020-10-28 17:26:57,709 StartupChecks.java:286 - Directory /var/lib/cassandra/hints doesn't exist
17:26:57 Ports for container: e2e-cortex-test-consul Mapping: map[8500:32939]
17:26:57 consul: ==> Starting Consul agent...
17:26:57 consul: Version: '1.8.4'
17:26:57 consul: Node ID: 'd7e32fc8-ba7c-3343-1de9-6a94a903d1fd'
17:26:57 consul: Node name: 'consul'
17:26:57 consul: Datacenter: 'dc1' (Segment: '<all>')
17:26:57 consul: Server: true (Bootstrap: false)
17:26:57 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
17:26:57 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
17:26:57 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
17:26:57 consul: ==> Log data will now stream in as it occurs:
17:26:57 consul: ==> Consul agent running!
17:26:57 cassandra: INFO  [main] 2020-10-28 17:26:57,880 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
17:26:58 cassandra: INFO  [main] 2020-10-28 17:26:58,875 ColumnFamilyStore.java:406 - Initializing system.IndexInfo
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,085 ColumnFamilyStore.java:406 - Initializing system.batches
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,092 ColumnFamilyStore.java:406 - Initializing system.paxos
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,105 ColumnFamilyStore.java:406 - Initializing system.local
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,121 ColumnFamilyStore.java:406 - Initializing system.peers
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,138 ColumnFamilyStore.java:406 - Initializing system.peer_events
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,152 ColumnFamilyStore.java:406 - Initializing system.range_xfers
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,162 ColumnFamilyStore.java:406 - Initializing system.compaction_history
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,174 ColumnFamilyStore.java:406 - Initializing system.sstable_activity
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,187 ColumnFamilyStore.java:406 - Initializing system.size_estimates
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,197 ColumnFamilyStore.java:406 - Initializing system.available_ranges
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,211 ColumnFamilyStore.java:406 - Initializing system.transferred_ranges
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,223 ColumnFamilyStore.java:406 - Initializing system.views_builds_in_progress
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,233 ColumnFamilyStore.java:406 - Initializing system.built_views
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,243 ColumnFamilyStore.java:406 - Initializing system.hints
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,252 ColumnFamilyStore.java:406 - Initializing system.batchlog
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,259 ColumnFamilyStore.java:406 - Initializing system.prepared_statements
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,265 ColumnFamilyStore.java:406 - Initializing system.schema_keyspaces
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,273 ColumnFamilyStore.java:406 - Initializing system.schema_columnfamilies
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,281 ColumnFamilyStore.java:406 - Initializing system.schema_columns
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,290 ColumnFamilyStore.java:406 - Initializing system.schema_triggers
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,299 ColumnFamilyStore.java:406 - Initializing system.schema_usertypes
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,308 ColumnFamilyStore.java:406 - Initializing system.schema_functions
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,320 ColumnFamilyStore.java:406 - Initializing system.schema_aggregates
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,322 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
17:27:01 cassandra: INFO  [main] 2020-10-28 17:27:01,431 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
17:27:02 cassandra: INFO  [MemtableFlushWriter:1] 2020-10-28 17:27:02,344 CacheService.java:112 - Initializing key cache with capacity of 92 MBs.
17:27:02 cassandra: INFO  [MemtableFlushWriter:1] 2020-10-28 17:27:02,356 CacheService.java:134 - Initializing row cache with capacity of 0 MBs
17:27:02 cassandra: INFO  [MemtableFlushWriter:1] 2020-10-28 17:27:02,359 CacheService.java:163 - Initializing counter cache with capacity of 46 MBs
17:27:02 cassandra: INFO  [MemtableFlushWriter:1] 2020-10-28 17:27:02,361 CacheService.java:174 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
17:27:02 cassandra: INFO  [main] 2020-10-28 17:27:02,716 StorageService.java:599 - Populating token metadata from system tables
17:27:02 cassandra: INFO  [main] 2020-10-28 17:27:02,771 BufferPool.java:230 - Global buffer pool is enabled, when pool is exhausted (max is 462.000MiB) it will allocate on heap
17:27:02 cassandra: INFO  [main] 2020-10-28 17:27:02,952 StorageService.java:606 - Token metadata:
17:27:02 cassandra: INFO  [main] 2020-10-28 17:27:02,984 ColumnFamilyStore.java:406 - Initializing system_schema.keyspaces
17:27:02 cassandra: INFO  [main] 2020-10-28 17:27:02,994 ColumnFamilyStore.java:406 - Initializing system_schema.tables
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,010 ColumnFamilyStore.java:406 - Initializing system_schema.columns
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,033 ColumnFamilyStore.java:406 - Initializing system_schema.triggers
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,052 ColumnFamilyStore.java:406 - Initializing system_schema.dropped_columns
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,059 ColumnFamilyStore.java:406 - Initializing system_schema.views
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,067 ColumnFamilyStore.java:406 - Initializing system_schema.types
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,077 ColumnFamilyStore.java:406 - Initializing system_schema.functions
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,094 ColumnFamilyStore.java:406 - Initializing system_schema.aggregates
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,108 ColumnFamilyStore.java:406 - Initializing system_schema.indexes
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,115 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
17:27:03 cassandra: INFO  [pool-3-thread-1] 2020-10-28 17:27:03,180 AutoSavingCache.java:173 - Completed loading (13 ms; 4 keys) KeyCache cache
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,238 CommitLog.java:152 - No commitlog files found; skipping replay
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,238 StorageService.java:599 - Populating token metadata from system tables
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,292 StorageService.java:606 - Token metadata:
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,567 QueryProcessor.java:162 - Preloaded 0 prepared statements
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,569 StorageService.java:617 - Cassandra version: 3.11.0
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,569 StorageService.java:618 - Thrift API version: 20.1.0
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,570 StorageService.java:619 - CQL supported versions: 3.4.4 (default: 3.4.4)
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,570 StorageService.java:621 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,650 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 92 MB and a resize interval of 60 minutes
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,679 MessagingService.java:753 - Starting Messaging Service on /192.168.48.2:7000 (eth0)
17:27:03 cassandra: WARN  [main] 2020-10-28 17:27:03,699 SystemKeyspace.java:1083 - No host ID found, created 809ce138-4196-424c-bdf8-b6b79111fe01 (Note: This should happen exactly once per node).
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,792 StorageService.java:706 - Loading persisted ring state
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,816 StorageService.java:819 - Starting up server gossip
17:27:03 cassandra: INFO  [main] 2020-10-28 17:27:03,975 StorageService.java:857 - This node will not auto bootstrap because it is configured to be a seed node.
17:27:04 cassandra: INFO  [main] 2020-10-28 17:27:04,015 StorageService.java:984 - Saved tokens not found. Using configuration value: [0]
17:27:04 cassandra: INFO  [main] 2020-10-28 17:27:04,040 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@e3b3b75[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[session_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[client, command, session_id, coordinator, request, started_at, duration, parameters],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@54f732c4[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[session_id],clusteringColumns=[event_id],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[activity, event_id, session_id, source, thread, source_elapsed],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,597 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,658 ColumnFamilyStore.java:406 - Initializing system_traces.events
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,701 ColumnFamilyStore.java:406 - Initializing system_traces.sessions
17:27:04 cassandra: INFO  [main] 2020-10-28 17:27:04,790 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@62103d37[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[keyspace_name, columnfamily_name],clusteringColumns=[id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@53493c07[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[parent_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@550fa2cc[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[keyspace_name, view_name],clusteringColumns=[host_id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[view_name, status, keyspace_name, host_id],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,974 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,982 ColumnFamilyStore.java:406 - Initializing system_distributed.parent_repair_history
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,990 ColumnFamilyStore.java:406 - Initializing system_distributed.repair_history
17:27:04 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:04,998 ColumnFamilyStore.java:406 - Initializing system_distributed.view_build_status
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,051 StorageService.java:1439 - JOINING: Finish joining ring
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,191 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@50a18b03[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[role],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[salted_hash, member_of, role, can_login, is_superuser],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@4c3a5a64[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[role],clusteringColumns=[member],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, member],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@721639c8[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[role],clusteringColumns=[resource],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, resource, permissions],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@22f0a26c[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[resource],clusteringColumns=[role],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
17:27:05 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:05,320 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
17:27:05 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:05,325 ColumnFamilyStore.java:406 - Initializing system_auth.resource_role_permissons_index
17:27:05 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:05,330 ColumnFamilyStore.java:406 - Initializing system_auth.role_members
17:27:05 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:05,336 ColumnFamilyStore.java:406 - Initializing system_auth.role_permissions
17:27:05 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:05,341 ColumnFamilyStore.java:406 - Initializing system_auth.roles
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,481 NativeTransportService.java:70 - Netty using native Epoll event loop
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,549 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,550 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
17:27:05 cassandra: INFO  [main] 2020-10-28 17:27:05,607 CassandraDaemon.java:527 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
17:27:07 Starting table-manager
17:27:08 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:32941 9095:32940]
17:27:12 Stopping table-manager
17:27:13 Starting table-manager
17:27:14 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:32943 9095:32942]
17:27:14 Stopping table-manager
17:27:14 Starting table-manager
17:27:15 cassandra: INFO  [OptionalTasks:1] 2020-10-28 17:27:15,635 CassandraRoleManager.java:355 - Created default superuser role 'cassandra'
17:27:15 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:32945 9095:32944]
17:27:15 table-manager: level=error ts=2020-10-28T17:27:15.884067436Z caller=connectionpool.go:523 module=gocql client=table-manager msg="failed to connect" address=192.168.48.2:9042 error="Keyspace 'tests' does not exist"
17:27:16 table-manager: level=error ts=2020-10-28T17:27:16.011569008Z caller=connectionpool.go:523 module=gocql client=table-manager msg="failed to connect" address=192.168.48.2:9042 error="Keyspace 'tests' does not exist"
17:27:16 cassandra: INFO  [Native-Transport-Requests-3] 2020-10-28 17:27:16,118 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=tests, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
17:27:16 cassandra: INFO  [Native-Transport-Requests-2] 2020-10-28 17:27:16,361 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@24d9f512[cfId=d3024790-1942-11eb-a22c-7f6011a90b8b,ksName=tests,cfName=cortex_2650,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
17:27:16 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:16,512 ColumnFamilyStore.java:406 - Initializing tests.cortex_2650
17:27:16 cassandra: INFO  [Native-Transport-Requests-2] 2020-10-28 17:27:16,560 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@533764ba[cfId=d320a500-1942-11eb-a22c-7f6011a90b8b,ksName=tests,cfName=cortex_2651,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
17:27:16 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:16,698 ColumnFamilyStore.java:406 - Initializing tests.cortex_2651
17:27:16 cassandra: INFO  [Native-Transport-Requests-3] 2020-10-28 17:27:16,725 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@2884a214[cfId=d339ab40-1942-11eb-a22c-7f6011a90b8b,ksName=tests,cfName=cortex_chunks_2650,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
17:27:16 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:16,862 ColumnFamilyStore.java:406 - Initializing tests.cortex_chunks_2650
17:27:16 cassandra: INFO  [Native-Transport-Requests-2] 2020-10-28 17:27:16,907 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@71bd6bba[cfId=d35597b0-1942-11eb-a22c-7f6011a90b8b,ksName=tests,cfName=cortex_chunks_2651,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@9a6cfe3, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
17:27:17 cassandra: INFO  [MigrationStage:1] 2020-10-28 17:27:17,078 ColumnFamilyStore.java:406 - Initializing tests.cortex_chunks_2651
17:27:17 Stopping table-manager
17:27:17 Starting distributor
17:27:18 Ports for container: e2e-cortex-test-distributor Mapping: map[80:32947 9095:32946]
17:27:18 Starting ingester
17:27:19 Ports for container: e2e-cortex-test-ingester Mapping: map[80:32949 9095:32948]
17:27:19 Starting querier
17:27:21 Ports for container: e2e-cortex-test-querier Mapping: map[80:32951 9095:32950]
17:27:31 Starting purger
17:27:32 Ports for container: e2e-cortex-test-purger Mapping: map[80:32953 9095:32952]
17:27:32 purger: level=warn ts=2020-10-28T17:27:32.152361349Z caller=experimental.go:19 msg="experimental feature in use" feature="Delete series API"
17:27:32 Stopping purger
17:27:33 Starting purger
17:27:34 Ports for container: e2e-cortex-test-purger Mapping: map[80:32955 9095:32954]
17:27:34 purger: level=warn ts=2020-10-28T17:27:34.232641854Z caller=experimental.go:19 msg="experimental feature in use" feature="Delete series API"
17:27:34 purger: level=error ts=2020-10-28T17:27:34.82559081Z caller=purger.go:240 user_id=user-1 request_id=70043089 msg="error removing delete plan" plan_no=1 err="open /shared/user-1:70043089: no such file or directory"
    chunks_delete_series_test.go:182: 
            Error Trace:    chunks_delete_series_test.go:182
            Error:          Received unexpected error:
                            unable to find metrics [cortex_purger_delete_requests_processed_total] with expected values. Last error: <nil>. Last values: [2]
            Test:           TestDeleteSeriesAllIndexBackends
17:27:57 Killing purger
17:27:57 Killing querier
17:27:57 querier: level=error ts=2020-10-28T17:27:57.74970605Z caller=client.go:229 msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=23&stale=&wait=10000ms\": context canceled"
17:27:58 Killing ingester
17:27:58 ingester: level=warn ts=2020-10-28T17:27:58.162027377Z caller=transfer.go:294 msg="transfer attempt failed" err="cannot find ingester to transfer chunks to: no pending ingesters" attempt=1 max_retries=10
17:27:58 Killing distributor
17:27:58 distributor: level=error ts=2020-10-28T17:27:58.598556915Z caller=client.go:229 msg="error getting path" key=collectors/ring err="Get \"http://e2e-cortex-test-consul:8500/v1/kv/collectors/ring?index=25&stale=&wait=10000ms\": context canceled"
17:27:58 Killing consul
17:27:58 consul: 2020-10-28T17:27:58.983Z [ERROR] agent.server: error performing anti-entropy sync of federation state: error="context canceled"
17:27:59 Killing bigtable
17:27:59 bigtable: done
17:27:59 Killing dynamodb
17:28:00 Killing cassandra
17:28:00 cassandra: INFO  [StorageServiceShutdownHook] 2020-10-28 17:28:00,231 HintsService.java:220 - Paused hints dispatch
17:28:00 cassandra: INFO  [StorageServiceShutdownHook] 2020-10-28 17:28:00,240 Server.java:176 - Stop listening for CQL clients
17:28:00 cassandra: INFO  [StorageServiceShutdownHook] 2020-10-28 17:28:00,245 Gossiper.java:1530 - Announcing shutdown
17:28:00 cassandra: INFO  [StorageServiceShutdownHook] 2020-10-28 17:28:00,251 StorageService.java:2255 - Node /192.168.48.2 state jump to shutdown
--- FAIL: TestDeleteSeriesAllIndexBackends (69.63s)

To Reproduce It failed in this CI execution.

sandeepsukhani commented 4 years ago

I ran the test a couple of times locally, it never failed for me. The only option is to run tests with debug level logs and wait for it to fail in CI to see what all happened.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

pracucci commented 3 years ago

Closing for now. Will reopen if we see it again.

gouthamve commented 3 years ago 
=== RUN   TestDeleteSeriesAllIndexBackends
13:20:03 Starting cassandra
13:20:04 Ports for container: e2e-cortex-test-cassandra Mapping: map[9042:49954]
13:20:04 Starting dynamodb
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z
13:20:05 cassandra: CompilerOracle: dontinline
13:20:05 cassandra: org/apache/cassandra/db/transform/StoppingTransformation.stop ()V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V
13:20:05 cassandra: CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I
13:20:05 cassandra: CompilerOracle: inline
13:20:05 cassandra: org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I
13:20:05 cassandra: CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B
13:20:05 Ports for container: e2e-cortex-test-dynamodb Mapping: map[8000:49956]
13:20:05 Starting bigtable
13:20:06 dynamodb: Initializing DynamoDB Local with the following configuration:
13:20:06 dynamodb: Port:    8000
13:20:06 dynamodb: InMemory:    true
13:20:06 dynamodb: DbPath:  null
13:20:06 dynamodb: SharedDb:    true
13:20:06 dynamodb: shouldDelayTransientStatuses:    false
13:20:06 dynamodb: CorsParams:  *
13:20:06 cassandra: INFO  [main] 2021-04-28 13:20:06,478 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/cassandra.yaml
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,113 Config.java:481 - Node configuration:[allocate_tokens_for_keyspace=null; authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=true; auto_snapshot=true; back_pressure_enabled=false; back_pressure_strategy=org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.20.0.2; broadcast_rpc_address=172.20.0.2; buffer_pool_use_heap_if_exhausted=true; cas_contention_timeout_in_ms=1000; cdc_enabled=false; cdc_free_space_check_interval_ms=250; cdc_raw_directory=null; cdc_total_space_in_mb=0; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_cache_size_in_kb=2; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_compression=null; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_max_compression_buffers_in_pool=3; commitlog_periodic_queue_size=-1; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_batch_window_in_ms=NaN; commitlog_sync_period_in_ms=10000; commitlog_total_space_in_mb=null; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_compactors=null; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_replicates=null; concurrent_writes=32; counter_cache_keys_to_save=2147483647; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_cache_max_entries=1000; credentials_update_interval_in_ms=-1; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[Ljava.lang.String;@37e547da; disk_access_mode=auto; disk_failure_policy=stop; disk_optimization_estimate_percentile=0.95; disk_optimization_page_cross_chance=0.1; disk_optimization_strategy=ssd; dynamic_snitch=true; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; enable_user_defined_functions_threads=true; encryption_options=null; endpoint_snitch=SimpleSnitch; file_cache_size_in_mb=null; gc_log_threshold_in_ms=200; gc_warn_threshold_in_ms=1000; hinted_handoff_disabled_datacenters=[]; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_compression=null; hints_directory=null; hints_flush_period_in_ms=10000; incremental_backups=false; index_interval=null; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; initial_token=0; inter_dc_stream_throughput_outbound_megabits_per_sec=200; inter_dc_tcp_nodelay=false; internode_authenticator=null; internode_compression=dc; internode_recv_buff_size_in_bytes=0; internode_send_buff_size_in_bytes=0; key_cache_keys_to_save=2147483647; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.20.0.2; listen_interface=null; listen_interface_prefer_ipv6=false; listen_on_broadcast_address=false; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; max_mutation_size_in_kb=null; max_streaming_retries=3; max_value_size_in_mb=256; memtable_allocation_type=heap_buffers; memtable_cleanup_threshold=null; memtable_flush_writers=0; memtable_heap_space_in_mb=null; memtable_offheap_space_in_mb=null; min_free_space_per_drive_in_mb=50; native_transport_max_concurrent_connections=-1; native_transport_max_concurrent_connections_per_ip=-1; native_transport_max_frame_size_in_mb=256; native_transport_max_threads=128; native_transport_port=9042; native_transport_port_ssl=null; num_tokens=1; otc_backlog_expiration_interval_ms=200; otc_coalescing_enough_coalesced_messages=8; otc_coalescing_strategy=DISABLED; otc_coalescing_window_us=200; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_cache_max_entries=1000; permissions_update_interval_in_ms=-1; permissions_validity_in_ms=2000; phi_convict_threshold=8.0; prepared_statements_
13:20:07 cassandra: cache_size_mb=null; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_scheduler_id=null; request_scheduler_options=null; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_cache_max_entries=1000; roles_update_interval_in_ms=-1; roles_validity_in_ms=2000; row_cache_class_name=org.apache.cassandra.cache.OHCProvider; row_cache_keys_to_save=2147483647; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_interface=null; rpc_interface_prefer_ipv6=false; rpc_keepalive=true; rpc_listen_backlog=50; rpc_max_threads=2147483647; rpc_min_threads=16; rpc_port=9160; rpc_recv_buff_size_in_bytes=null; rpc_send_buff_size_in_bytes=null; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=org.apache.cassandra.locator.SimpleSeedProvider{seeds=172.20.0.2}; server_encryption_options=<REDACTED>; slow_query_log_timeout_in_ms=500; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=false; storage_port=7000; stream_throughput_outbound_megabits_per_sec=200; streaming_keep_alive_period_in_secs=300; streaming_socket_timeout_in_ms=86400000; thrift_framed_transport_size_in_mb=15; thrift_max_message_length_in_mb=16; thrift_prepared_statements_cache_size_mb=null; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options=org.apache.cassandra.config.TransparentDataEncryptionOptions@2b6856dd; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; unlogged_batch_across_partitions_warn_threshold=10; user_defined_function_fail_timeout=1500; user_defined_function_warn_timeout=500; user_function_timeout_policy=die; windows_timer_interval=1; write_request_timeout_in_ms=2000]
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,114 DatabaseDescriptor.java:366 - DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,114 DatabaseDescriptor.java:420 - Global memtable on-heap threshold is enabled at 430MB
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,114 DatabaseDescriptor.java:424 - Global memtable off-heap threshold is enabled at 430MB
13:20:07 bigtable: Bigtable emulator running on [::]:9035
13:20:07 Ports for container: e2e-cortex-test-bigtable Mapping: map[9035:49958]
13:20:07 Starting consul
13:20:07 cassandra: WARN  [main] 2021-04-28 13:20:07,318 DatabaseDescriptor.java:540 - Only 20.778GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,392 RateBasedBackPressure.java:123 - Initialized back-pressure with high ratio: 0.9, factor: 5, flow: FAST, window size: 2000.
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,392 DatabaseDescriptor.java:710 - Back-pressure is disabled with strategy org.apache.cassandra.net.RateBasedBackPressure{high_ratio=0.9, factor=5, flow=FAST}.
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,803 JMXServerUtils.java:249 - Configured JMX server at: service:jmx:rmi://127.0.0.1/jndi/rmi://127.0.0.1:7199/jmxrmi
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,813 CassandraDaemon.java:471 - Hostname: cassandra
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,813 CassandraDaemon.java:478 - JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_131
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,813 CassandraDaemon.java:479 - Heap size: 1.682GiB/1.682GiB
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - Code Cache Non-heap memory: init = 2555904(2496K) used = 4734784(4623K) committed = 4784128(4672K) max = 251658240(245760K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - Metaspace Non-heap memory: init = 0(0K) used = 17824488(17406K) committed = 18612224(18176K) max = -1(-1K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2132480(2082K) committed = 2359296(2304K) max = 1073741824(1048576K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - Par Eden Space Heap memory: init = 167772160(163840K) used = 97333184(95051K) committed = 167772160(163840K) max = 167772160(163840K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - Par Survivor Space Heap memory: init = 20971520(20480K) used = 0(0K) committed = 20971520(20480K) max = 20971520(20480K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,814 CassandraDaemon.java:484 - CMS Old Gen Heap memory: init = 1616904192(1579008K) used = 0(0K) committed = 1616904192(1579008K) max = 1616904192(1579008K)
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,815 CassandraDaemon.java:486 - Classpath: /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/asm-5.0.4.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.9.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/guava-18.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.5.4.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.4.0.jar:/usr/share/cassandra/lib/joda-time-2.4.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jstackjunit-0.0.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/logback-classic-1.1.3.jar:/usr/share/cassandra/lib/logback-core-1.1.3.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.0.jar:/usr/share/cassandra/lib/metrics-logback-3.1.0.jar:/usr/share/cassandra/lib/netty-all-4.0.44.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.4.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.7.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-3.11.0.jar:/usr/share/cassandra/apache-cassandra-thrift-3.11.0.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,815 CassandraDaemon.java:488 - JVM Arguments: [-Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms1742M, -Xmx1742M, -Xmn200M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dcassandra.initial_token=0, -Dcassandra.skip_wait_for_gossip_to_settle=0, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes]
13:20:07 cassandra: WARN  [main] 2021-04-28 13:20:07,951 NativeLibrary.java:187 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,951 StartupChecks.java:131 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
13:20:07 cassandra: WARN  [main] 2021-04-28 13:20:07,966 StartupChecks.java:160 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
13:20:07 cassandra: WARN  [main] 2021-04-28 13:20:07,967 StartupChecks.java:197 - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
13:20:07 cassandra: INFO  [main] 2021-04-28 13:20:07,989 SigarLibrary.java:44 - Initializing SIGAR library
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,006 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,007 StartupChecks.java:265 - Maximum number of memory map areas per process (vm.max_map_count) 262144 is too low, recommended value: 1048575, you can change it with sysctl.
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,014 StartupChecks.java:286 - Directory /var/lib/cassandra/data doesn't exist
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,018 StartupChecks.java:286 - Directory /var/lib/cassandra/commitlog doesn't exist
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,019 StartupChecks.java:286 - Directory /var/lib/cassandra/saved_caches doesn't exist
13:20:08 cassandra: WARN  [main] 2021-04-28 13:20:08,024 StartupChecks.java:286 - Directory /var/lib/cassandra/hints doesn't exist
13:20:08 cassandra: INFO  [main] 2021-04-28 13:20:08,092 QueryProcessor.java:115 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
13:20:08 cassandra: INFO  [main] 2021-04-28 13:20:08,509 ColumnFamilyStore.java:406 - Initializing system.IndexInfo
13:20:08 Ports for container: e2e-cortex-test-consul Mapping: map[8500:49960]
13:20:08 consul: ==> Starting Consul agent...
13:20:08 consul: Version: '1.8.4'
13:20:08 consul: Node ID: 'da03338a-6e2d-ecec-534e-70635dae2a69'
13:20:08 consul: Node name: 'consul'
13:20:08 consul: Datacenter: 'dc1' (Segment: '<all>')
13:20:08 consul: Server: true (Bootstrap: false)
13:20:08 consul: Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: 8502, DNS: 8600)
13:20:08 consul: Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
13:20:08 consul: Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
13:20:08 consul: ==> Log data will now stream in as it occurs:
13:20:08 consul: ==> Consul agent running!
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,602 ColumnFamilyStore.java:406 - Initializing system.batches
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,624 ColumnFamilyStore.java:406 - Initializing system.paxos
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,651 ColumnFamilyStore.java:406 - Initializing system.local
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,660 ColumnFamilyStore.java:406 - Initializing system.peers
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,673 ColumnFamilyStore.java:406 - Initializing system.peer_events
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,677 ColumnFamilyStore.java:406 - Initializing system.range_xfers
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,682 ColumnFamilyStore.java:406 - Initializing system.compaction_history
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,689 ColumnFamilyStore.java:406 - Initializing system.sstable_activity
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,697 ColumnFamilyStore.java:406 - Initializing system.size_estimates
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,708 ColumnFamilyStore.java:406 - Initializing system.available_ranges
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,726 ColumnFamilyStore.java:406 - Initializing system.transferred_ranges
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,733 ColumnFamilyStore.java:406 - Initializing system.views_builds_in_progress
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,746 ColumnFamilyStore.java:406 - Initializing system.built_views
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,751 ColumnFamilyStore.java:406 - Initializing system.hints
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,762 ColumnFamilyStore.java:406 - Initializing system.batchlog
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,770 ColumnFamilyStore.java:406 - Initializing system.prepared_statements
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,784 ColumnFamilyStore.java:406 - Initializing system.schema_keyspaces
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,788 ColumnFamilyStore.java:406 - Initializing system.schema_columnfamilies
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,793 ColumnFamilyStore.java:406 - Initializing system.schema_columns
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,803 ColumnFamilyStore.java:406 - Initializing system.schema_triggers
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,810 ColumnFamilyStore.java:406 - Initializing system.schema_usertypes
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,828 ColumnFamilyStore.java:406 - Initializing system.schema_functions
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,839 ColumnFamilyStore.java:406 - Initializing system.schema_aggregates
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,842 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
13:20:09 cassandra: INFO  [main] 2021-04-28 13:20:09,967 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
13:20:10 cassandra: INFO  [MemtableFlushWriter:1] 2021-04-28 13:20:10,667 CacheService.java:112 - Initializing key cache with capacity of 86 MBs.
13:20:10 cassandra: INFO  [MemtableFlushWriter:1] 2021-04-28 13:20:10,681 CacheService.java:134 - Initializing row cache with capacity of 0 MBs
13:20:10 cassandra: INFO  [MemtableFlushWriter:1] 2021-04-28 13:20:10,695 CacheService.java:163 - Initializing counter cache with capacity of 43 MBs
13:20:10 cassandra: INFO  [MemtableFlushWriter:1] 2021-04-28 13:20:10,697 CacheService.java:174 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,880 StorageService.java:599 - Populating token metadata from system tables
13:20:10 cassandra: INFO  [CompactionExecutor:2] 2021-04-28 13:20:10,903 BufferPool.java:230 - Global buffer pool is enabled, when pool is exhausted (max is 430.000MiB) it will allocate on heap
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,958 StorageService.java:606 - Token metadata:
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,974 ColumnFamilyStore.java:406 - Initializing system_schema.keyspaces
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,978 ColumnFamilyStore.java:406 - Initializing system_schema.tables
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,982 ColumnFamilyStore.java:406 - Initializing system_schema.columns
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,992 ColumnFamilyStore.java:406 - Initializing system_schema.triggers
13:20:10 cassandra: INFO  [main] 2021-04-28 13:20:10,998 ColumnFamilyStore.java:406 - Initializing system_schema.dropped_columns
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,007 ColumnFamilyStore.java:406 - Initializing system_schema.views
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,012 ColumnFamilyStore.java:406 - Initializing system_schema.types
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,015 ColumnFamilyStore.java:406 - Initializing system_schema.functions
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,020 ColumnFamilyStore.java:406 - Initializing system_schema.aggregates
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,027 ColumnFamilyStore.java:406 - Initializing system_schema.indexes
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,028 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
13:20:11 cassandra: INFO  [pool-3-thread-1] 2021-04-28 13:20:11,052 AutoSavingCache.java:173 - Completed loading (3 ms; 5 keys) KeyCache cache
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,071 CommitLog.java:152 - No commitlog files found; skipping replay
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,072 StorageService.java:599 - Populating token metadata from system tables
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,081 StorageService.java:606 - Token metadata:
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,164 QueryProcessor.java:162 - Preloaded 0 prepared statements
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,172 StorageService.java:617 - Cassandra version: 3.11.0
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,173 StorageService.java:618 - Thrift API version: 20.1.0
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,173 StorageService.java:619 - CQL supported versions: 3.4.4 (default: 3.4.4)
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,173 StorageService.java:621 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,200 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 86 MB and a resize interval of 60 minutes
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,205 MessagingService.java:753 - Starting Messaging Service on /172.20.0.2:7000 (eth0)
13:20:11 cassandra: WARN  [main] 2021-04-28 13:20:11,213 SystemKeyspace.java:1083 - No host ID found, created 53cfe7c3-a507-43ca-98f3-17c8e04400cc (Note: This should happen exactly once per node).
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,227 StorageService.java:706 - Loading persisted ring state
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,228 StorageService.java:819 - Starting up server gossip
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,261 StorageService.java:857 - This node will not auto bootstrap because it is configured to be a seed node.
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,265 StorageService.java:984 - Saved tokens not found. Using configuration value: [0]
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,268 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_traces, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=2}}, tables=[org.apache.cassandra.config.CFMetaData@5f589199[cfId=c5e99f16-8677-3914-b17e-960613512345,ksName=system_traces,cfName=sessions,flags=[COMPOUND],params=TableParams{comment=tracing sessions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [client command coordinator duration request started_at parameters]],partitionKeyColumns=[session_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[client, command, session_id, coordinator, request, started_at, duration, parameters],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@459f7d6e[cfId=8826e8e9-e16a-3728-8753-3bc1fc713c25,ksName=system_traces,cfName=events,flags=[COMPOUND],params=TableParams{comment=tracing events, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [activity source source_elapsed thread]],partitionKeyColumns=[session_id],clusteringColumns=[event_id],keyValidator=org.apache.cassandra.db.marshal.UUIDType,columnMetadata=[activity, event_id, session_id, source, thread, source_elapsed],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,484 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,497 ColumnFamilyStore.java:406 - Initializing system_traces.events
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,501 ColumnFamilyStore.java:406 - Initializing system_traces.sessions
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,530 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_distributed, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=3}}, tables=[org.apache.cassandra.config.CFMetaData@2ca06c38[cfId=759fffad-624b-3181-80ee-fa9a52d1f627,ksName=system_distributed,cfName=repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.TimeUUIDType),partitionColumns=[[] | [coordinator exception_message exception_stacktrace finished_at parent_id range_begin range_end started_at status participants]],partitionKeyColumns=[keyspace_name, columnfamily_name],clusteringColumns=[id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[status, id, coordinator, finished_at, participants, exception_stacktrace, parent_id, range_end, range_begin, exception_message, keyspace_name, started_at, columnfamily_name],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@138f8fd5[cfId=deabd734-b99d-3b9c-92e5-fd92eb5abf14,ksName=system_distributed,cfName=parent_repair_history,flags=[COMPOUND],params=TableParams{comment=Repair history, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [exception_message exception_stacktrace finished_at keyspace_name started_at columnfamily_names options requested_ranges successful_ranges]],partitionKeyColumns=[parent_id],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.TimeUUIDType,columnMetadata=[requested_ranges, exception_message, keyspace_name, successful_ranges, started_at, finished_at, options, exception_stacktrace, parent_id, columnfamily_names],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@6a2eb000[cfId=5582b59f-8e4e-35e1-b913-3acada51eb04,ksName=system_distributed,cfName=view_build_status,flags=[COMPOUND],params=TableParams{comment=Materialized View build status, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UUIDType),partitionColumns=[[] | [status]],partitionKeyColumns=[keyspace_name, view_name],clusteringColumns=[host_id],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.
13:20:11 cassandra: db.marshal.UTF8Type),columnMetadata=[view_name, status, keyspace_name, host_id],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,621 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,628 ColumnFamilyStore.java:406 - Initializing system_distributed.parent_repair_history
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,634 ColumnFamilyStore.java:406 - Initializing system_distributed.repair_history
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,644 ColumnFamilyStore.java:406 - Initializing system_distributed.view_build_status
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,656 StorageService.java:1439 - JOINING: Finish joining ring
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,707 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=system_auth, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[org.apache.cassandra.config.CFMetaData@786f79fc[cfId=5bc52802-de25-35ed-aeab-188eecebb090,ksName=system_auth,cfName=roles,flags=[COMPOUND],params=TableParams{comment=role definitions, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(),partitionColumns=[[] | [can_login is_superuser salted_hash member_of]],partitionKeyColumns=[role],clusteringColumns=[],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[salted_hash, member_of, role, can_login, is_superuser],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@3fb8b39a[cfId=0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d,ksName=system_auth,cfName=role_members,flags=[COMPOUND],params=TableParams{comment=role memberships lookup table, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[role],clusteringColumns=[member],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, member],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@700a091c[cfId=3afbe79f-2194-31a7-add7-f5ab90d8ec9c,ksName=system_auth,cfName=role_permissions,flags=[COMPOUND],params=TableParams{comment=permissions granted to db roles, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | [permissions]],partitionKeyColumns=[role],clusteringColumns=[resource],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[role, resource, permissions],droppedColumns={},triggers=[],indexes=[]], org.apache.cassandra.config.CFMetaData@281703e5[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,flags=[COMPOUND],params=TableParams{comment=index of db roles with permissions granted on a resource, read_repair_chance=0.0, dclocal_read_repair_chance=0.0, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=7776000, default_time_to_live=0, memtable_flush_period_in_ms=3600000, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache
13:20:11 cassandra: .cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.UTF8Type),partitionColumns=[[] | []],partitionKeyColumns=[resource],clusteringColumns=[role],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[resource, role],droppedColumns={},triggers=[],indexes=[]]], views=[], functions=[], types=[]}
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,768 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,770 ColumnFamilyStore.java:406 - Initializing system_auth.resource_role_permissons_index
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,773 ColumnFamilyStore.java:406 - Initializing system_auth.role_members
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,781 ColumnFamilyStore.java:406 - Initializing system_auth.role_permissions
13:20:11 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:11,787 ColumnFamilyStore.java:406 - Initializing system_auth.roles
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,878 NativeTransportService.java:70 - Netty using native Epoll event loop
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,936 Server.java:155 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
13:20:11 cassandra: INFO  [main] 2021-04-28 13:20:11,937 Server.java:156 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
13:20:12 cassandra: INFO  [main] 2021-04-28 13:20:12,022 CassandraDaemon.java:527 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
13:20:12 Starting table-manager
13:20:13 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:49964 9095:49962]
13:20:18 Stopping table-manager
13:20:18 Starting table-manager
13:20:19 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:49967 9095:49965]
13:20:19 Stopping table-manager
13:20:19 Starting table-manager
13:20:20 Ports for container: e2e-cortex-test-table-manager Mapping: map[80:49970 9095:49968]
13:20:20 table-manager: level=error ts=2021-04-28T13:20:20.668951495Z caller=connectionpool.go:523 module=gocql client=table-manager msg="failed to connect" address=172.20.0.2:9042 error="Keyspace 'tests' does not exist"
13:20:20 table-manager: level=error ts=2021-04-28T13:20:20.778122993Z caller=connectionpool.go:523 module=gocql client=table-manager msg="failed to connect" address=172.20.0.2:9042 error="Keyspace 'tests' does not exist"
13:20:20 cassandra: INFO  [Native-Transport-Requests-3] 2021-04-28 13:20:20,842 MigrationManager.java:310 - Create new Keyspace: KeyspaceMetadata{name=tests, params=KeyspaceParams{durable_writes=true, replication=ReplicationParams{class=org.apache.cassandra.locator.SimpleStrategy, replication_factor=1}}, tables=[], views=[], functions=[], types=[]}
13:20:20 cassandra: INFO  [Native-Transport-Requests-4] 2021-04-28 13:20:20,943 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@50191e0f[cfId=7b8365f0-a824-11eb-9b56-d7ee8570c286,ksName=tests,cfName=cortex_2677,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
13:20:20 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:20,994 ColumnFamilyStore.java:406 - Initializing tests.cortex_2677
13:20:21 cassandra: INFO  [Native-Transport-Requests-1] 2021-04-28 13:20:21,020 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@2cbe47f[cfId=7b8f25c0-a824-11eb-9b56-d7ee8570c286,ksName=tests,cfName=cortex_chunks_2676,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
13:20:21 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:21,069 ColumnFamilyStore.java:406 - Initializing tests.cortex_chunks_2676
13:20:21 cassandra: INFO  [Native-Transport-Requests-3] 2021-04-28 13:20:21,085 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@5dcb85a8[cfId=7b98e9c0-a824-11eb-9b56-d7ee8570c286,ksName=tests,cfName=cortex_chunks_2677,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
13:20:21 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:21,130 ColumnFamilyStore.java:406 - Initializing tests.cortex_chunks_2677
13:20:21 cassandra: INFO  [Native-Transport-Requests-2] 2021-04-28 13:20:21,150 MigrationManager.java:355 - Create new table: org.apache.cassandra.config.CFMetaData@617285e4[cfId=7ba2fbe0-a824-11eb-9b56-d7ee8570c286,ksName=tests,cfName=cortex_2676,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=864000, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@4c1ce6bb, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.BytesType),partitionColumns=[[] | [value]],partitionKeyColumns=[hash],clusteringColumns=[range],keyValidator=org.apache.cassandra.db.marshal.UTF8Type,columnMetadata=[hash, range, value],droppedColumns={},triggers=[],indexes=[]]
13:20:21 cassandra: INFO  [MigrationStage:1] 2021-04-28 13:20:21,202 ColumnFamilyStore.java:406 - Initializing tests.cortex_2676
13:20:21 Stopping table-manager
13:20:21 Starting distributor
13:20:21 cassandra: INFO  [OptionalTasks:1] 2021-04-28 13:20:21,968 CassandraRoleManager.java:355 - Created default superuser role 'cassandra'
13:20:22 Ports for container: e2e-cortex-test-distributor Mapping: map[80:49973 9095:49971]
13:20:22 Starting ingester
13:20:23 Ports for container: e2e-cortex-test-ingester Mapping: map[80:49977 9095:49975]
13:20:23 Starting querier
13:20:24 Ports for container: e2e-cortex-test-querier Mapping: map[80:49981 9095:49979]
13:20:34 Starting purger
13:20:35 purger: level=warn ts=2021-04-28T13:20:35.399209376Z caller=experimental.go:19 msg="experimental feature in use" feature="Delete series API"
13:20:35 Ports for container: e2e-cortex-test-purger Mapping: map[80:49985 9095:49983]
13:20:35 Stopping purger
13:20:35 Starting purger
13:20:36 purger: level=warn ts=2021-04-28T13:20:36.678726864Z caller=experimental.go:19 msg="experimental feature in use" feature="Delete series API"
13:20:36 Ports for container: e2e-cortex-test-purger Mapping: map[80:49988 9095:49986]
13:20:37 purger: level=error ts=2021-04-28T13:20:37.200508784Z caller=purger.go:240 user_id=user-1 request_id=dffb31dc msg="error removing delete plan" plan_no=0 err="open /shared/user-1:dffb31dc: no such file or directory"
    chunks_delete_series_test.go:182: 
            Error Trace:    chunks_delete_series_test.go:182
            Error:          Received unexpected error:
                            unable to find metrics [cortex_purger_delete_requests_processed_total] with expected values. Last error: <nil>. Last values: [2]
            Test:           TestDeleteSeriesAllIndexBackends
13:20:59 Killing purger
13:20:59 Killing querier
13:20:59 Killing ingester
13:21:00 Killing distributor
13:21:00 Killing consul
13:21:00 Killing bigtable
13:21:01 Killing dynamodb
13:21:01 Killing cassandra
--- FAIL: TestDeleteSeriesAllIndexBackends (58.74s)

From: https://github.com/cortexproject/cortex/pull/4137/checks?check_run_id=2457801400

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

pracucci commented 3 years ago

Still valid

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.