search

strapdata / helm-charts

Strapdata HELM charts
24 stars 11 forks source link

Helm chart not deploying #3

Closed steebchen closed 5 years ago

steebchen commented 5 years ago

I wanted to create an Elassandra cluster with helm but it does not start: CrashLoopBackOff: Back-off 2m40s restarting failed container=elassandra pod=elassandra-0_elassandra

I am using image strapdata/elassandra:6.2.3.15 just like described in the readme. The logs don't seem to report any error. I am using a small machine (1vcpu, 2g RAM) but I can't seem to find any performance-related error or anything similar.

Selection_081

Full container logs ``` /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-init.d/* Starting with JVM debug address=4242 suspend=n Starting cassandra with org.apache.cassandra.service.ElassandraDaemon CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset (Lorg/apache/cassandra/io/util/DataInputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns; CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubset (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;ILorg/apache/cassandra/io/util/DataOutputPlus;)V CompilerOracle: dontinline org/apache/cassandra/db/Columns$Serializer.serializeLargeSubsetSize (Ljava/util/Collection;ILorg/apache/cassandra/db/Columns;I)I CompilerOracle: dontinline org/apache/cassandra/db/commitlog/AbstractCommitLogSegmentManager.advanceAllocatingFrom (Lorg/apache/cassandra/db/commitlog/CommitLogSegment;)V CompilerOracle: dontinline org/apache/cassandra/db/transform/BaseIterator.tryGetMoreContents ()Z CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stop ()V CompilerOracle: dontinline org/apache/cassandra/db/transform/StoppingTransformation.stopInPartition ()V CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.doFlush (I)V CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeExcessSlow ()V CompilerOracle: dontinline org/apache/cassandra/io/util/BufferedDataOutputStreamPlus.writeSlow (JI)V CompilerOracle: dontinline org/apache/cassandra/io/util/RebufferingInputStream.readPrimitiveSlowly (I)J CompilerOracle: inline org/apache/cassandra/db/rows/UnfilteredSerializer.serializeRowBody (Lorg/apache/cassandra/db/rows/Row;ILorg/apache/cassandra/db/SerializationHeader;Lorg/apache/cassandra/io/util/DataOutputPlus;)V CompilerOracle: inline org/apache/cassandra/io/util/Memory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/io/util/SafeMemory.checkBounds (JJ)V CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.selectBoundary (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;II)I CompilerOracle: inline org/apache/cassandra/utils/AsymmetricOrdering.strictnessOfLessThan (Lorg/apache/cassandra/utils/AsymmetricOrdering/Op;)I CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.indexes (Lorg/apache/cassandra/utils/IFilter/FilterKey;)[J CompilerOracle: inline org/apache/cassandra/utils/BloomFilter.setIndexes (JJIJ[J)V CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare (Ljava/nio/ByteBuffer;[B)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compare ([BLjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/ByteBufferUtil.compareUnsigned (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/lang/Object;JI)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/lang/Object;JILjava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/FastByteOperations$UnsafeOperations.compareTo (Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)I CompilerOracle: inline org/apache/cassandra/utils/vint/VIntCoding.encodeVInt (JI)[B Listening for transport dt_socket at address: 4242 2019-05-10 12:44:37,854 WARN [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:480) Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2503. You can override this in cassandra.yaml 2019-05-10 12:44:37,867 WARN [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:507) Small cdc volume detected at /var/lib/cassandra/cdc_raw; setting cdc_total_space_in_mb to 1251. You can override this in cassandra.yaml 2019-05-10 12:44:38,137 WARN [main] org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:556) Only 9.745GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots 2019-05-10 12:44:39,975 WARN [main] org.elasticsearch.node.Node.warnIfPreRelease(Node.java:549) version [6.2.3-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production 2019-05-10 12:44:46,214 WARN [main] org.apache.cassandra.db.SystemKeyspace.getLocalHostId(SystemKeyspace.java:1087) No host ID found, created 7d3dadfe-8d34-4cd1-bae7-c683e65930d3 (Note: This should happen exactly once per node). 2019-05-10 12:44:53,939 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:484) Hostname: elassandra-0.elassandra.elassandra.svc.cluster.local 2019-05-10 12:44:53,941 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:491) JVM vendor/version: OpenJDK 64-Bit Server VM/1.8.0_212 2019-05-10 12:44:53,950 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:492) Heap size: 1.950GiB/1.950GiB 2019-05-10 12:44:53,954 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) Code Cache Non-heap memory: init = 2555904(2496K) used = 12664320(12367K) committed = 12713984(12416K) max = 251658240(245760K) 2019-05-10 12:44:53,955 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) Metaspace Non-heap memory: init = 0(0K) used = 57594856(56244K) committed = 61181952(59748K) max = -1(-1K) 2019-05-10 12:44:53,956 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) Compressed Class Space Non-heap memory: init = 0(0K) used = 7463208(7288K) committed = 8798208(8592K) max = 1073741824(1048576K) 2019-05-10 12:44:53,956 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) Par Eden Space Heap memory: init = 429522944(419456K) used = 429522944(419456K) committed = 429522944(419456K) max = 429522944(419456K) 2019-05-10 12:44:53,958 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) Par Survivor Space Heap memory: init = 53673984(52416K) used = 35288920(34461K) committed = 53673984(52416K) max = 53673984(52416K) 2019-05-10 12:44:53,964 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:497) CMS Old Gen Heap memory: init = 1610612736(1572864K) used = 6803352(6643K) committed = 1610612736(1572864K) max = 1610612736(1572864K) 2019-05-10 12:44:53,983 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:499) Classpath: /etc/cassandra:/usr/share/cassandra/lib/HdrHistogram-2.1.9.jar:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/ant-1.7.0.jar:/usr/share/cassandra/lib/ant-launcher-1.7.0.jar:/usr/share/cassandra/lib/antlr-3.5.2.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/caffeine-2.2.6.jar:/usr/share/cassandra/lib/cassandra-all-3.11.4.1.jar:/usr/share/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/usr/share/cassandra/lib/cassandra-thrift-3.11.4.1.jar:/usr/share/cassandra/lib/commons-cli-1.3.1.jar:/usr/share/cassandra/lib/commons-codec-1.10.jar:/usr/share/cassandra/lib/commons-lang3-3.4.jar:/usr/share/cassandra/lib/commons-logging-1.2.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrent-trees-2.4.0.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/ecj-4.4.2.jar:/usr/share/cassandra/lib/elasticsearch-6.2.3.15.jar:/usr/share/cassandra/lib/elasticsearch-cli-6.2.3.15.jar:/usr/share/cassandra/lib/elasticsearch-core-6.2.3.15.jar:/usr/share/cassandra/lib/elasticsearch-launchers-6.2.3.15.jar:/usr/share/cassandra/lib/fastutil-6.5.7.jar:/usr/share/cassandra/lib/guava-19.0.jar:/usr/share/cassandra/lib/hibernate-validator-4.3.0.Final.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/hppc-0.7.1.jar:/usr/share/cassandra/lib/httpclient-4.5.2.jar:/usr/share/cassandra/lib/httpcore-4.4.5.jar:/usr/share/cassandra/lib/jackson-core-2.8.10.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.13.jar:/usr/share/cassandra/lib/jackson-dataformat-cbor-2.8.10.jar:/usr/share/cassandra/lib/jackson-dataformat-smile-2.8.10.jar:/usr/share/cassandra/lib/jackson-dataformat-yaml-2.8.10.jar:/usr/share/cassandra/lib/jackso n-mapper-asl-1.9.13.jar:/usr/share/cassandra/lib/javassist-3.20.0-GA.jar:/usr/share/cassandra/lib/javax.inject-1.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jboss-logging-3.1.0.CR2.jar:/usr/share/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/jctools-core-1.2.1.jar:/usr/share/cassandra/lib/jflex-1.6.0.jar:/usr/share/cassandra/lib/jna-4.5.1.jar:/usr/share/cassandra/lib/joda-time-2.9.9.jar:/usr/share/cassandra/lib/jopt-simple-5.0.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/jts-1.13.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/log4j-api-2.9.1.jar:/usr/share/cassandra/lib/log4j-core-2.9.1.jar:/usr/share/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/usr/share/cassandra/lib/log4j-to-slf4j-2.9.1.jar:/usr/share/cassandra/lib/logback-classic-1.1.8.jar:/usr/share/cassandra/lib/logback-core-1.1.8.jar:/usr/share/cassandra/lib/lucene-analyzers-common-7.2.1.jar:/usr/share/cassandra/lib/lucene-backward-codecs-7.2.1.jar:/usr/share/cassandra/lib/lucene-core-7.2.1.jar:/usr/share/cassandra/lib/lucene-grouping-7.2.1.jar:/usr/share/cassandra/lib/lucene-highlighter-7.2.1.jar:/usr/share/cassandra/lib/lucene-join-7.2.1.jar:/usr/share/cassandra/lib/lucene-memory-7.2.1.jar:/usr/share/cassandra/lib/lucene-misc-7.2.1.jar:/usr/share/cassandra/lib/lucene-queries-7.2.1.jar:/usr/share/cassandra/lib/lucene-queryparser-7.2.1.jar:/usr/share/cassandra/lib/lucene-sandbox-7.2.1.jar:/usr/share/cassandra/lib/lucene-spatial-7.2.1.jar:/usr/share/cassandra/lib/lucene-spatial-extras-7.2.1.jar:/usr/share/cassandra/lib/lucene-spatial3d-7.2.1.jar:/usr/share/cassandra/lib/lucene-suggest-7.2.1.jar:/usr/share/cassandra/lib/lz4-1.3.0.jar:/usr/share/cassandra/lib/metrics-core-3.1.0.jar:/usr/share/cassandra/lib/metrics-jvm-3.1.5.jar:/usr/share/cassandra/lib/netty-all-4.1.16.Final.jar:/usr/share/cassandra/lib/ohc-core-0.4.4.jar:/usr/share/cassandra/lib/ohc-core-j8-0.4.4.jar:/usr/share/cassandra/lib/plugin-classloader-6.2.3.15.jar:/usr/share/cassandra/lib/plugin-cli-6 .2.3.15.jar:/usr/share/cassandra/lib/reporter-config-base-3.0.3.jar:/usr/share/cassandra/lib/reporter-config3-3.0.3.jar:/usr/share/cassandra/lib/securesm-1.2.jar:/usr/share/cassandra/lib/sigar-1.6.4.jar:/usr/share/cassandra/lib/slf4j-api-1.7.25.jar:/usr/share/cassandra/lib/snakeyaml-1.17.jar:/usr/share/cassandra/lib/snappy-java-1.1.1.7.jar:/usr/share/cassandra/lib/spatial4j-0.6.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/t-digest-3.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/lib/validation-api-1.0.0.GA.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar 2019-05-10 12:44:53,985 INFO [main] org.apache.cassandra.service.CassandraDaemon.logSystemInfo(CassandraDaemon.java:501) JVM Arguments: [-Dcassandra.ring_delay_ms=60000, -XX:+ExitOnOutOfMemoryError, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -XX:MaxRAMFraction=2, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Xloggc:/var/log/cassandra/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xss1m, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Xms2048M, -Xmx2048M, -Xmn512M, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/usr/share/cassandra/lib/sigar-bin, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -Dcassan dra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -ea, -Xdebug, -Xnoagent, -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=4242, -Djava.awt.headless=true, -Dlogback.configurationFile=/etc/cassandra/logback.xml, -Dcassandra.logdir=/var/log/cassandra, -Dcassandra.storagedir=/var/lib/cassandra, -Dcassandra-foreground=yes] ```
Details Normal Scheduled Successfully assigned elassandra/elassandra-0 to worker1 27 minutes ago Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-8e34343d-731e-11e9-ba6b-960000250e88" 27 minutes ago Normal Pulling pulling image "busybox" 27 minutes ago Normal Pulled Successfully pulled image "busybox" 27 minutes ago Normal Created Created container 27 minutes ago Normal Started Started container 27 minutes ago Normal Pulling pulling image "busybox" 27 minutes ago Normal Pulled Successfully pulled image "busybox" 27 minutes ago Normal Created Created container 27 minutes ago Normal Started Started container 27 minutes ago Normal Pulling pulling image "strapdata/elassandra:6.2.3.11" 27 minutes ago Normal Pulled Successfully pulled image "strapdata/elassandra:6.2.3.11" 26 minutes ago Normal Created Created container 24 minutes ago Normal Pulled Container image "strapdata/elassandra:6.2.3.11" already present on machine 24 minutes ago Normal Started Started container 24 minutes ago Warning Unhealthy Liveness probe failed: dial tcp 10.42.192.9:7000: connect: connection refused 17 minutes ago Warning BackOff Back-off restarting failed container 12 minutes ago Normal Scheduled Successfully assigned elassandra/elassandra-0 to worker1 9 minutes ago Normal SuccessfulAttachVolume AttachVolume.Attach succeeded for volume "pvc-8e34343d-731e-11e9-ba6b-960000250e88" 9 minutes ago Normal Pulled Container image "busybox" already present on machine 9 minutes ago Normal Created Created container 9 minutes ago Normal Started Started container 9 minutes ago Normal Pulling pulling image "busybox" 9 minutes ago Normal Pulled Successfully pulled image "busybox" 9 minutes ago Normal Created Created container 9 minutes ago Normal Started Started container 9 minutes ago Normal Pulling pulling image "strapdata/elassandra:6.2.3.15" 8 minutes ago Normal Pulled Successfully pulled image "strapdata/elassandra:6.2.3.15" 8 minutes ago Normal Created Created container 6 minutes ago Normal Pulled Container image "strapdata/elassandra:6.2.3.15" already present on machine 6 minutes ago Normal Started Started container 6 minutes ago Warning Unhealthy Liveness probe failed: dial tcp 10.42.192.9:7000: connect: connection refused 5 minutes ago Warning BackOff Back-off restarting failed container 4 minutes ago
Elassandra Pod Events Normal | Scheduled | Successfully assigned elassandra/elassandra-0 to worker1 | 27 minutes ago -- | -- | -- | -- Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-8e34343d-731e-11e9-ba6b-960000250e88" | 27 minutes ago Normal | Pulling | pulling image "busybox" | 27 minutes ago Normal | Pulled | Successfully pulled image "busybox" | 27 minutes ago Normal | Created | Created container | 27 minutes ago Normal | Started | Started container | 27 minutes ago Normal | Pulling | pulling image "busybox" | 27 minutes ago Normal | Pulled | Successfully pulled image "busybox" | 27 minutes ago Normal | Created | Created container | 27 minutes ago Normal | Started | Started container | 27 minutes ago Normal | Pulling | pulling image "strapdata/elassandra:6.2.3.11" | 27 minutes ago Normal | Pulled | Successfully pulled image "strapdata/elassandra:6.2.3.11" | 26 minutes ago Normal | Created | Created container | 24 minutes ago Normal | Pulled | Container image "strapdata/elassandra:6.2.3.11" already present on machine | 24 minutes ago Normal | Started | Started container | 24 minutes ago Warning | Unhealthy | Liveness probe failed: dial tcp 10.42.192.9:7000: connect: connection refused | 17 minutes ago Warning | BackOff | Back-off restarting failed container | 12 minutes ago Normal | Scheduled | Successfully assigned elassandra/elassandra-0 to worker1 | 9 minutes ago Normal | SuccessfulAttachVolume | AttachVolume.Attach succeeded for volume "pvc-8e34343d-731e-11e9-ba6b-960000250e88" | 9 minutes ago Normal | Pulled | Container image "busybox" already present on machine | 9 minutes ago Normal | Created | Created container | 9 minutes ago Normal | Started | Started container | 9 minutes ago Normal | Pulling | pulling image "busybox" | 9 minutes ago Normal | Pulled | Successfully pulled image "busybox" | 9 minutes ago Normal | Created | Created container | 9 minutes ago Normal | Started | Started container | 9 minutes ago Normal | Pulling | pulling image "strapdata/elassandra:6.2.3.15" | 8 minutes ago Normal | Pulled | Successfully pulled image "strapdata/elassandra:6.2.3.15" | 8 minutes ago Normal | Created | Created container | 6 minutes ago Normal | Pulled | Container image "strapdata/elassandra:6.2.3.15" already present on machine | 6 minutes ago Normal | Started | Started container | 6 minutes ago Warning | Unhealthy | Liveness probe failed: dial tcp 10.42.192.9:7000: connect: connection refused | 5 minutes ago Warning | BackOff | Back-off restarting failed container | 4 minutes ago
steebchen commented 5 years ago

Ok well, it just needed a lot of time! After around 10-15 minutes it's in state "running". I guess it just takes a long time on small test instances.