search

typhon-project / typhondl

http://www.typhon-project.org
Eclipse Public License 2.0
1 stars 0 forks source link

Cassandra keeps crashing #41

Closed tvdstorm closed 4 years ago

tvdstorm commented 4 years ago

After a while after startup Cassandra crashes with exit code 137

See title.

Docker Compose / Kubernetes

If the bug happens when deploying the Polystore:

INFO  [main] 2020-06-04 14:57:37,545 CassandraDaemon.java:502 - Compressed Class Space Non-heap memory: init = 0(0K) used = 2062440(2014K) committed = 2228224(2176K) max = 1073741824(1048576K)
INFO  [main] 2020-06-04 14:57:37,546 CassandraDaemon.java:502 - Par Eden Space Heap memory: init = 208142336(203264K) used = 87453080(85403K) committed = 208142336(203264K) max = 208142336(203264K)
INFO  [main] 2020-06-04 14:57:37,547 CassandraDaemon.java:502 - Par Survivor Space Heap memory: init = 25952256(25344K) used = 0(0K) committed = 25952256(25344K) max = 25952256(25344K)
INFO  [main] 2020-06-04 14:57:37,548 CassandraDaemon.java:502 - CMS Old Gen Heap memory: init = 782237696(763904K) used = 0(0K) committed = 782237696(763904K) max = 782237696(763904K)
INFO  [main] 2020-06-04 14:57:37,548 CassandraDaemon.java:504 - Classpath: /etc/cassandra:/opt/cassandra/build/classes/main:/opt/cassandra/build/classes/thrift:/opt/cassandra/lib/HdrHistogram-2.1.9.jar:/opt/cassandra/lib/ST4-4.0.8.jar:/opt/cassandra/lib/airline-0.6.jar:/opt/cassandra/lib/antlr-runtime-3.5.2.jar:/opt/cassandra/lib/apache-cassandra-3.11.6.jar:/opt/cassandra/lib/apache-cassandra-thrift-3.11.6.jar:/opt/cassandra/lib/asm-5.0.4.jar:/opt/cassandra/lib/caffeine-2.2.6.jar:/opt/cassandra/lib/cassandra-driver-core-3.0.1-shaded.jar:/opt/cassandra/lib/commons-cli-1.1.jar:/opt/cassandra/lib/commons-codec-1.9.jar:/opt/cassandra/lib/commons-lang3-3.1.jar:/opt/cassandra/lib/commons-math3-3.2.jar:/opt/cassandra/lib/compress-lzf-0.8.4.jar:/opt/cassandra/lib/concurrent-trees-2.4.0.jar:/opt/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/opt/cassandra/lib/disruptor-3.0.1.jar:/opt/cassandra/lib/ecj-4.4.2.jar:/opt/cassandra/lib/guava-18.0.jar:/opt/cassandra/lib/high-scale-lib-1.0.6.jar:/opt/cassandra/lib/hppc-0.5.4.jar:/opt/cassandra/lib/jackson-core-asl-1.9.13.jar:/opt/cassandra/lib/jackson-mapper-asl-1.9.13.jar:/opt/cassandra/lib/jamm-0.3.0.jar:/opt/cassandra/lib/javax.inject.jar:/opt/cassandra/lib/jbcrypt-0.3m.jar:/opt/cassandra/lib/jcl-over-slf4j-1.7.7.jar:/opt/cassandra/lib/jctools-core-1.2.1.jar:/opt/cassandra/lib/jflex-1.6.0.jar:/opt/cassandra/lib/jna-4.2.2.jar:/opt/cassandra/lib/joda-time-2.4.jar:/opt/cassandra/lib/json-simple-1.1.jar:/opt/cassandra/lib/jstackjunit-0.0.1.jar:/opt/cassandra/lib/libthrift-0.9.2.jar:/opt/cassandra/lib/log4j-over-slf4j-1.7.7.jar:/opt/cassandra/lib/logback-classic-1.1.3.jar:/opt/cassandra/lib/logback-core-1.1.3.jar:/opt/cassandra/lib/lz4-1.3.0.jar:/opt/cassandra/lib/metrics-core-3.1.5.jar:/opt/cassandra/lib/metrics-jvm-3.1.5.jar:/opt/cassandra/lib/metrics-logback-3.1.5.jar:/opt/cassandra/lib/netty-all-4.0.44.Final.jar:/opt/cassandra/lib/ohc-core-0.4.4.jar:/opt/cassandra/lib/ohc-core-j8-0.4.4.jar:/opt/cassandra/lib/reporter-config-base-3.0.3.jar:/opt/cassandra/lib/reporter-config3-3.0.3.jar:/opt/cassandra/lib/sigar-1.6.4.jar:/opt/cassandra/lib/slf4j-api-1.7.7.jar:/opt/cassandra/lib/snakeyaml-1.11.jar:/opt/cassandra/lib/snappy-java-1.1.1.7.jar:/opt/cassandra/lib/snowball-stemmer-1.3.0.581.1.jar:/opt/cassandra/lib/stream-2.5.2.jar:/opt/cassandra/lib/thrift-server-0.3.7.jar:/opt/cassandra/lib/jsr223/*/*.jar:/opt/cassandra/lib/jamm-0.3.0.jar
INFO  [main] 2020-06-04 14:57:37,549 CassandraDaemon.java:506 - JVM Arguments: [-Xloggc:/opt/cassandra/logs/gc.log, -ea, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+AlwaysPreTouch, -XX:-UseBiasedLocking, -XX:+UseTLAB, -XX:+ResizeTLAB, -XX:+UseNUMA, -XX:+PerfDisableSharedMem, -Djava.net.preferIPv4Stack=true, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:+CMSClassUnloadingEnabled, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Xms994M, -Xmx994M, -Xmn248M, -XX:+UseCondCardMark, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -Dcassandra.jmx.local.port=7199, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes]
WARN  [main] 2020-06-04 14:57:37,618 NativeLibrary.java:189 - Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  [main] 2020-06-04 14:57:37,619 StartupChecks.java:140 - jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
WARN  [main] 2020-06-04 14:57:37,619 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
INFO  [main] 2020-06-04 14:57:37,623 SigarLibrary.java:44 - Initializing SIGAR library
WARN  [main] 2020-06-04 14:57:37,639 SigarLibrary.java:174 - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : true 
WARN  [main] 2020-06-04 14:57:37,649 StartupChecks.java:311 - Maximum number of memory map areas per process (vm.max_map_count) 262144 is too low, recommended value: 1048575, you can change it with sysctl.
INFO  [main] 2020-06-04 14:57:37,804 QueryProcessor.java:116 - Initialized prepared statement caches with 10 MB (native) and 10 MB (Thrift)
INFO  [main] 2020-06-04 14:57:38,348 ColumnFamilyStore.java:427 - Initializing system.IndexInfo
INFO  [main] 2020-06-04 14:57:39,285 ColumnFamilyStore.java:427 - Initializing system.batches
INFO  [main] 2020-06-04 14:57:39,297 ColumnFamilyStore.java:427 - Initializing system.paxos
INFO  [main] 2020-06-04 14:57:39,312 ColumnFamilyStore.java:427 - Initializing system.local
INFO  [SSTableBatchOpen:1] 2020-06-04 14:57:39,354 BufferPool.java:234 - Global buffer pool is enabled, when pool is exhausted (max is 242.000MiB) it will allocate on heap
INFO  [main] 2020-06-04 14:57:39,642 CacheService.java:100 - Initializing key cache with capacity of 48 MBs.
INFO  [main] 2020-06-04 14:57:39,651 CacheService.java:122 - Initializing row cache with capacity of 0 MBs
INFO  [main] 2020-06-04 14:57:39,653 CacheService.java:151 - Initializing counter cache with capacity of 24 MBs
INFO  [main] 2020-06-04 14:57:39,655 CacheService.java:162 - Scheduling counter cache save to every 7200 seconds (going to save all keys).
INFO  [main] 2020-06-04 14:57:39,668 ColumnFamilyStore.java:427 - Initializing system.peers
INFO  [main] 2020-06-04 14:57:39,676 ColumnFamilyStore.java:427 - Initializing system.peer_events
INFO  [main] 2020-06-04 14:57:39,685 ColumnFamilyStore.java:427 - Initializing system.range_xfers
INFO  [main] 2020-06-04 14:57:39,694 ColumnFamilyStore.java:427 - Initializing system.compaction_history
INFO  [main] 2020-06-04 14:57:39,705 ColumnFamilyStore.java:427 - Initializing system.sstable_activity
INFO  [main] 2020-06-04 14:57:39,726 ColumnFamilyStore.java:427 - Initializing system.size_estimates
INFO  [main] 2020-06-04 14:57:39,751 ColumnFamilyStore.java:427 - Initializing system.available_ranges
INFO  [main] 2020-06-04 14:57:39,768 ColumnFamilyStore.java:427 - Initializing system.transferred_ranges
INFO  [main] 2020-06-04 14:57:39,785 ColumnFamilyStore.java:427 - Initializing system.views_builds_in_progress
INFO  [main] 2020-06-04 14:57:39,797 ColumnFamilyStore.java:427 - Initializing system.built_views
INFO  [main] 2020-06-04 14:57:39,810 ColumnFamilyStore.java:427 - Initializing system.hints
INFO  [main] 2020-06-04 14:57:39,823 ColumnFamilyStore.java:427 - Initializing system.batchlog
INFO  [main] 2020-06-04 14:57:39,833 ColumnFamilyStore.java:427 - Initializing system.prepared_statements
INFO  [main] 2020-06-04 14:57:39,842 ColumnFamilyStore.java:427 - Initializing system.schema_keyspaces
INFO  [main] 2020-06-04 14:57:39,852 ColumnFamilyStore.java:427 - Initializing system.schema_columnfamilies
INFO  [main] 2020-06-04 14:57:39,860 ColumnFamilyStore.java:427 - Initializing system.schema_columns
INFO  [main] 2020-06-04 14:57:39,869 ColumnFamilyStore.java:427 - Initializing system.schema_triggers
INFO  [main] 2020-06-04 14:57:39,879 ColumnFamilyStore.java:427 - Initializing system.schema_usertypes
INFO  [main] 2020-06-04 14:57:39,888 ColumnFamilyStore.java:427 - Initializing system.schema_functions
INFO  [main] 2020-06-04 14:57:39,897 ColumnFamilyStore.java:427 - Initializing system.schema_aggregates
INFO  [main] 2020-06-04 14:57:39,899 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO  [main] 2020-06-04 14:57:40,067 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO  [main] 2020-06-04 14:57:40,132 ColumnFamilyStore.java:427 - Initializing system_schema.keyspaces
INFO  [main] 2020-06-04 14:57:40,152 ColumnFamilyStore.java:427 - Initializing system_schema.tables
INFO  [main] 2020-06-04 14:57:40,175 ColumnFamilyStore.java:427 - Initializing system_schema.columns
INFO  [main] 2020-06-04 14:57:40,196 ColumnFamilyStore.java:427 - Initializing system_schema.triggers
INFO  [main] 2020-06-04 14:57:40,222 ColumnFamilyStore.java:427 - Initializing system_schema.dropped_columns
INFO  [main] 2020-06-04 14:57:40,239 ColumnFamilyStore.java:427 - Initializing system_schema.views
INFO  [main] 2020-06-04 14:57:40,260 ColumnFamilyStore.java:427 - Initializing system_schema.types
INFO  [main] 2020-06-04 14:57:40,278 ColumnFamilyStore.java:427 - Initializing system_schema.functions
INFO  [main] 2020-06-04 14:57:40,298 ColumnFamilyStore.java:427 - Initializing system_schema.aggregates
INFO  [main] 2020-06-04 14:57:40,315 ColumnFamilyStore.java:427 - Initializing system_schema.indexes
INFO  [main] 2020-06-04 14:57:40,334 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
INFO  [main] 2020-06-04 14:57:40,823 StorageService.java:639 - Populating token metadata from system tables
INFO  [main] 2020-06-04 14:57:40,894 StorageService.java:646 - Token metadata: Normal Tokens:
/172.24.0.2:[-9110933509675858318, -9102611499644548251, -9001737776322486288, -8975873469575017325, -8837461651877734670, -8773843698617924603, -8758878423063659471, -8722429166421776128, -8664411308647268882, -8524256260386826074, -8433813062076946300, -8364338713413255611, -8272434170905034148, -8250173416815130999, -8242864214346252118, -8151506272481111994, -8145273435038356322, -8112169723085329200, -8099905017693175785, -8049239708904164219, -7968190734134387539, -7961821912880481054, -7861673710797337128, -7803193250664319567, -7761654926480456320, -7753071081582155110, -7716434767383563062, -7703478151506933522, -7667385505001636252, -7573081657994387310, -7498085340912766529, -7262902446336380808, -7251524338625882897, -7150792793698714483, -7051227101941971489, -6937566777559217667, -6882781776583428203, -6841648582289211519, -6808051142755243376, -6556518871051652247, -6491746420490062134, -6487005356184508567, -6243182081147053309, -6159596494630924657, -6157146023298066843, -6146175203928400645, -6142969631438936117, -6126002871893431585, -5920235713421339395, -5920136931690786407, -5917780388365449643, -5877728891796509681, -5771980991761507205, -5762412061073399223, -5716905827887267025, -5648600272378206780, -5547013241722777696, -5511374159120223567, -5482824550154447206, -5454723760173530383, -5452143533398565966, -5432392305226596815, -5291279675998473229, -5272191305183528260, -4991437457062301792, -4985351874494069547, -4922982703881004170, -4904661581317682116, -4892526294310479428, -4806357079630791060, -4803270176994016204, -4417500326515754388, -4329151097388498479, -4263835667954975769, -4235607606923041824, -4210996583599480813, -4057296430299274689, -4005498571770791858, -3975349253590109549, -3973302274850547613, -3729856880657071205, -3464023026539912006, -3459142575638244171, -3205527083128015075, -3200559769811244640, -3046649129932838731, -3035348897108458816, -3031302169089722822, -3016262674245973135, -3001506412211442866, -2994107318057681270, -2975501814149339699, -2943017271218194837, -2935561066193544946, -2772582044931106195, -2739180351991220065, -2677500664065503239, -2654391712727460815, -2619691185627366392, -2453253793693736203, -2292621609128040373, -2234848256170997657, -2154815911101813369, -2110205886033303587, -1967283723463737447, -1952146525255244809, -1952099844906840407, -1804608021577009340, -1781400591603052642, -1666532148166742492, -1666406857126911911, -1625340911234825905, -1529288047900654075, -1509777577867457571, -1164481670525738906, -1048956726713134069, -944180884564279203, -852793047917314299, -698822087983221945, -519440922602775867, -253595969341901364, -163118885196205216, -103116434733518110, -86176174993960944, -54447141358353074, -4198890977083177, 58504055030052325, 68548111731399249, 819953373549108751, 929816423490035777, 934013651077785986, 938189166868016063, 960797375572478319, 1033153838073041501, 1097267942896464148, 1193013598325673416, 1284772976847394984, 1320127336571328005, 1378257238614172802, 1422015501900088341, 1542652570763190037, 1570704103201550353, 1593258365989284513, 1734554029740605164, 1764981937919860282, 1791808318474883363, 1859462714919649069, 1998290027384679244, 2144866524064245853, 2155177995668344947, 2190932587520151246, 2257086790475107177, 2257385464347349956, 2381870617274124349, 2397840855774907645, 2399269510510401331, 2433915487311493821, 2504590128898424312, 2605689806265455559, 2610474816332908928, 2623495388945848970, 2735173145791963728, 2941427638520635163, 2970796981977488007, 3014038165768411751, 3035985609431873118, 3075988112337771145, 3102932024446510008, 3212746388243022207, 3311141487510929052, 3315457812142325147, 3338599368204141878, 3502283913697505486, 3552492310341455874, 3604265326402142097, 3679202131987159506, 3691620740306109668, 3702949411028890124, 3730052637853276442, 3945006239944586196, 3953205637391013363, 3993556841901751254, 4067548067508889695, 4183604048826263812, 4283635524549753952, 4419212556139899837, 4464858001666556739, 4482777682814990149, 4487929526037418918, 4508809147463911258, 4745339967000974285, 4798701344157939567, 4854312140184635553, 4915412132452654332, 4933729472089285338, 4961224906093174214, 5017077486649907794, 5038929434998756818, 5221305255947439503, 5257832277597375611, 5296355215051848221, 5353542202830618964, 5355596435910954390, 5381162871330122089, 5501207320196372753, 5560381405816917251, 5637782905183952359, 5723751050885056170, 5751678340633690896, 5804243486141281578, 5807931021029301987, 5941833519557251700, 6157232605188408562, 6225538843243971609, 6242084409547769371, 6312860401113548583, 6405641821358187003, 6413733599795408160, 6424111909754483342, 6469522230448126892, 6508511985167266257, 6565920316640122519, 6650737856263949280, 6658111773513259562, 6698718517225390437, 6921918798840753839, 6929206370507911655, 6967360449963326965, 7079818226152792212, 7082731990497895116, 7111850088750472142, 7159503226210146458, 7195309720212246741, 7209557517270816728, 7308105619115492685, 7554378603211313217, 7596751173924289612, 7598828588446199130, 7682517070329150364, 7879917327111729204, 8092683237278426674, 8247608211390827235, 8320720559951483726, 8382402346684646017, 8504580392038796988, 8642850611960367817, 8645578445312027505, 8678876449429709045, 8688658085766675495, 8742936246973296635, 8870793463070788351, 8899703912542396363, 8946299000152352179, 9058168544544360453, 9166168629418498168, 9205868291605555732]

INFO  [main] 2020-06-04 14:57:41,061 ColumnFamilyStore.java:427 - Initializing system_distributed.parent_repair_history
INFO  [main] 2020-06-04 14:57:41,110 ColumnFamilyStore.java:427 - Initializing system_distributed.repair_history
INFO  [main] 2020-06-04 14:57:41,133 ColumnFamilyStore.java:427 - Initializing system_distributed.view_build_status
INFO  [main] 2020-06-04 14:57:41,146 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_distributed as storage service is not initialized
INFO  [main] 2020-06-04 14:57:41,153 ColumnFamilyStore.java:427 - Initializing system_auth.resource_role_permissons_index
INFO  [main] 2020-06-04 14:57:41,163 ColumnFamilyStore.java:427 - Initializing system_auth.role_members
INFO  [main] 2020-06-04 14:57:41,174 ColumnFamilyStore.java:427 - Initializing system_auth.role_permissions
INFO  [main] 2020-06-04 14:57:41,183 ColumnFamilyStore.java:427 - Initializing system_auth.roles
INFO  [main] 2020-06-04 14:57:41,187 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_auth as storage service is not initialized
INFO  [main] 2020-06-04 14:57:41,194 ColumnFamilyStore.java:427 - Initializing system_traces.events
INFO  [main] 2020-06-04 14:57:41,212 ColumnFamilyStore.java:427 - Initializing system_traces.sessions
INFO  [main] 2020-06-04 14:57:41,223 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
INFO  [pool-3-thread-1] 2020-06-04 14:57:41,226 AutoSavingCache.java:174 - Completed loading (1 ms; 11 keys) KeyCache cache
INFO  [main] 2020-06-04 14:57:41,246 CommitLog.java:147 - Replaying /opt/cassandra/data/commitlog/CommitLog-6-1591282280616.log, /opt/cassandra/data/commitlog/CommitLog-6-1591282280617.log
INFO  [main] 2020-06-04 14:57:41,268 CommitLogReader.java:105 - Skipping playback of empty log: CommitLog-6-1591282280617.log
INFO  [main] 2020-06-04 14:57:41,706 CommitLog.java:149 - Log replay complete, 25 replayed mutations
INFO  [main] 2020-06-04 14:57:41,707 StorageService.java:639 - Populating token metadata from system tables
INFO  [main] 2020-06-04 14:57:41,719 StorageService.java:646 - Token metadata: Normal Tokens:
INFO  [main] 2020-06-04 14:57:41,844 QueryProcessor.java:163 - Preloaded 0 prepared statements
INFO  [main] 2020-06-04 14:57:41,846 StorageService.java:657 - Cassandra version: 3.11.6
INFO  [main] 2020-06-04 14:57:41,846 StorageService.java:658 - Thrift API version: 20.1.0
INFO  [main] 2020-06-04 14:57:41,846 StorageService.java:659 - CQL supported versions: 3.4.4 (default: 3.4.4)
INFO  [main] 2020-06-04 14:57:41,847 StorageService.java:661 - Native protocol supported versions: 3/v3, 4/v4, 5/v5-beta (default: 4/v4)
INFO  [main] 2020-06-04 14:57:41,894 IndexSummaryManager.java:87 - Initializing index summary manager with a memory pool size of 48 MB and a resize interval of 60 minutes
INFO  [main] 2020-06-04 14:57:41,907 MessagingService.java:750 - Starting Messaging Service on /172.24.0.2:7000 (eth0)
INFO  [main] 2020-06-04 14:57:41,947 StorageService.java:589 - Unable to gossip with any peers but continuing anyway since node is in its own seed list
INFO  [main] 2020-06-04 14:57:41,963 StorageService.java:743 - Loading persisted ring state
INFO  [main] 2020-06-04 14:57:41,964 StorageService.java:871 - Starting up server gossip
INFO  [main] 2020-06-04 14:57:42,019 TokenMetadata.java:497 - Updating topology for /172.24.0.2
INFO  [main] 2020-06-04 14:57:42,021 TokenMetadata.java:497 - Updating topology for /172.24.0.2
INFO  [main] 2020-06-04 14:57:42,079 StorageService.java:1038 - Using saved tokens [-103116434733518110, -1048956726713134069, -1164481670525738906, -1509777577867457571, -1529288047900654075, -1625340911234825905, -163118885196205216, -1666406857126911911, -1666532148166742492, -1781400591603052642, -1804608021577009340, -1952099844906840407, -1952146525255244809, -1967283723463737447, -2110205886033303587, -2154815911101813369, -2234848256170997657, -2292621609128040373, -2453253793693736203, -253595969341901364, -2619691185627366392, -2654391712727460815, -2677500664065503239, -2739180351991220065, -2772582044931106195, -2935561066193544946, -2943017271218194837, -2975501814149339699, -2994107318057681270, -3001506412211442866, -3016262674245973135, -3031302169089722822, -3035348897108458816, -3046649129932838731, -3200559769811244640, -3205527083128015075, -3459142575638244171, -3464023026539912006, -3729856880657071205, -3973302274850547613, -3975349253590109549, -4005498571770791858, -4057296430299274689, -4198890977083177, -4210996583599480813, -4235607606923041824, -4263835667954975769, -4329151097388498479, -4417500326515754388, -4803270176994016204, -4806357079630791060, -4892526294310479428, -4904661581317682116, -4922982703881004170, -4985351874494069547, -4991437457062301792, -519440922602775867, -5272191305183528260, -5291279675998473229, -5432392305226596815, -54447141358353074, -5452143533398565966, -5454723760173530383, -5482824550154447206, -5511374159120223567, -5547013241722777696, -5648600272378206780, -5716905827887267025, -5762412061073399223, -5771980991761507205, -5877728891796509681, -5917780388365449643, -5920136931690786407, -5920235713421339395, -6126002871893431585, -6142969631438936117, -6146175203928400645, -6157146023298066843, -6159596494630924657, -6243182081147053309, -6487005356184508567, -6491746420490062134, -6556518871051652247, -6808051142755243376, -6841648582289211519, -6882781776583428203, -6937566777559217667, -698822087983221945, -7051227101941971489, -7150792793698714483, -7251524338625882897, -7262902446336380808, -7498085340912766529, -7573081657994387310, -7667385505001636252, -7703478151506933522, -7716434767383563062, -7753071081582155110, -7761654926480456320, -7803193250664319567, -7861673710797337128, -7961821912880481054, -7968190734134387539, -8049239708904164219, -8099905017693175785, -8112169723085329200, -8145273435038356322, -8151506272481111994, -8242864214346252118, -8250173416815130999, -8272434170905034148, -8364338713413255611, -8433813062076946300, -8524256260386826074, -852793047917314299, -86176174993960944, -8664411308647268882, -8722429166421776128, -8758878423063659471, -8773843698617924603, -8837461651877734670, -8975873469575017325, -9001737776322486288, -9102611499644548251, -9110933509675858318, -944180884564279203, 1033153838073041501, 1097267942896464148, 1193013598325673416, 1284772976847394984, 1320127336571328005, 1378257238614172802, 1422015501900088341, 1542652570763190037, 1570704103201550353, 1593258365989284513, 1734554029740605164, 1764981937919860282, 1791808318474883363, 1859462714919649069, 1998290027384679244, 2144866524064245853, 2155177995668344947, 2190932587520151246, 2257086790475107177, 2257385464347349956, 2381870617274124349, 2397840855774907645, 2399269510510401331, 2433915487311493821, 2504590128898424312, 2605689806265455559, 2610474816332908928, 2623495388945848970, 2735173145791963728, 2941427638520635163, 2970796981977488007, 3014038165768411751, 3035985609431873118, 3075988112337771145, 3102932024446510008, 3212746388243022207, 3311141487510929052, 3315457812142325147, 3338599368204141878, 3502283913697505486, 3552492310341455874, 3604265326402142097, 3679202131987159506, 3691620740306109668, 3702949411028890124, 3730052637853276442, 3945006239944586196, 3953205637391013363, 3993556841901751254, 4067548067508889695, 4183604048826263812, 4283635524549753952, 4419212556139899837, 4464858001666556739, 4482777682814990149, 4487929526037418918, 4508809147463911258, 4745339967000974285, 4798701344157939567, 4854312140184635553, 4915412132452654332, 4933729472089285338, 4961224906093174214, 5017077486649907794, 5038929434998756818, 5221305255947439503, 5257832277597375611, 5296355215051848221, 5353542202830618964, 5355596435910954390, 5381162871330122089, 5501207320196372753, 5560381405816917251, 5637782905183952359, 5723751050885056170, 5751678340633690896, 5804243486141281578, 5807931021029301987, 58504055030052325, 5941833519557251700, 6157232605188408562, 6225538843243971609, 6242084409547769371, 6312860401113548583, 6405641821358187003, 6413733599795408160, 6424111909754483342, 6469522230448126892, 6508511985167266257, 6565920316640122519, 6650737856263949280, 6658111773513259562, 6698718517225390437, 68548111731399249, 6921918798840753839, 6929206370507911655, 6967360449963326965, 7079818226152792212, 7082731990497895116, 7111850088750472142, 7159503226210146458, 7195309720212246741, 7209557517270816728, 7308105619115492685, 7554378603211313217, 7596751173924289612, 7598828588446199130, 7682517070329150364, 7879917327111729204, 8092683237278426674, 819953373549108751, 8247608211390827235, 8320720559951483726, 8382402346684646017, 8504580392038796988, 8642850611960367817, 8645578445312027505, 8678876449429709045, 8688658085766675495, 8742936246973296635, 8870793463070788351, 8899703912542396363, 8946299000152352179, 9058168544544360453, 9166168629418498168, 9205868291605555732, 929816423490035777, 934013651077785986, 938189166868016063, 960797375572478319]
INFO  [main] 2020-06-04 14:57:42,094 StorageService.java:1478 - JOINING: Finish joining ring
INFO  [main] 2020-06-04 14:57:42,173 StorageService.java:2386 - Node /172.24.0.2 state jump to NORMAL
INFO  [main] 2020-06-04 14:57:42,182 Gossiper.java:1780 - Waiting for gossip to settle...
INFO  [main] 2020-06-04 14:57:50,192 Gossiper.java:1811 - No gossip backlog; proceeding
INFO  [main] 2020-06-04 14:57:50,425 NativeTransportService.java:68 - Netty using native Epoll event loop
INFO  [main] 2020-06-04 14:57:50,493 Server.java:158 - Using Netty Version: [netty-buffer=netty-buffer-4.0.44.Final.452812a, netty-codec=netty-codec-4.0.44.Final.452812a, netty-codec-haproxy=netty-codec-haproxy-4.0.44.Final.452812a, netty-codec-http=netty-codec-http-4.0.44.Final.452812a, netty-codec-socks=netty-codec-socks-4.0.44.Final.452812a, netty-common=netty-common-4.0.44.Final.452812a, netty-handler=netty-handler-4.0.44.Final.452812a, netty-tcnative=netty-tcnative-1.1.33.Fork26.142ecbb, netty-transport=netty-transport-4.0.44.Final.452812a, netty-transport-native-epoll=netty-transport-native-epoll-4.0.44.Final.452812a, netty-transport-rxtx=netty-transport-rxtx-4.0.44.Final.452812a, netty-transport-sctp=netty-transport-sctp-4.0.44.Final.452812a, netty-transport-udt=netty-transport-udt-4.0.44.Final.452812a]
INFO  [main] 2020-06-04 14:57:50,494 Server.java:159 - Starting listening for CQL clients on /0.0.0.0:9042 (unencrypted)...
INFO  [main] 2020-06-04 14:57:50,539 CassandraDaemon.java:556 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it

Model

import user-review-product.xmi
import Reviews.tdl
import Stuff.tdl
import Inventory.tdl
import dbTypes.tdl
containertype Docker
clustertype DockerCompose
platformtype localhost
platform platformName : localhost {
    cluster clusterName : DockerCompose {
        application Polystore {
            container reviews : Docker {
                deploys Reviews
                ports {
                    target = 27017 ;
                }
            }
            container stuff : Docker {
                deploys Stuff
                ports {
                    target = 7001 ;
                }
            }
            container inventory : Docker {
                deploys Inventory
                ports {
                    target = 3306 ;
                }
            }
        }
    }
}

Inventory.tdl:

database Inventory : MariaDB {
    environment {
        MYSQL_ROOT_PASSWORD = example ;
    }
}

Reviews.tdl

database Reviews : Mongo {
    environment {
        MONGO_INITDB_ROOT_USERNAME = username ;
        MONGO_INITDB_ROOT_PASSWORD = password ;
    }
}

Stuff.tdl:

database Stuff : Cassandra {
}

dbTypes.tdl:

dbtype Mongo {
    default image = mongo:latest;
}
dbtype Cassandra {
    default image = cassandra:latest;
}
dbtype MariaDB {
    default image = mariadb:latest;
}

Expected behavior

Not crashing.

Screen shot

Screen Shot 2020-06-04 at 16 48 43

Additional context

I've updated TyphonDL today to the latest version. Then generated the DL stuff from our TyphonML model (https://github.com/typhon-project/typhonql/blob/dev/typhonql/src/lang/typhonql/test/resources/user-review-product/user-review-product.xmi). And used the deploy.sh script to start.

MarieSaphira commented 4 years ago
Error 137 in Docker denotes that the container was ‘KILL’ed by ‘oom-killer’ (Out of Memory). This happens when there isn’t enough memory in the container for running the process. Cassandra is using (resp. reserving) a lot of resources. I did `docker ps -q xargs docker stats --no-stream` and it reserves more than 4GiB of memory CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
9d6582c91414 test_Stuff_1 0.78% 4.127GiB / 15.55GiB 26.54% 3.94kB / 1.78kB 25.3MB / 2.56MB 50

To change this add the following environment to your cassandra container:

    environment:
      MAX_HEAP_SIZE: 2G
      HEAP_NEWSIZE: 400M

This will reduce the amount of memory reserved, of course you can change the values but I don't know what the minimum values are for Cassandra to still work properly.

The environment will be included in the next version, for now please just add it to your Cassandra container in databases.yaml

tvdstorm commented 4 years ago

Thanks, I'll try it out.

tvdstorm commented 4 years ago

Ok, no observable change. Doing your command above gives me this:

CONTAINER ID        NAME                                             CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
380376edd748        user-review-product_polystore-ui_1               0.00%               328.3MiB / 1.943GiB   16.50%              1.02kB / 0B         0B / 0B             18
816c8e420dbb        user-review-product_typhon-polystore-service_1   0.44%               212.2MiB / 1.943GiB   10.66%              77.5kB / 16.8kB     0B / 0B             39
3faed7fce5f3        user-review-product_Inventory_1                  0.15%               11.65MiB / 1.943GiB   0.59%               1.11kB / 0B         0B / 0B             30
72b642b89915        user-review-product_Reviews_1                    0.60%               10.21MiB / 1.943GiB   0.51%               5.98kB / 3.76kB     0B / 0B             34
70f7d5a0c92d        user-review-product_polystore-mongo_1            0.58%               12MiB / 1.943GiB      0.60%               18kB / 76.5kB       0B / 0B             34
335a632c0612        user-review-product_MoreStuff_1                  1.79%               159.5MiB / 1.943GiB   8.02%               1.29kB / 0B         0B / 0B             47
eef23746ba39        user-review-product_typhonql-server_1            0.22%               55.57MiB / 1.943GiB   2.79%               1.11kB / 0B         0B / 0B             25

Note the limits around 2Gb, could that be the problem?

MarieSaphira commented 4 years ago

It looks like you only have 1.9GiB of memory available, so setting Cassandra to use 2GiB will still produce the out-of-memory exception. Try setting it to 800M or something like that.

MarieSaphira commented 4 years ago

To assign more memory to your docker VM see here

tvdstorm commented 4 years ago

This did the trick. Thanks.