deenav
commented
5 years ago Sharing the cluster logs in internal channel due to size constraint.
Closed deenav closed 5 years ago
deenav
commented
5 years ago Sharing the cluster logs in internal channel due to size constraint.
fpj
commented
5 years ago Please share more detail about the deployment, like:
fpj
commented
5 years ago Is this the workload that you are running:
mid-scale: |
{
"type": "Pravega",
"name": "longevity/midScale",
"forever": true,
"scope": "longevity",
"stream": "midScale",
"payload": {
"numberOfKeys": 512,
"dynamicKeyProbability": true
},
"createStream": true,
"throttle": {
"maxEventsPerSecond": 4000,
"maxOutstandingAcks": 4000,
"dynamicThrottlePeriod": 180
},
"streamPolicies": {
"scalingType": "EVENTS_PER_SECOND",
"targetRate": 2000,
"scaleFactor": 2,
"minNumSegments": 8,
"retentionType": "LIMITED_TIME_MILLIS",
"retentionLimit": 3600000
},
"tasks": [
{
"numReaders": 8,
"numWriters": 4,
"duplicates": 4
}
],
"longevityAssertions": {
"isRunningState": null,
"hasAtLeastXActiveReaders": 32,
"hasAtLeastXActiveWriters": 16,
"readerBytesAreIncreasing": null,
"writerBytesAreIncreasing": null,
"eventsInSequence": null
}
}@fpj Yes, I was using the same workload you posted above. Please see below other inlines.
What's the size of the BK volumes that you are using?
BK volumes (ledgers & journals) size are 100GB, this is the same manifest we are using for all the test runs including #2284 build where no problem was there.
/dev/sdg 99G 12G 87G 13% /bk/journal
/dev/sde 99G 94G 4.9G 96% /bk/ledgers
/dev/sdf 9.8G 40M 9.7G 1% /bk/indexAre Tier 2 writes happening at all? If so, at what rate?
Yes, Tier2 writes were happening but I am not sure about the rate.
Is Tier 2 full or shared?
Tier2 is based on Isilon backend and shared across other environments but it is not full (see below)
[deena@master pravega-k8s-logs]$ kubectl exec -it pravega-pravega-segmentstore-0 /bin/sh
/opt/pravega #
/opt/pravega # df -h
Filesystem Size Used Available Use% Mounted on
overlay 49.1G 3.3G 43.2G 7% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 15.7G 0 15.7G 0% /sys/fs/cgroup
10.249.249.131:/ifs/pravega-test/pks/default-pravega-tier2-pvc-8ffb99f8-9e3b-11e9-b8e4-005056ba9070
194.7T 34.2T 156.7T 18% /mnt/tier2
RaulGracia
commented
5 years ago The main problem described in this issue is related to a mismatch among:
I have executed a run of 0.5 with a more aggressive Bookkeeper data compaction parameters:
minorCompactionThreshold: "0.4"
minorCompactionInterval: "1800"
majorCompactionThreshold: "0.8"
majorCompactionInterval: "43200"
isForceGCAllowWhenNoSpace: "true"
diskUsageThreshold: "0.90"
diskUsageWarnThreshold: "0.85"
diskUsageLwmThreshold: "0.85"Apart from that, I have also used correct prefix for RocksDB params:
rocksdb.writeBufferSizeMB: "512"
rocksdb.readCacheSizeMB: "2048"After +11 hours of execution of mid-scale longevity workload, Bookies have been able to keep the storage of their volumes under reasonable levels:
kubectl exec -it pravega-bookie-0 -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb 9.8G 60M 9.8G 1% /bk/index
/dev/sda1 95G 21G 74G 23% /etc/hosts
/dev/sdd 98G 65G 34G 67% /bk/ledgers
/dev/sdc 98G 13G 85G 14% /bk/journalkubectl exec -it pravega-bookie-1 -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdd 9.8G 65M 9.8G 1% /bk/index
/dev/sda1 95G 25G 70G 26% /etc/hosts
/dev/sdb 98G 26G 73G 26% /bk/ledgers
/dev/sdc 98G 13G 86G 13% /bk/journalkubectl exec -it pravega-bookie-2 -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 8.7G 86G 10% /etc/hosts
/dev/sde 98G 24G 75G 24% /bk/ledgers
/dev/sdc 98G 12G 87G 12% /bk/journal
/dev/sdd 9.8G 65M 9.8G 1% /bk/indexkubectl exec -it pravega-bookie-3 -- df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdd 98G 38G 61G 39% /bk/ledgers
/dev/sdc 98G 13G 86G 13% /bk/journal
/dev/sde 9.8G 67M 9.7G 1% /bk/indexPlease, incorporate the parameter changes in your Pravega manifests to avoid this issue from appearing again.
Problem description While running longevity test (4 worker pods with 4 writer & 8 reader each) IO with autoscaling enabled based on "EVENTS_PER_SECOND" on Pravega:
0.5.0-2291.3ccff63, Observing readers started failing within30 minsof run and eventually all readers are stalled after~10 hrsof run.NOTE :- This test was running fine with Pravega:
0.5.0-2284.8c6176build for more than 5 days consistently. But with0.5.0-2291.3ccff63build multiple times (more than 3 times) same test failed after running for 1 to ~10 hours with similar error.From the logs, I see SegmentStore-1 restarted due to
java.lang.OutOfMemoryError: Java heap spaceerror and following that couple of Bookie pods are moved to not ready (0/1) state due toERROR: Ledger directory /bk/ledgers/current is out-of-space.Observing that Tier2 was accepting data but seems like tier1 data accumulation is very fast causing Bookies out of ledger space.
SS-1 log snip
Bookkeeper-0 log snip:
Bookie-3 log snip
POD status