Describe the bug
None of following metrics go up when Alluxio FUSE can successfully read data.
"Cluster.BytesReadDirectThroughput": {
"value": 0
},
"Cluster.BytesReadDomainThroughput": {
"value": 0
},
"Cluster.BytesReadLocalThroughput": {
"value": 0
},
"Cluster.BytesReadRemoteThroughput": {
"value": 0
},
"Cluster.BytesReadUfsThroughput": {
"value": 161781413 # after all data are loaded into alluxio, this number doesn't go up anymore
},
To Reproduce
Deploy alluxio on K8s (yamls are shown below)
Do distributedLoad with replications so that each worker has a replication of target file
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in two weeks if no further activity occurs. Thank you for your contributions.
Alluxio Version: 2.6.0-RC2
Describe the bug None of following metrics go up when Alluxio FUSE can successfully read data. "Cluster.BytesReadDirectThroughput": { "value": 0 }, "Cluster.BytesReadDomainThroughput": { "value": 0 }, "Cluster.BytesReadLocalThroughput": { "value": 0 }, "Cluster.BytesReadRemoteThroughput": { "value": 0 }, "Cluster.BytesReadUfsThroughput": { "value": 161781413 # after all data are loaded into alluxio, this number doesn't go up anymore },
To Reproduce
Expected behavior At least ONE of bytesRead* metrics go up after read is done.
Urgency Urgent because without correct metrics, we are shooting in the dark.
Additional context
Source: alluxio/templates/config/alluxio-conf.yaml
#
The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
(the "License"). You may not use this work except in compliance with the License, which is
available at www.apache.org/licenses/LICENSE-2.0
#
This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
either express or implied, as more fully set forth in the License.
#
See the NOTICE file distributed with this work for information regarding copyright ownership.
#
------alluxio-configmap.yaml------ apiVersion: v1 kind: ConfigMap metadata: annotations: "helm.sh/hook": "pre-install" "helm.sh/hook-delete-policy": before-hook-creation name: alluxio-config labels: name: alluxio-config app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm data: ALLUXIO_JAVA_OPTS: |- -Dalluxio.license.file=/secrets/alluxio-license/license.json -Dalluxio.hub.agent.rpc.hostname=${ALLUXIO_HUB_AGENT_RPC_HOSTNAME} -Dalluxio.master.hostname=alluxio-master-0 -Dalluxio.master.journal.type=UFS -Dalluxio.master.journal.folder=/journal -Dalluxio.fuse.logging.threshold=1000ms -Dalluxio.hub.manager.rpc.hostname=alluxio-master-0 -Dalluxio.hub.manager.web.login.password=alluxio -Dalluxio.hub.manager.web.login.username=alluxio -Dalluxio.security.stale.channel.purge.interval=365d -Dalluxio.user.block.master.client.pool.gc.threshold=1h -Dalluxio.user.block.master.client.pool.size.max=1024 -Dalluxio.user.block.read.metrics.enabled=false -Dalluxio.user.block.worker.client.pool.max=10240 -Dalluxio.user.file.master.client.pool.size.max=1024 -Dalluxio.user.file.passive.cache.enabled=false -Dalluxio.user.metadata.cache.enabled=true -Dalluxio.user.metadata.cache.expiration.time=2h -Dalluxio.user.metadata.cache.max.size=20000 -Dalluxio.user.metrics.collection.enabled=true -Dalluxio.user.short.circuit.enabled=true -Dalluxio.user.update.file.accesstime.disabled=true -Dalluxio.worker.block.master.client.pool.size=1024 -Dalluxio.worker.data.server.domain.socket.as.uuid=false -Dalluxio.worker.fuse.enabled=true -Dalluxio.worker.fuse.mount.alluxio.path=/ -Dalluxio.worker.fuse.mount.options=allow_other,kernel_cache,max_read=131072,attr_timeout=7200,entry_timeout=7200 -Dalluxio.worker.fuse.mount.point=/tmp/alluxio-fuse -Dalluxio.worker.network.reader.buffer.size=32MB -Dalluxio.worker.tieredstore.block.locks=100000 ALLUXIO_MASTER_JAVA_OPTS: |- -Dalluxio.master.hostname=${ALLUXIO_MASTER_HOSTNAME} -Xms32G -Xmx32G -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 ALLUXIO_JOB_MASTER_JAVA_OPTS: |- -Dalluxio.master.hostname=${ALLUXIO_MASTER_HOSTNAME} -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 ALLUXIO_WORKER_JAVA_OPTS: |- -Dalluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME} -Dalluxio.worker.rpc.port=29999 -Dalluxio.worker.web.port=30000 -Dalluxio.worker.secure.rpc.port=29997 -Dalluxio.worker.container.hostname=${ALLUXIO_WORKER_CONTAINER_HOSTNAME} -Dalluxio.worker.ramdisk.size=6G -Dalluxio.worker.tieredstore.levels=1 -Dalluxio.worker.tieredstore.level0.alias=MEM -Dalluxio.worker.tieredstore.level0.dirs.mediumtype=MEM -Dalluxio.worker.tieredstore.level0.dirs.path=/dev/shm -Dalluxio.worker.tieredstore.level0.dirs.quota=8G -Dalluxio.worker.tieredstore.level0.watermark.high.ratio=0.95 -Dalluxio.worker.tieredstore.level0.watermark.low.ratio=0.7 -Xmx20G -Xms20G -XX:MaxDirectMemorySize=18g -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 ALLUXIO_JOB_WORKER_JAVA_OPTS: |- -Dalluxio.worker.hostname=${ALLUXIO_WORKER_HOSTNAME} -Dalluxio.job.worker.rpc.port=30001 -Dalluxio.job.worker.data.port=30002 -Dalluxio.job.worker.web.port=30003 -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 ALLUXIO_FUSE_JAVA_OPTS: |- -Dalluxio.user.hostname=${ALLUXIO_CLIENT_HOSTNAME} -XX:MaxDirectMemorySize=2g ALLUXIO_WORKER_TIEREDSTORE_LEVEL0_DIRS_PATH: /dev/shm
------alluxio-master-service.yaml------
Source: alluxio/templates/master/service.yaml
#
The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
(the "License"). You may not use this work except in compliance with the License, which is
available at www.apache.org/licenses/LICENSE-2.0
#
This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
either express or implied, as more fully set forth in the License.
#
See the NOTICE file distributed with this work for information regarding copyright ownership.
#
apiVersion: v1 kind: Service metadata: name: alluxio-master-0 labels: app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm role: alluxio-master spec: ports:
port: 19998 name: rpc
port: 19999 name: web
port: 20001 name: job-rpc
port: 20002 name: job-web
port: 19200 name: embedded
port: 20003 name: job-embedded
port: 30075 name: hub-agent
port: 30077 name: hub-web
port: 30076 name: hub-rpc
clusterIP: None selector: role: alluxio-master app: alluxio release: alluxio statefulset.kubernetes.io/pod-name: alluxio-master-0
------alluxio-master-statefulset.yaml------
Source: alluxio/templates/master/statefulset.yaml
#
The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
(the "License"). You may not use this work except in compliance with the License, which is
available at www.apache.org/licenses/LICENSE-2.0
#
This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
either express or implied, as more fully set forth in the License.
#
See the NOTICE file distributed with this work for information regarding copyright ownership.
#
apiVersion: apps/v1 kind: StatefulSet metadata: name: alluxio-master labels: name: alluxio-master app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm role: alluxio-master spec: selector: matchLabels: app: alluxio role: alluxio-master name: alluxio-master serviceName: alluxio-master replicas: 1 template: metadata: labels: name: alluxio-master app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm role: alluxio-master spec: hostNetwork: false dnsPolicy: ClusterFirst nodeSelector: securityContext: fsGroup: 0 initContainers:
name: journal-format image: alluxio/alluxio:2.6.0-RC2 imagePullPolicy: IfNotPresent securityContext: runAsUser: 0 command: ["alluxio","formatJournal"] volumeMounts:
name: alluxio-journal mountPath: /journal
containers:
- --no-format
env:
------alluxio-worker-daemonset.yaml------
Source: alluxio/templates/worker/daemonset.yaml
#
The Alluxio Open Foundation licenses this work under the Apache License, version 2.0
(the "License"). You may not use this work except in compliance with the License, which is
available at www.apache.org/licenses/LICENSE-2.0
#
This software is distributed on an "AS IS" basis, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
either express or implied, as more fully set forth in the License.
#
See the NOTICE file distributed with this work for information regarding copyright ownership.
#
apiVersion: apps/v1 kind: DaemonSet metadata: name: alluxio-worker labels: app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm role: alluxio-worker spec: selector: matchLabels: app: alluxio release: alluxio role: alluxio-worker template: metadata: labels: app: alluxio chart: alluxio-0.6.13 release: alluxio heritage: Helm role: alluxio-worker spec: hostNetwork: false hostPID: false dnsPolicy: ClusterFirst securityContext: fsGroup: 0 nodeSelector: