Closed takersk closed 2 years ago
ssz1997
commented
2 years ago @takersk Saw your comment at https://github.com/Alluxio/alluxio/issues/15235#issuecomment-1138044215. To confirm, were you using a non-root Alluxio path for mounting? Did you try use the Alluxio root path?
takersk
commented
2 years ago @ssz1997 thank you for confirmation. I am using Alluxio root path .
properties
alluxio.master.mount.table.root.ufs: "hdfs://cdp/cdpdev/prod/"
pv/pvc
apiVersion: v1
kind: PersistentVolume
metadata:
name: alluxio-airflow-pv
namespace: airflow
labels:
name: alluxio-airflow-pv
annotations:
pv.kubernetes.io/bound-by-controller: 'yes'
spec:
capacity:
storage: 1000Gi
csi:
driver: alluxio
volumeHandle: alluxio
volumeAttributes:
alluxioPath: /
javaOptions: '-Dalluxio.user.metadata.cache.enabled=true'
accessModes:
- ReadWriteMany
persistentVolumeReclaimPolicy: Retain
mountOptions:
- allow_other
- entry_timeout=36000
- attr_timeout=36000
- max_readahead=0
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alluxio-airflow-pvc
namespace: airflow
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteMany
selector:
matchExpressions:
- key: name
operator: In
values:
- alluxio-airflow-pv
resources:
requests:
storage: 1000Gi
volumeName: alluxio-airflow-pv
storageClassName: ''
volumeMode: Filesystemairflow
'volume_devices': None,
'volume_mounts': [{'mount_path': '/tmp/alluxio',
'mount_propagation': None,
'name': 'data-volume',
'read_only': None,
'sub_path': None,
'sub_path_expr': None},...
'persistent_volume_claim': {'claim_name': 'alluxio-airflow-pvc', 'read_only': None},
...
'state': {'running': None, 'terminated': {'container_id': 'docker://671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43', 'exit_code': 128, 'finished_at': datetime.datetime(2022, 5, 25, 21, 31, 16, tzinfo=tzlocal()), 'message': 'error ' 'while ' 'creating ' 'mount ' 'source ' 'path ' "'/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': " 'mkdir ' '/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: ' 'file ' 'exists', 'reason': 'ContainerCannotRun', 'signal': None, 'started_at': datetime.datetime(2022, 5, 25, 21, 31, 16, tzinfo=tzlocal())}, 'waiting': None}}],
* kubelet logsroot@dkosv3-cdp-ml-prod-worker-8:/var/log# cat kubelet.ERROR | grep cf0d1c24-a325-433b-b371-464e63181b95 E0526 06:31:17.236673 4536 remote_runtime.go:222] StartContainer "671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43" from runtime service failed: rpc error: code = Unknown desc = failed to start container "671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists E0526 06:31:17.236746 4536 kuberuntime_manager.go:809] container start failed: RunContainerError: failed to start container "671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists E0526 06:31:17.236792 4536 pod_workers.go:191] Error syncing pod cf0d1c24-a325-433b-b371-464e63181b95 ("etl-gift-stats-also-buy-product.19ad0eec871c41a39f3336791c166347_airflow(cf0d1c24-a325-433b-b371-464e63181b95)"), skipping: failed to "StartContainer" for "base" with RunContainerError: "failed to start container \"671240747c7f34a0718dc7b097dafa4f58fdfae6de7af1306626cb1b01e4fc43\": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/cf0d1c24-a325-433b-b371-464e63181b95/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists"
@takersk I'll look into this. In the mean time, can you try change the accessMode in your pv and pvc to ReadWriteOnce and try again? Thanks!
takersk
commented
2 years ago @ssz1997
I changed accessMode to ReadWriteOnce but, Unfortunately I am getting the same error.
pv/pvc (change accessMode)
apiVersion: v1
kind: PersistentVolume
metadata:
name: alluxio-airflow-pv
namespace: airflow
labels:
name: alluxio-airflow-pv
annotations:
pv.kubernetes.io/bound-by-controller: 'yes'
spec:
capacity:
storage: 1000Gi
csi:
driver: alluxio
volumeHandle: alluxio
volumeAttributes:
alluxioPath: /
javaOptions: '-Dalluxio.user.metadata.cache.enabled=true'
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
mountOptions:
- allow_other
- entry_timeout=36000
- attr_timeout=36000
- max_readahead=0
volumeMode: Filesystem
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alluxio-airflow-pvc
namespace: airflow
annotations:
pv.kubernetes.io/bind-completed: 'yes'
pv.kubernetes.io/bound-by-controller: 'yes'
finalizers:
- kubernetes.io/pvc-protection
spec:
accessModes:
- ReadWriteOnce
selector:
matchExpressions:
- key: name
operator: In
values:
- alluxio-airflow-pv
resources:
requests:
storage: 1000Gi
volumeName: alluxio-airflow-pv
storageClassName: ''
volumeMode: Filesystemlogs
[2022-05-30 03:00:45,451] {kubernetes_pod.py:366} INFO - creating pod with labels {'dag_id': 'etl_processing_dag', 'task_id': 'etl_gift_meta_extract', 'execution_date': '2022-05-28T2100000000-a54f30553', 'try_number': '8'} and launcher <airflow.providers.cncf.kubernetes.utils.pod_launcher.PodLauncher object at 0x7f8b186916d8>
[2022-05-30 03:00:45,488] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Pending
[2022-05-30 03:00:45,488] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc
[2022-05-30 03:00:46,497] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Pending
[2022-05-30 03:00:46,498] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc
[2022-05-30 03:00:47,505] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Pending
[2022-05-30 03:00:47,505] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc
[2022-05-30 03:00:48,513] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Pending
[2022-05-30 03:00:48,513] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc
[2022-05-30 03:00:49,522] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Pending
[2022-05-30 03:00:49,522] {pod_launcher.py:128} WARNING - Pod not yet started: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc
[2022-05-30 03:00:50,530] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Failed
[2022-05-30 03:00:50,530] {pod_launcher.py:308} ERROR - Event with job id etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc Failed
[2022-05-30 03:00:50,547] {pod_launcher.py:193} ERROR - Error parsing timestamp. Will continue execution but won't update timestamp
[2022-05-30 03:00:50,548] {pod_launcher.py:149} INFO - failed to try resolving symlinks in path "/var/log/pods/airflow_etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc_3cd692fa-1202-43ad-a555-07c43b4c92f6/base/0.log": lstat /var/log/pods/airflow_etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc_3cd692fa-1202-43ad-a555-07c43b4c92f6/base/0.log: no such file or directory
[2022-05-30 03:00:51,564] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Failed
[2022-05-30 03:00:51,564] {pod_launcher.py:308} ERROR - Event with job id etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc Failed
[2022-05-30 03:00:51,572] {pod_launcher.py:198} INFO - Event: etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc had an event of type Failed
[2022-05-30 03:00:51,572] {pod_launcher.py:308} ERROR - Event with job id etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc Failed
[2022-05-30 03:00:51,603] {taskinstance.py:1463} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 369, in execute
raise AirflowException(f'Pod {self.pod.metadata.name} returned a failure: {remote_pod}')
airflow.exceptions.AirflowException: Pod etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc returned a failure: {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal()),
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': {'airflow_version': '2.1.4',
'dag_id': 'etl_processing_dag',
'execution_date': '2022-05-28T2100000000-a54f30553',
'kubernetes_pod_operator': 'True',
'task_id': 'etl_gift_meta_extract',
'try_number': '8'},
'managed_fields': None,
'name': 'etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc',
'namespace': 'airflow',
'owner_references': None,
'resource_version': '169019623',
'self_link': '/api/v1/namespaces/airflow/pods/etl-gift-meta-extract.ecc4ec9418784776b4114ad9aa1b03cc',
'uid': '3cd692fa-1202-43ad-a555-07c43b4c92f6'},
'spec': {'active_deadline_seconds': None,
'affinity': {'node_affinity': {'preferred_during_scheduling_ignored_during_execution': None,
'required_during_scheduling_ignored_during_execution': {'node_selector_terms': [{'match_expressions': [{'key': 'node-type',
'operator': 'In',
'values': ['app']}],
'match_fields': None}]}},
'pod_affinity': None,
'pod_anti_affinity': None},
'automount_service_account_token': None,
'containers': [{'args': ['/app/src/app/gift/meta/gift-meta-extract.py',
'2022-05-29'],
'command': ['sh', '/app/bin/deploy.sh'],
'env': [{'name': 'PHASE',
'value': 'stage',
'value_from': None}],
'env_from': None,
'image': 'idock.daumkakao.io/kcai/kcai-pyspark:0.0.8',
'image_pull_policy': 'Always',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': None,
'readiness_probe': None,
'resources': {'limits': {'cpu': '5',
'memory': '10G',
'nvidia.com/gpu': '0'},
'requests': {'cpu': '2',
'memory': '5G',
'nvidia.com/gpu': '0'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': '/dev/termination-log',
'termination_message_policy': 'File',
'tty': None,
'volume_devices': None,
'volume_mounts': [{'mount_path': '/tmp/alluxio',
'mount_propagation': None,
'name': 'data-volume',
'read_only': None,
'sub_path': None,
'sub_path_expr': None},
{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'mount_propagation': None,
'name': 'airflow-worker-token-vcwnb',
'read_only': True,
'sub_path': None,
'sub_path_expr': None}],
'working_dir': None}],
'dns_config': None,
'dns_policy': 'ClusterFirst',
'enable_service_links': True,
'host_aliases': None,
'host_ipc': None,
'host_network': None,
'host_pid': None,
'hostname': None,
'image_pull_secrets': None,
'init_containers': None,
'node_name': 'dkosv3-cdp-ml-stage-worker-8nzx',
'node_selector': None,
'preemption_policy': None,
'priority': 0,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': 'Never',
'runtime_class_name': None,
'scheduler_name': 'default-scheduler',
'security_context': {'fs_group': None,
'run_as_group': None,
'run_as_non_root': None,
'run_as_user': None,
'se_linux_options': None,
'supplemental_groups': None,
'sysctls': None,
'windows_options': None},
'service_account': 'airflow-worker',
'service_account_name': 'airflow-worker',
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': 30,
'tolerations': [{'effect': 'NoExecute',
'key': 'node.kubernetes.io/not-ready',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None},
{'effect': 'NoExecute',
'key': 'node.kubernetes.io/unreachable',
'operator': 'Exists',
'toleration_seconds': 300,
'value': None}],
'volumes': [{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'data-volume',
'nfs': None,
'persistent_volume_claim': {'claim_name': 'alluxio-airflow-pvc',
'read_only': None},
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None},
{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'csi': None,
'downward_api': None,
'empty_dir': None,
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'airflow-worker-token-vcwnb',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': {'default_mode': 420,
'items': None,
'optional': None,
'secret_name': 'airflow-worker-token-vcwnb'},
'storageos': None,
'vsphere_volume': None}]},
'status': {'conditions': [{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'Initialized'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'Ready'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal()),
'message': 'containers with unready status: [base]',
'reason': 'ContainersNotReady',
'status': 'False',
'type': 'ContainersReady'},
{'last_probe_time': None,
'last_transition_time': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal()),
'message': None,
'reason': None,
'status': 'True',
'type': 'PodScheduled'}],
'container_statuses': [{'container_id': 'docker://d34dbc4b867be7fbc562cb43193994ab65884e6f0be2332ad466c0bae15a2cd4',
'image': 'idock.daumkakao.io/kcai/kcai-pyspark:0.0.8',
'image_id': 'docker-pullable://idock.daumkakao.io/kcai/kcai-pyspark@sha256:2da01ed02034362eac99445a8780de4fe64d14cf6ada4f524274110612cf582b',
'last_state': {'running': None,
'terminated': None,
'waiting': None},
'name': 'base',
'ready': False,
'restart_count': 0,
'state': {'running': None,
'terminated': {'container_id': 'docker://d34dbc4b867be7fbc562cb43193994ab65884e6f0be2332ad466c0bae15a2cd4',
'exit_code': 128,
'finished_at': datetime.datetime(2022, 5, 30, 3, 0, 49, tzinfo=tzlocal()),
'message': 'error '
'while '
'creating '
'mount '
'source '
'path '
"'/var/lib/kubelet/pods/3cd692fa-1202-43ad-a555-07c43b4c92f6/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': "
'mkdir '
'/var/lib/kubelet/pods/3cd692fa-1202-43ad-a555-07c43b4c92f6/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: '
'file '
'exists',
'reason': 'ContainerCannotRun',
'signal': None,
'started_at': datetime.datetime(2022, 5, 30, 3, 0, 49, tzinfo=tzlocal())},
'waiting': None}}],
'host_ip': '10.92.175.31',
'init_container_statuses': None,
'message': None,
'nominated_node_name': None,
'phase': 'Failed',
'pod_ip': '10.240.15.211',
'qos_class': 'Burstable',
'reason': None,
'start_time': datetime.datetime(2022, 5, 30, 3, 0, 45, tzinfo=tzlocal())}}csi nodeplugin bash
bash-5.0# cd /var/lib/kubelet/pods/3cd692fa-1202-43ad-a555-07c43b4c92f6
bash: cd: /var/lib/kubelet/pods/3cd692fa-1202-43ad-a555-07c43b4c92f6: No such file or directory@takersk Can you help check:
/opt/alluxio/fuse.log and /opt/alluxio/fuse.out?
Thanks!takersk
commented
2 years ago @ssz1997 hi!! I'll answer your question~~ thanks!!
What version of Kubernetes are you using? k8s version : v1.17.12
In csi-nodeserver container in csi nodeplugin pod, can you upload the log in /opt/alluxio/fuse.log and /opt/alluxio/fuse.out?
2022-05-31 08:16:45,553 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 08:16:45,911 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse4451883667961280789.so.
2022-05-31 08:16:45,911 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 08:16:45,992 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 08:16:46,000 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-5934370064752418343. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 08:16:46,168 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 08:16:46,679 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 08:16:46,978 INFO Reflections - Reflections took 199 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 08:16:46,997 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/92630f07-4562-4691-9400-966e2978869b/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 08:16:46,997 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/92630f07-4562-4691-9400-966e2978869b/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 08:16:57,741 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 08:16:57,741 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/92630f07-4562-4691-9400-966e2978869b/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 08:16:57,743 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/92630f07-4562-4691-9400-966e2978869b/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 08:40:20,195 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 08:40:20,550 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse2293778644277798178.so.
2022-05-31 08:40:20,551 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 08:40:20,632 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 08:40:20,641 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-5829953303781093297. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 08:40:20,809 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 08:40:21,328 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 08:40:21,602 INFO Reflections - Reflections took 248 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 08:40:21,620 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/d780b6e8-8ff9-48bc-a7d9-b31ed4a1acf4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 08:40:21,620 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/d780b6e8-8ff9-48bc-a7d9-b31ed4a1acf4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 08:40:24,599 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 08:40:24,599 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/d780b6e8-8ff9-48bc-a7d9-b31ed4a1acf4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 08:40:24,602 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/d780b6e8-8ff9-48bc-a7d9-b31ed4a1acf4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 10:15:44,055 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 10:15:44,409 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse3578543681998596587.so.
2022-05-31 10:15:44,409 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 10:15:44,489 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 10:15:44,497 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-8110655119959181384. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 10:15:44,664 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 10:15:45,183 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 10:15:45,468 INFO Reflections - Reflections took 187 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 10:15:45,487 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/aa1e3991-cafe-4e8a-a771-a145954f8805/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 10:15:45,487 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/aa1e3991-cafe-4e8a-a771-a145954f8805/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 10:15:48,157 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 10:15:48,157 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/aa1e3991-cafe-4e8a-a771-a145954f8805/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 10:15:48,160 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/aa1e3991-cafe-4e8a-a771-a145954f8805/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 11:15:20,665 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 11:15:21,024 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse5473583003759642326.so.
2022-05-31 11:15:21,024 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 11:15:21,105 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 11:15:21,113 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-3941587019658893285. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 11:15:21,280 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 11:15:21,811 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 11:15:22,089 INFO Reflections - Reflections took 183 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 11:15:22,108 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/1d07b83d-e7e6-4b56-8b69-a32797eece64/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 11:15:22,108 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/1d07b83d-e7e6-4b56-8b69-a32797eece64/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 11:15:28,083 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 11:15:28,084 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/1d07b83d-e7e6-4b56-8b69-a32797eece64/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 11:15:28,087 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/1d07b83d-e7e6-4b56-8b69-a32797eece64/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 21:02:12,605 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 21:02:12,961 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse742358552558683138.so.
2022-05-31 21:02:12,961 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 21:02:13,043 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 21:02:13,052 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-5165555452803147494. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 21:02:13,223 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 21:02:13,747 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 21:02:14,034 INFO Reflections - Reflections took 262 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 21:02:14,053 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/d724e427-46fc-4c77-901f-cac136906a61/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 21:02:14,054 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/d724e427-46fc-4c77-901f-cac136906a61/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 21:02:16,639 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 21:02:16,639 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/d724e427-46fc-4c77-901f-cac136906a61/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 21:02:16,642 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/d724e427-46fc-4c77-901f-cac136906a61/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-05-31 21:03:30,418 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-05-31 21:03:30,783 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse928728738809412849.so.
2022-05-31 21:03:30,783 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-05-31 21:03:30,864 INFO MetricsSystem - Starting sinks with config: {}.
2022-05-31 21:03:30,872 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-7022630693580772478. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-05-31 21:03:31,042 INFO NettyUtils - EPOLL_MODE is available
2022-05-31 21:03:31,583 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-05-31 21:03:31,868 INFO Reflections - Reflections took 261 ms to scan 1 urls, producing 58 keys and 199 values
2022-05-31 21:03:31,887 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/a6240bec-8233-43f0-85f1-a53ab03e9218/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 21:03:31,887 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/a6240bec-8233-43f0-85f1-a53ab03e9218/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-05-31 21:03:42,009 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-05-31 21:03:42,009 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/a6240bec-8233-43f0-85f1-a53ab03e9218/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-05-31 21:03:42,011 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/a6240bec-8233-43f0-85f1-a53ab03e9218/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-01 21:02:15,511 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-01 21:02:15,865 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse559833246198398079.so.
2022-06-01 21:02:15,865 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-01 21:02:15,947 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-01 21:02:15,955 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-1402868818922188779. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-01 21:02:16,123 INFO NettyUtils - EPOLL_MODE is available
2022-06-01 21:02:16,637 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-06-01 21:02:16,920 INFO Reflections - Reflections took 182 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-01 21:02:16,939 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/74351f20-d301-4ca9-86b8-6da8271f3806/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:02:16,939 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/74351f20-d301-4ca9-86b8-6da8271f3806/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:02:19,546 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-01 21:02:19,546 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/74351f20-d301-4ca9-86b8-6da8271f3806/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-01 21:02:19,549 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/74351f20-d301-4ca9-86b8-6da8271f3806/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-01 21:03:32,721 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-01 21:03:33,115 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse8964700075927719462.so.
2022-06-01 21:03:33,115 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-01 21:03:33,212 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-01 21:03:33,220 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-2409094487747906557. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-01 21:03:33,389 INFO NettyUtils - EPOLL_MODE is available
2022-06-01 21:03:33,908 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-06-01 21:03:34,199 INFO Reflections - Reflections took 196 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-01 21:03:34,217 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/53c12925-4343-4e84-9f82-f7f55fcb9917/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:03:34,217 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/53c12925-4343-4e84-9f82-f7f55fcb9917/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:03:44,108 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-01 21:03:44,108 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/53c12925-4343-4e84-9f82-f7f55fcb9917/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-01 21:03:44,111 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/53c12925-4343-4e84-9f82-f7f55fcb9917/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-01 21:29:00,586 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-01 21:29:00,939 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse302573837137524508.so.
2022-06-01 21:29:00,939 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-01 21:29:01,018 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-01 21:29:01,026 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-5763879677290663494. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-01 21:29:01,192 INFO NettyUtils - EPOLL_MODE is available
2022-06-01 21:29:01,701 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-06-01 21:29:01,983 INFO Reflections - Reflections took 184 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-01 21:29:02,001 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/081dd837-446e-4a96-83d6-97e91cdc76f2/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:29:02,001 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/081dd837-446e-4a96-83d6-97e91cdc76f2/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:29:04,601 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-01 21:29:04,601 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/081dd837-446e-4a96-83d6-97e91cdc76f2/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-01 21:29:04,604 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/081dd837-446e-4a96-83d6-97e91cdc76f2/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-01 21:31:35,603 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-01 21:31:35,958 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse7499809346851275902.so.
2022-06-01 21:31:35,958 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-01 21:31:36,037 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-01 21:31:36,045 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-1661266304049020397. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-01 21:31:36,209 INFO NettyUtils - EPOLL_MODE is available
2022-06-01 21:31:36,720 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-06-01 21:31:36,999 INFO Reflections - Reflections took 181 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-01 21:31:37,018 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/204d241e-0be8-46ad-9987-e8fd128c17a4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:31:37,018 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/204d241e-0be8-46ad-9987-e8fd128c17a4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 21:31:39,625 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-01 21:31:39,626 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/204d241e-0be8-46ad-9987-e8fd128c17a4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-01 21:31:39,629 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/204d241e-0be8-46ad-9987-e8fd128c17a4/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-01 22:29:10,557 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-01 22:29:10,917 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse7381637947513603576.so.
2022-06-01 22:29:10,917 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-01 22:29:11,000 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-01 22:29:11,009 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-5320928855606848367. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-01 22:29:11,178 INFO NettyUtils - EPOLL_MODE is available
2022-06-01 22:29:11,699 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.140.23, rack=null)
2022-06-01 22:29:11,990 INFO Reflections - Reflections took 195 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-01 22:29:12,008 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/52edcb76-9bd6-4eeb-adf5-f5f6c2062062/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 22:29:12,009 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/52edcb76-9bd6-4eeb-adf5-f5f6c2062062/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=false, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-01 22:29:18,572 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-01 22:29:18,573 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/52edcb76-9bd6-4eeb-adf5-f5f6c2062062/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-01 22:29:18,575 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/52edcb76-9bd6-4eeb-adf5-f5f6c2062062/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
* /opt/alluxio/fuse.outException in thread "Thread-3" alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1 at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172) at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125) at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041) at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91) at java.lang.Thread.run(Thread.java:748) INFO ../../src/main/native/libjnifuse/jnifuse_helper.cc:33 Start initializing JNIFuse INFO ../../src/main/native/libjnifuse/jnifuse_helper.cc:88 JNIFuse initialized
@takersk Thanks. Unfortunately the info-level log doesn't provide enough information. Can you add the property alluxio.fuse.debug.enabled: true, restart the Alluxio cluster, reproduce this issue, then send the fuse.log and fuse.out over? We can then get the debug-level logs for further investigation.
takersk
commented
2 years ago @ssz1997 I'll switch to debug and share it again next week.
ssz1997
commented
2 years ago @takersk Another thing you can try is to use the alluxio/alluxio:2.9.0-SNAPSHOT image, and set mountInPod in your storageclass/pv to true. The fuse processes will be isolated into respective pods in this mode, and may solve the issue.
takersk
commented
2 years ago @ssz1997 There are many things to consider when upgrading the version, so I will share the fuse-logs first.
** pods logs
'state': {'running': None,
'terminated': {'container_id': 'docker://5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2',
'exit_code': 128,
'finished_at': datetime.datetime(2022, 6, 6, 21, 1, tzinfo=tzlocal()),
'message': 'error '
'while '
'creating '
'mount '
'source '
'path '
"'/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': "
'mkdir '
'/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: '
'file '
'exists',
'reason': 'ContainerCannotRun',
'signal': None,
'started_at': datetime.datetime(2022, 6, 6, 21, 1, tzinfo=tzlocal())},
'waiting': None}}],** fuse.log (debug)
2022-06-03 21:00:30,083 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-03 21:00:30,466 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse4450524986429375457.so.
2022-06-03 21:00:30,466 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-03 21:00:30,563 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-03 21:00:30,574 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-68316877811854769. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-03 21:00:30,761 INFO NettyUtils - EPOLL_MODE is available
2022-06-03 21:00:31,314 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.174.163, rack=null)
2022-06-03 21:00:31,573 INFO Reflections - Reflections took 175 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-03 21:00:31,615 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/6e209397-0425-4357-89b1-60345d19389e/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-03 21:00:31,615 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/6e209397-0425-4357-89b1-60345d19389e/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=true, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-03 21:00:34,600 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/)
2022-06-03 21:00:34,801 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/) in 200 ms
2022-06-03 21:00:40,188 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage)
2022-06-03 21:00:40,201 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage) in 12 ms
2022-06-03 21:00:40,201 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai)
2022-06-03 21:00:40,214 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai) in 13 ms
2022-06-03 21:00:40,215 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz)
2022-06-03 21:00:40,223 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz) in 8 ms
2022-06-03 21:00:40,223 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages)
2022-06-03 21:00:40,234 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages) in 11 ms
2022-06-03 21:00:40,234 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03)
2022-06-03 21:00:40,244 DEBUG AlluxioJniFuseFileSystem - Failed to getattr /stage/kcai/giftbiz/messages/2022-06-03: path does not exist or is invalid
2022-06-03 21:00:40,247 DEBUG AlluxioJniFuseFileSystem - Exit (-2): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03) in 13 ms
2022-06-03 21:00:40,249 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Mkdir(path=/stage/kcai/giftbiz/messages/2022-06-03,mode=755,)
2022-06-03 21:00:40,290 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Mkdir(path=/stage/kcai/giftbiz/messages/2022-06-03,mode=755,) in 41 ms
2022-06-03 21:00:40,290 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03)
2022-06-03 21:00:40,294 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03) in 4 ms
2022-06-03 21:00:40,295 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet)
2022-06-03 21:00:40,298 DEBUG AlluxioJniFuseFileSystem - Failed to getattr /stage/kcai/giftbiz/messages/2022-06-03/raw.parquet: path does not exist or is invalid
2022-06-03 21:00:40,299 DEBUG AlluxioJniFuseFileSystem - Exit (-2): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet) in 4 ms
2022-06-03 21:00:40,422 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet)
2022-06-03 21:00:40,427 DEBUG AlluxioJniFuseFileSystem - Failed to getattr /stage/kcai/giftbiz/messages/2022-06-03/raw.parquet: path does not exist or is invalid
2022-06-03 21:00:40,427 DEBUG AlluxioJniFuseFileSystem - Exit (-2): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet) in 5 ms
2022-06-03 21:00:40,428 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet)
2022-06-03 21:00:40,432 DEBUG AlluxioJniFuseFileSystem - Failed to getattr /stage/kcai/giftbiz/messages/2022-06-03/raw.parquet: path does not exist or is invalid
2022-06-03 21:00:40,432 DEBUG AlluxioJniFuseFileSystem - Exit (-2): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet) in 4 ms
2022-06-03 21:00:40,435 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Create(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet,mode=100644)
2022-06-03 21:00:40,469 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Create(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet,mode=100644) in 34 ms
2022-06-03 21:00:40,469 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-03/raw.parquet)
...
2022-06-05 21:01:06,588 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=2678784)
2022-06-05 21:01:06,588 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=2678784) in 0 ms
2022-06-05 21:01:06,588 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=103087,offset=2809856)
2022-06-05 21:01:06,588 DEBUG AlluxioJniFuseFileSystem - Exit (103087): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=103087,offset=2809856) in 0 ms
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=581,offset=2912943)
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Exit (581): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=581,offset=2912943) in 0 ms
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2828,offset=2913524)
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Exit (2828): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2828,offset=2913524) in 0 ms
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=2916352)
2022-06-05 21:01:06,592 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=2916352) in 0 ms
2022-06-05 21:01:06,593 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3047424)
2022-06-05 21:01:06,593 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3047424) in 0 ms
2022-06-05 21:01:06,594 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=91021,offset=3178496)
2022-06-05 21:01:06,594 DEBUG AlluxioJniFuseFileSystem - Exit (91021): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=91021,offset=3178496) in 0 ms
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=323,offset=3269517)
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Exit (323): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=323,offset=3269517) in 0 ms
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2864,offset=3269840)
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Exit (2864): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2864,offset=3269840) in 0 ms
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3272704)
2022-06-05 21:01:06,598 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3272704) in 0 ms
2022-06-05 21:01:06,599 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3403776)
2022-06-05 21:01:06,599 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3403776) in 0 ms
2022-06-05 21:01:06,599 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=114494,offset=3534848)
2022-06-05 21:01:06,599 DEBUG AlluxioJniFuseFileSystem - Exit (114494): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=114494,offset=3534848) in 0 ms
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=194,offset=3649342)
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Exit (194): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=194,offset=3649342) in 0 ms
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=62,offset=3649536)
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Exit (62): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=62,offset=3649536) in 0 ms
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=4034,offset=3649598)
2022-06-05 21:01:06,602 DEBUG AlluxioJniFuseFileSystem - Exit (4034): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=4034,offset=3649598) in 0 ms
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3653632)
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3653632) in 0 ms
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3784704)
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3784704) in 0 ms
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=9830,offset=3915776)
2022-06-05 21:01:06,603 DEBUG AlluxioJniFuseFileSystem - Exit (9830): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=9830,offset=3915776) in 0 ms
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=518,offset=3925606)
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Exit (518): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=518,offset=3925606) in 0 ms
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1940,offset=3926124)
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Exit (1940): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1940,offset=3926124) in 0 ms
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3928064)
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=3928064) in 0 ms
2022-06-05 21:01:06,607 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=4059136)
2022-06-05 21:01:06,608 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=4059136) in 1 ms
2022-06-05 21:01:06,608 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=68087,offset=4190208)
2022-06-05 21:01:06,609 DEBUG AlluxioJniFuseFileSystem - Exit (68087): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=68087,offset=4190208) in 1 ms
2022-06-05 21:01:06,611 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=345,offset=4258295)
2022-06-05 21:01:06,611 DEBUG AlluxioJniFuseFileSystem - Exit (345): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=345,offset=4258295) in 0 ms
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1200,offset=4258640)
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Exit (1200): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1200,offset=4258640) in 0 ms
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=4259840)
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Exit (131072): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=131072,offset=4259840) in 0 ms
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=116694,offset=4390912)
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Exit (116694): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=116694,offset=4390912) in 0 ms
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2090,offset=4507606)
2022-06-05 21:01:06,612 DEBUG AlluxioJniFuseFileSystem - Exit (2090): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=2090,offset=4507606) in 0 ms
2022-06-05 21:01:06,613 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1536,offset=4509696)
2022-06-05 21:01:06,613 DEBUG AlluxioJniFuseFileSystem - Exit (1536): Fuse.Write(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1,size=1536,offset=4509696) in 0 ms
2022-06-05 21:01:06,613 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Flush(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1)
2022-06-05 21:01:06,613 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Flush(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1) in 0 ms
2022-06-05 21:01:06,613 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Release(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1)
2022-06-05 21:01:06,675 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Release(path=/stage/kcai/giftbiz/messages/2022-06-05/transform.parquet,fd=1) in 62 ms
2022-06-05 21:01:07,464 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-05 21:01:07,464 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/4deada83-9292-405e-b000-49591649cd10/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-05 21:01:07,466 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/4deada83-9292-405e-b000-49591649cd10/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-05 21:25:44,893 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-05 21:25:45,253 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse2210574007035359877.so.
2022-06-05 21:25:45,253 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-05 21:25:45,334 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-05 21:25:45,343 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-1700457275457099311. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-05 21:25:45,512 INFO NettyUtils - EPOLL_MODE is available
2022-06-05 21:25:46,023 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.174.163, rack=null)
2022-06-05 21:25:46,319 INFO Reflections - Reflections took 193 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-05 21:25:46,360 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/3ede028d-66be-4a64-929e-c67a70b6a22a/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-05 21:25:46,360 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/3ede028d-66be-4a64-929e-c67a70b6a22a/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=true, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-05 21:25:47,909 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/)
2022-06-05 21:25:48,069 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/) in 160 ms
2022-06-05 21:25:48,305 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage)
2022-06-05 21:25:48,314 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage) in 9 ms
2022-06-05 21:25:48,314 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai)
2022-06-05 21:25:48,323 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai) in 9 ms
2022-06-05 21:25:48,323 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz)
2022-06-05 21:25:48,329 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz) in 6 ms
2022-06-05 21:25:48,329 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages)
2022-06-05 21:25:48,337 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages) in 8 ms
2022-06-05 21:25:48,340 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Readdir(path=/stage/kcai/giftbiz/messages)
2022-06-05 21:25:48,363 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Readdir(path=/stage/kcai/giftbiz/messages) in 22 ms
2022-06-05 21:25:48,364 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05)
2022-06-05 21:25:48,390 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05) in 26 ms
2022-06-05 21:25:48,390 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Readdir(path=/stage/kcai/giftbiz/messages/2022-05-05)
2022-06-05 21:25:48,403 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Readdir(path=/stage/kcai/giftbiz/messages/2022-05-05) in 13 ms
2022-06-05 21:25:48,403 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/transform.parquet)
2022-06-05 21:25:48,407 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/transform.parquet) in 4 ms
2022-06-05 21:25:48,408 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/transform.parquet)
2022-06-05 21:25:48,431 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/transform.parquet) in 23 ms
2022-06-05 21:25:48,431 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/raw.parquet)
2022-06-05 21:25:48,436 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/raw.parquet) in 5 ms
2022-06-05 21:25:48,436 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/raw.parquet)
2022-06-05 21:25:48,446 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/raw.parquet) in 10 ms
2022-06-05 21:25:48,447 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/group_inference.parquet)
2022-06-05 21:25:48,451 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/group_inference.parquet) in 4 ms
2022-06-05 21:25:48,451 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/group_inference.parquet)
2022-06-05 21:25:48,465 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/group_inference.parquet) in 14 ms
2022-06-05 21:25:48,466 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/inference.parquet)
2022-06-05 21:25:48,470 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-05-05/inference.parquet) in 4 ms
2022-06-05 21:25:48,470 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/inference.parquet)
2022-06-05 21:25:48,481 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Unlink(path=/stage/kcai/giftbiz/messages/2022-05-05/inference.parquet) in 11 ms
2022-06-05 21:25:48,481 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Rmdir(path=/stage/kcai/giftbiz/messages/2022-05-05)
2022-06-05 21:25:48,499 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Rmdir(path=/stage/kcai/giftbiz/messages/2022-05-05) in 18 ms
2022-06-05 21:25:49,499 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-05 21:25:49,499 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/3ede028d-66be-4a64-929e-c67a70b6a22a/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-05 21:25:49,502 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/3ede028d-66be-4a64-929e-c67a70b6a22a/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-06 21:00:57,850 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-06 21:00:58,204 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse2717559219175875493.so.
2022-06-06 21:00:58,204 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-06 21:00:58,283 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-06 21:00:58,291 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-1987941659293887464. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-06 21:00:58,457 INFO NettyUtils - EPOLL_MODE is available
2022-06-06 21:00:58,975 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.174.163, rack=null)
2022-06-06 21:00:59,264 INFO Reflections - Reflections took 191 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-06 21:00:59,304 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-06 21:00:59,305 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=true, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-06 21:01:01,879 INFO AbstractFuseFileSystem - Unmounting Fuse through shutdown hook
2022-06-06 21:01:01,880 INFO AbstractFuseFileSystem - Umounting /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
2022-06-06 21:01:01,882 ERROR AbstractFuseFileSystem - Failed to umount /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount
alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
2022-06-06 21:02:15,546 INFO AlluxioFuse - Alluxio version: 2.8.0-bcb29e5a8e5bab1f4559b38c161e090a51c3fa8f
2022-06-06 21:02:15,902 INFO NativeLibraryLoader - Loaded lib by jar from path /tmp/libjnifuse1844611896834546330.so.
2022-06-06 21:02:15,902 INFO NativeLibraryLoader - Loaded libjnifuse with libfuse version 2.
2022-06-06 21:02:15,981 INFO MetricsSystem - Starting sinks with config: {}.
2022-06-06 21:02:15,989 INFO MetricsHeartbeatContext - Created metrics heartbeat with ID app-7192947363258653197. This ID will be used for identifying info from the client. It can be set manually through the alluxio.user.app.id property
2022-06-06 21:02:16,156 INFO NettyUtils - EPOLL_MODE is available
2022-06-06 21:02:16,663 INFO TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=10.92.174.163, rack=null)
2022-06-06 21:02:16,951 INFO Reflections - Reflections took 188 ms to scan 1 urls, producing 58 keys and 199 values
2022-06-06 21:02:16,992 INFO AlluxioFuse - Mounting AlluxioJniFuseFileSystem: mount point="/var/lib/kubelet/pods/bedf0a80-fddd-4296-8afd-9608af7ad108/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount", OPTIONS="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-06 21:02:16,992 INFO AbstractFuseFileSystem - Mounting /var/lib/kubelet/pods/bedf0a80-fddd-4296-8afd-9608af7ad108/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: blocking=true, debug=true, fuseOpts="[-obig_writes, -oallow_other, -oentry_timeout=36000, -oattr_timeout=36000, -omax_readahead=0, -omax_write=131072]"
2022-06-06 21:02:18,560 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/)
2022-06-06 21:02:18,724 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/) in 164 ms
2022-06-06 21:02:20,159 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage)
2022-06-06 21:02:20,167 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage) in 8 ms
2022-06-06 21:02:20,167 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai)
2022-06-06 21:02:20,177 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai) in 9 ms
2022-06-06 21:02:20,177 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz)
2022-06-06 21:02:20,182 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz) in 5 ms
2022-06-06 21:02:20,183 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages)
2022-06-06 21:02:20,194 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages) in 11 ms
2022-06-06 21:02:20,194 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-06)
2022-06-06 21:02:20,200 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-06) in 6 ms
2022-06-06 21:02:20,201 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet)
2022-06-06 21:02:20,216 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Getattr(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet) in 15 ms
2022-06-06 21:02:20,225 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Open(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,flags=0x8000(READ_ONLY))
2022-06-06 21:02:20,242 DEBUG AlluxioJniFuseFileSystem - Exit (0): Fuse.Open(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,flags=0x8000(READ_ONLY)) in 17 ms
2022-06-06 21:02:20,265 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3854336)
2022-06-06 21:02:20,351 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3854336) in 86 ms
2022-06-06 21:02:20,351 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3858432)
2022-06-06 21:02:20,351 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3858432) in 0 ms
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3862528)
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3862528) in 0 ms
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3866624)
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3866624) in 0 ms
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3870720)
2022-06-06 21:02:20,352 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3870720) in 0 ms
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3874816)
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3874816) in 0 ms
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3878912)
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3878912) in 0 ms
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3883008)
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3883008) in 0 ms
2022-06-06 21:02:20,353 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3887104)
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3887104) in 0 ms
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3891200)
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3891200) in 0 ms
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3895296)
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3895296) in 0 ms
2022-06-06 21:02:20,354 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3899392)
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3899392) in 0 ms
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3903488)
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3903488) in 0 ms
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3907584)
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3907584) in 0 ms
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3911680)
2022-06-06 21:02:20,355 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3911680) in 0 ms
2022-06-06 21:02:20,356 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3915776)
2022-06-06 21:02:20,356 DEBUG AlluxioJniFuseFileSystem - Exit (4096): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3915776) in 0 ms
2022-06-06 21:02:20,356 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3919872)
2022-06-06 21:02:20,366 DEBUG AlluxioJniFuseFileSystem - Exit (3252): Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=3919872) in 10 ms
2022-06-06 21:02:20,368 DEBUG AlluxioJniFuseFileSystem - Enter: Fuse.Read(path=/stage/kcai/giftbiz/messages/2022-06-06/raw.parquet,fd=0,size=4096,offset=0)
... END** fuse.out (debug)
Exception in thread "Thread-3" alluxio.jnifuse.FuseException: Unable to umount FS with exit code 1
at alluxio.jnifuse.AbstractFuseFileSystem.umountInternal(AbstractFuseFileSystem.java:172)
at alluxio.jnifuse.AbstractFuseFileSystem.umount(AbstractFuseFileSystem.java:125)
at alluxio.fuse.AlluxioJniFuseFileSystem.umount(AlluxioJniFuseFileSystem.java:1041)
at alluxio.jnifuse.AbstractFuseFileSystem.lambda$mount$0(AbstractFuseFileSystem.java:91)
at java.lang.Thread.run(Thread.java:748)
INFO ../../src/main/native/libjnifuse/jnifuse_helper.cc:33 Start initializing JNIFuse
INFO ../../src/main/native/libjnifuse/jnifuse_helper.cc:88 JNIFuse initialized@takersk Thanks. Can you also share the log of the nodeserver on the sama machine?
takersk
commented
2 years ago @ssz1997 Can I share the sama machine kubelet log?
$ cat kubelet.ERROR | grep 34fa28d2-2a44-4e01-b37a-85fa6f56730f
E0607 06:01:00.666435 538690 remote_runtime.go:222] StartContainer "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2" from runtime service failed: rpc error: code = Unknown desc = failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666500 538690 kuberuntime_manager.go:809] container start failed: RunContainerError: failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666540 538690 pod_workers.go:191] Error syncing pod 34fa28d2-2a44-4e01-b37a-85fa6f56730f ("etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"), skipping: failed to "StartContainer" for "base" with RunContainerError: "failed to start container \"5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2\": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists"$ kubelet.INFO | grep 34fa28d2-2a44-4e01-b37a-85fa6f56730f
I0607 06:00:57.220360 538690 kubelet.go:1909] SyncLoop (ADD, "api"): "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"
I0607 06:00:57.391702 538690 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "alluxio-airflow-pv" (UniqueName: "kubernetes.io/csi/alluxio^alluxio") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:57.391735 538690 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "airflow-worker-token-bkx8f" (UniqueName: "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:57.491925 538690 reconciler.go:254] operationExecutor.MountVolume started for volume "alluxio-airflow-pv" (UniqueName: "kubernetes.io/csi/alluxio^alluxio") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:57.491962 538690 reconciler.go:254] operationExecutor.MountVolume started for volume "airflow-worker-token-bkx8f" (UniqueName: "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:57.493422 538690 operation_generator.go:587] MountVolume.MountDevice succeeded for volume "alluxio-airflow-pv" (UniqueName: "kubernetes.io/csi/alluxio^alluxio") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/alluxio-airflow-pv/globalmount"
I0607 06:00:57.502202 538690 operation_generator.go:634] MountVolume.SetUp succeeded for volume "airflow-worker-token-bkx8f" (UniqueName: "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:59.621511 538690 operation_generator.go:634] MountVolume.SetUp succeeded for volume "alluxio-airflow-pv" (UniqueName: "kubernetes.io/csi/alluxio^alluxio") pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:00:59.646046 538690 kuberuntime_manager.go:430] No sandbox for pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)" can be found. Need to start a new one
I0607 06:00:59.850369 538690 kubelet.go:1954] SyncLoop (PLEG): "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)", event: &pleg.PodLifecycleEvent{ID:"34fa28d2-2a44-4e01-b37a-85fa6f56730f", Type:"ContainerDied", Data:"c1ca73aab945f45118741f4b730be64d2ad5e05778e92b988181d544d0fb612e"}
E0607 06:01:00.666435 538690 remote_runtime.go:222] StartContainer "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2" from runtime service failed: rpc error: code = Unknown desc = failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666500 538690 kuberuntime_manager.go:809] container start failed: RunContainerError: failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666540 538690 pod_workers.go:191] Error syncing pod 34fa28d2-2a44-4e01-b37a-85fa6f56730f ("etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"), skipping: failed to "StartContainer" for "base" with RunContainerError: "failed to start container \"5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2\": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists"
I0607 06:01:00.864475 538690 kubelet.go:1954] SyncLoop (PLEG): "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)", event: &pleg.PodLifecycleEvent{ID:"34fa28d2-2a44-4e01-b37a-85fa6f56730f", Type:"ContainerStarted", Data:"c1ca73aab945f45118741f4b730be64d2ad5e05778e92b988181d544d0fb612e"}
I0607 06:01:00.998028 538690 reconciler.go:183] operationExecutor.UnmountVolume started for volume "data-volume" (UniqueName: "kubernetes.io/csi/alluxio^alluxio") pod "34fa28d2-2a44-4e01-b37a-85fa6f56730f" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:01:00.998072 538690 reconciler.go:183] operationExecutor.UnmountVolume started for volume "airflow-worker-token-bkx8f" (UniqueName: "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f") pod "34fa28d2-2a44-4e01-b37a-85fa6f56730f" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f")
I0607 06:01:01.000088 538690 operation_generator.go:713] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f" (OuterVolumeSpecName: "airflow-worker-token-bkx8f") pod "34fa28d2-2a44-4e01-b37a-85fa6f56730f" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f"). InnerVolumeSpecName "airflow-worker-token-bkx8f". PluginName "kubernetes.io/secret", VolumeGidValue ""
I0607 06:01:01.098401 538690 reconciler.go:303] Volume detached for volume "airflow-worker-token-bkx8f" (UniqueName: "kubernetes.io/secret/34fa28d2-2a44-4e01-b37a-85fa6f56730f-airflow-worker-token-bkx8f") on node "dkosv3-cdp-ml-stage-worker-t694" DevicePath ""
I0607 06:01:01.116446 538690 operation_generator.go:713] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/alluxio^alluxio" (OuterVolumeSpecName: "data-volume") pod "34fa28d2-2a44-4e01-b37a-85fa6f56730f" (UID: "34fa28d2-2a44-4e01-b37a-85fa6f56730f"). InnerVolumeSpecName "alluxio-airflow-pv". PluginName "kubernetes.io/csi", VolumeGidValue ""
I0607 06:01:01.316095 538690 kubelet.go:1925] SyncLoop (DELETE, "api"): "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"
I0607 06:01:01.321218 538690 kubelet.go:1919] SyncLoop (REMOVE, "api"): "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"
I0607 06:01:01.321245 538690 kubelet.go:2118] Failed to delete pod "etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)", err: pod not found
W0607 06:01:01.741685 538690 kubelet_getters.go:297] Path "/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes" does not exist
I0607 06:01:01.874931 538690 status_manager.go:542] Status for pod "34fa28d2-2a44-4e01-b37a-85fa6f56730f" is up-to-date; skipping$ cat kubelet.WARNING | grep 34fa28d2-2a44-4e01-b37a-85fa6f56730f
E0607 06:01:00.666435 538690 remote_runtime.go:222] StartContainer "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2" from runtime service failed: rpc error: code = Unknown desc = failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666500 538690 kuberuntime_manager.go:809] container start failed: RunContainerError: failed to start container "5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists
E0607 06:01:00.666540 538690 pod_workers.go:191] Error syncing pod 34fa28d2-2a44-4e01-b37a-85fa6f56730f ("etl-giftbiz-message-transform-dfa1a881daf04279a65dfd2c937e960b_airflow(34fa28d2-2a44-4e01-b37a-85fa6f56730f)"), skipping: failed to "StartContainer" for "base" with RunContainerError: "failed to start container \"5965c53d01d673adf44ea791d4483ab63344d0dda549e3f9a41874fce6a29cd2\": Error response from daemon: error while creating mount source path '/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount': mkdir /var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes/kubernetes.io~csi/alluxio-airflow-pv/mount: file exists"
W0607 06:01:01.741685 538690 kubelet_getters.go:297] Path "/var/lib/kubelet/pods/34fa28d2-2a44-4e01-b37a-85fa6f56730f/volumes" does not existhi!! Are you still in the process of resolving the issue?
ssz1997
commented
2 years ago @takersk Seems like this is not a deterministic failure right? This pr https://github.com/Alluxio/alluxio/pull/15788 may fix it. You can try it out.
takersk
commented
2 years ago @ssz1997 thank you I'll test it when the tag version goes up.
takersk
commented
2 years ago @ssz1997 I applied the code patch, and after a few days of monitoring, I will share the results.
takersk
commented
2 years ago @ssz1997 hi!! No issues have occurred since apply on 08.03. please close the issue
ssz1997
commented
2 years ago @tak Glad to hear it's resolved. Thanks for reporting!
Alluxio Version: v2.8.0 (v2.7.1 same error)
Describe the bug Intermittent error when creating pods
To Reproduce see above
Expected behavior mount success
Urgency Describe the impact and urgency of the bug.
Are you planning to fix it No.
Additional context ...