search

Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.87k stars 2.94k forks source link

Failed to free 67108864 bytes space at location #17650

Open lingchuanzhou opened 1 year ago

lingchuanzhou commented 1 year ago

Alluxio Version: 2.9.1

Describe the bug when the alluxio disk space is occupied, the alluxio can't free disk space. and this is my config: -Dalluxio.master.journal.type=EMBEDDED -Dalluxio.master.journal.folder=/journal -Dalluxio.master.embedded.journal.addresses=,alluxio-master-0:19200,alluxio-master-1:19200,alluxio-master-2:19200 -Dalluxio.security.stale.channel.purge.interval=365d -Dalluxio.master.mount.table.root.ufs=s3://alluxio -Dalluxio.underfs.s3.endpoint=http://xxx:12001 -Dalluxio.underfs.s3.disable.dns.buckets=true -Dalluxio.underfs.s3.inherit.acl=false -Ds3a.accessKeyId=xxx -Ds3a.secretKey=xxx -Dalluxio.security.authentication.type=NOSASL -Dalluxio.security.authorization.permission.enabled=false -Dalluxio.debug=true -Dalluxio.user.short.circuit.enabled=false -Dalluxio.worker.data.server.domain.socket.as.uuid=false -Dalluxio.user.ufs.block.read.location.policy=alluxio.client.block.policy.DeterministicHashPolicy -Dalluxio.user.ufs.block.read.location.policy.deterministic.hash.shards=3 -Dalluxio.user.file.replication.durable=1 -Dalluxio.user.file.replication.min=1 -Dalluxio.user.network.netty.worker.threads=1920 -Dalluxio.user.block.worker.client.pool.max=4048 -Dalluxio.user.network.streaming.max.connections=1280 -Dalluxio.worker.network.block.reader.threads.max=4048 -Dalluxio.worker.network.async.cache.manager.threads.max=768 -Dalluxio.worker.network.async.cache.manager.queue.max=1024 -Dalluxio.master.metadata.sync.concurrency.level=192 -Dalluxio.master.metadata.sync.executor.pool.size=384 -Dalluxio.user.streaming.data.read.timeout=500s -Dalluxio.user.streaming.data.write.timeout=500s -Dalluxio.underfs.s3.socket.timeout=600sec -Dalluxio.underfs.s3.request.timeout=6min -Dalluxio.underfs.s3.admin.threads.max=160 -Dalluxio.underfs.s3.threads.max=320 -Dalluxio.underfs.s3.upload.threads.max=160 -Dalluxio.underfs.object.store.service.threads=160 -Dalluxio.worker.tieredstore.levels=1 -Dalluxio.worker.tieredstore.level0.alias=HDD -Dalluxio.worker.tieredstore.level0.dirs.mediumtype=HDD -Dalluxio.worker.tieredstore.level0.dirs.path=/data/alluxio -Dalluxio.worker.tieredstore.level0.dirs.quota=17TB -Dalluxio.worker.tieredstore.level0.watermark.high.ratio=0.95 -Dalluxio.worker.tieredstore.level0.watermark.low.ratio=0.7

and this is the error log: 2023-06-19 01:56:34,621 WARN UnderFileSystemBlockReader - Failed to update block writer for UFS block [blockId: 3505515397120, ufsPath: s3://aip/datalake_latest/network_security_log_latest/tables/data_table/dt=2023-06-16/network_security_log-081b49b81ae043299bb40ca44a3deef1.snappy.parquet, offset: 0]: alluxio.exception.runtime.ResourceExhaustedRuntimeException: Failed to free 67108864 bytes space at location . Min contiguous requested: 67108864, Min available requested: 67108864, Blocks iterated: 507512, Blocks removed: 0, Space freed: 0

To Reproduce Steps to reproduce the behavior (as minimally and precisely as possible)

Expected behavior A clear and concise description of what you expected to happen.

Urgency Describe the impact and urgency of the bug.

Are you planning to fix it Please indicate if you are already working on a PR.

Additional context Add any other context about the problem here.

LuQQiu commented 1 year ago

@dbw9580 FYI

lingchuanzhou commented 1 year ago

@dbw9580