search

Azure / azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Other
659 stars 207 forks source link

BlockCache::OpenFile : Failed to get block list #1536

Open Niko-CZ opened 6 days ago

Niko-CZ commented 6 days ago

Which version of blobfuse was used?

blobfuse2-2.3.2-1.x86_64 fuse-2.9.2-11.el7.x86_64 fuse3-3.6.1-4.el7.x86_64 fuse3-libs-3.6.1-4.el7.x86_64

Which OS distribution and version are you using?

Red Hat Enterprise Linux release 8.9 (Ootpa) x86_64 4.18.0-513.11.1.el8_9.x86_64 #1 SMP Thu Dec 7 03:06:13 EST 2023 x86_64 x86_64 x86_64 GNU/Linux

If relevant, please share your mount command.

/usr/bin/blobfuse2 mount /oracle/Blob_BAK --config-file=/etc/Blobfuse2-BlockCacheConfig.yaml; cat /etc/Blobfuse2-BlockCacheConfig.yaml; allow-other: true logging: type: base level: log_debug file-path: /var/log/blobfuse2/blobfuse2.log max-file-size-mb: 64 file-count: 2 track-time: true components: -libfuse -block_cache -attr_cache -azstorage health_monitor: enable-monitoring: true stats-poll-interval-sec: 10 process-monitor-interval-sec: 30 output-path: /mnt/resource/blobfuse2_healthmon monitor-disable-list: -blobfuse_stats libfuse: fuse-trace: true default-permission: 0777 attribute-expiration-sec: 240 entry-expiration-sec: 240 negative-entry-expiration-sec: 120 ignore-open-flags: true block_cache: block-size-mb: 256 mem-size-mb: 20480 prefetch: 20 parallelism: 128 attr_cache: timeout-sec: 7200 azstorage: type: block account-name: xxx endpoint: xxx mode: msi container: xxx honour-acl: false

What was the issue encountered?

Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : response 200 Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_DEBUG [utils.go (156)]: SDK(Retry) : exit due to non-retriable status code Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [block_cache.go (349)]: BlockCache::OpenFile : Failed to get block list of X28_3965183084/X28_3965183084_20241008_51_1 [<nil>] Tue Oct 8 13:37:50 UTC 2024 : blobfuse2[159980] : [/oracle/Blob_BAK] LOG_ERR [libfuse_handler.go (712)]: Libfuse::libfuse_open : Failed to open X28_3965183084/X28_3965183084_20241008_51_1 [failed to retrieve bloc k list for X28_3965183084/X28_3965183084_20241008_51_1]

In the same time file is being created: ls -lsa /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1 0 -rwxrwxrwx 1 root root 67117056 Oct 8 13:37 /oracle/Blob_BAK/X28_3965183084/X28_3965183084_20241008_51_1

blobfuse2_log_block_cache.log We have backups working with the same configuration, but the streaming cache instead of block. After reading of fresh release notes (https://github.com/Azure/azure-storage-fuse/blob/main/README.md):

streaming mode is being deprecated.

I decided to enable block caching changing configuration part: stream: block-size-mb: 8 max-buffers: 16 buffer-size-mb: 384 To: block_cache: block-size-mb: 256 mem-size-mb: 20480 prefetch: 20 parallelism: 128 But facing issue above with any combination of parameters in block_cache section.

Have you found a mitigation/solution?

No.

Please share logs if available.

File blobfuse2_log_block_cache.log is attached.

ashruti-msft commented 3 days ago

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

Niko-CZ commented 3 days ago

In the same time file is being created:

You mean to say that the file was created successfully but you saw that error message in the logs, is that correct?

Yes, file created successfully with the 64M size.