Behavior of File_Cache Mode

[Shunya-Seki]

Which version of blobfuse was used? blobfuse2 version: 2.3.0~preview.1

Which OS distribution and version are you using? Rhel8.8

If relevant, please share your mount command. ◆mount command blobfuse2 mount /blobmount --config-file=/etc/blobfuse2config.yaml -o allow_other -o direct_io

◆config.yaml

Refer ./setup/baseConfig.yaml for full set of config parameters

logging: type: syslog level: log_debug

components:

• libfuse
• file_cache
• attr_cache
• azstorage

file_cache: path: /cache

azstorage: type: block account-name: xxx endpoint: xxx mode: msi container: xxx

What was the issue encountered?

Please provide information on the following regarding file_cache mode. ・What triggers the cache to be saved to the specified location? ・What is the default value for the timing of cache expiration? ・Is it possible for the capacity of the cache storage to exceed the capacity of the logs in the Blob Storage?

Have you found a mitigation/solution? No.

[souravgupta-msft]

Hi @Shunya-Seki. Thanks for reaching out. Please find the response below.

• Opening file handle will lead to downloading the blob from the container and saving it in the file cache path defined in the config file.
• 120 seconds is the default timeout for cache expiration.
• You can define the max size of the cache in the config. If the cache size goes beyond 80% of the size defined, then it will trigger eviction from the cache.

Also, in your mount command you are using direct_io. Please note that direct_io is used for bypassing kernel cache and it is recommended to disable all forms of caching (file and attribute cache) if you are using this.

[Shunya-Seki]

Thank you. I understand about direct_io as well.

I made the following settings in /etc/fstab and performed automatic mounting. The file cache file remains in the location of the cache and the mount failed. Is there a way to deal with it?

blobfuse2 /blobmount fuse3 defaults,_netdev,--config-file=/etc/blobfuse2config.yaml,allow_other,direct_io 0 0

[souravgupta-msft]

What error are you getting while trying to mount using fstab?

[Shunya-Seki]

When the OS was booted, the following error appeared on the boot screen.

[FAILED] Failed to mount /blobmount. See 'systemctl status blobmount.mount' for details. [DEPEND] Dependency failed for Remote File Systems. Starting Crash recovery kernel arming...

Also, when I tried to mount it manually, I got the following error.

Error: failed to initialize new pipeline [config error in file_cache [temp directory not empty]]

[souravgupta-msft]

Error: failed to initialize new pipeline [config error in file_cache [temp directory not empty]]

Your file cache directory is not empty. You can try one of the below options,

• Choose a directory for file cache which is empty.
• Use the config option allow-non-empty-temp to allow non-empty file cache directory.
• Use the config option cleanup-on-start to clean the file cache directory on startup.

[Shunya-Seki]

Thank you. I'll give it a try. What is the difference between the linked "devConfig.yaml" and "the baseConfig.yaml" below?

https://github.com/Azure/azure-storage-fuse/blob/main/setup/baseConfig.yaml

・Can I use this option(allow-non-empty-temp/cleanup-on-start) with the version I am using? (blobfuse2 version: 2.3.0~preview.1) ・Is this option(allow-non-empty-temp/cleanup-on-start) being verified?

[Shunya-Seki]

Excuse me for the addition. The following results were obtained when verified.

◆ Pattern 1 Add only cleanup-on-start to the config and restart the server with the cache file remaining in the cache storage destination. As a result, the mount failed. Even after the server was started, the cache file before the restart remained in the cache storage location.

◆ Pattern 2 Add both allow-non-empty-temp and cleanup-on-start to the config and restart the server with the cache files still in the cache store. As a result, the mount was successful. After the server was started, the cache files before the restart were deleted in the cache storage location.

Based on the above, please tell us the following.

• When the server is restarted, when does cleanup-on-start work? ・When using cleanup-on-start, is it also mandatory to set allow-non-empty-temp?

[souravgupta-msft]

What is the difference between the linked "devConfig.yaml" and "the baseConfig.yaml" below?

"baseConfig.yaml" contains the general options which can be used in most of the scenarios. "devConfig.yaml" contains the exhaustive list of all the config options which are supported in Blobfuse.

Can I use this option(allow-non-empty-temp/cleanup-on-start) with the version I am using? (blobfuse2 version: 2.3.0~preview.1) ・Is this option(allow-non-empty-temp/cleanup-on-start) being verified?

Yes, both the options are supported in the above version. You can also update to use the latest release blobfuse2-2.3.0.

[souravgupta-msft]

When the server is restarted, when does cleanup-on-start work? ・When using cleanup-on-start, is it also mandatory to set allow-non-empty-temp?

This is a bug in the code. Basically, allow-non-empty-temp check is done before which fails the mount since cache directory is not empty. We will fix this bug in our next release. For now, you can enable both the options in the config file.

[Shunya-Seki]

I understand that it is a bug.

「Basically, allow-non-empty-temp check is done before which fails the mount since cache directory is not empty.」 Could it be a mistake below? 「Basically, cleanup-on-start check is done before which fails the mount since cache directory is not empty.」

[Shunya-Seki]

I'm sorry so many times. Originally, it is assumed that it can be mounted at server startup only with cleanup-on-start, but it is currently not possible. Do you agree that it will be renovated in the future?

[souravgupta-msft]

I discussed with my team. This behavior is expected. allow-non-empty-temp indicates to allow nonempty local cache directory. cleanup-on-start is an added option on top of this. For your use case you can decide if you want to keep/delete the cache directory or not on mounting. For deleting on mount, you'll have to enable both the options.

[souravgupta-msft]

Closing, as this is expected. For more queries, you can post in this thread.

[Shunya-Seki]

@souravgupta-msft Sorry after closing. Let me confirm it additionally.

If "clean-up-start" is false and the server is restarted without manually unmounting. Won't the cache files that were left just before the reboot disappear? Even if you manually mount again with "allow-non-empty-temp" set to true after rebooting, Won't the cache files that were left just before the reboot disappear?

[souravgupta-msft]

You'll have to set both allow-non-empty-temp and clean-up-start as true in config for your use case.

[Shunya-Seki]

@souravgupta-msft Thank you. Sure. If you restart the server without manually unmounting it, Is it correct to understand that the cache files remaining just before the server restart do not disappear as expected?

[souravgupta-msft]

Yes, if you do not gracefully unmount the blobfuse process (like in case of server restarts), the files in the local cache directory will not be deleted. If you want to mount on server restarts using fstab, you will have to enable the two options mentioned above in your config, assuming your cache directory is not empty.