search

Azure / azure-storage-fuse

A virtual file system adapter for Azure Blob storage
Other
660 stars 206 forks source link

Abnormal behavior during MSSQL DB backup (file_cache) #1547

Open olegshv opened 6 days ago

olegshv commented 6 days ago

Which version of blobfuse was used?

blobfuse2 version 2.3.2

Which OS distribution and version are you using?

Debian GNU/Linux 11 (bullseye)

My fuse_connection.yaml file

Image The configuration was the same for tests on different blobfuse2 versions, and I have tried different parameters.

What was the issue encountered?

We are still searching for a reliable database backup config because after updating PostgreSQL to version 16, every large database (approximately more than 50GB) crashes with an error:

     pg_dump: error: could not open file "/mnt/Backup/pgdb-20241019-093629/site1/site1-db-Dump.backup": Device or resource busy

It seems that something was blocking access to this file, and this was the way to fix it: refresh-sec: 600 (maybe you can give some advice on this).

But we faced a problem with MSSQL backups. We use an outdated and stable version for us (2.0.0-preview.3), so we found it only during tests. When we start backup a 200 GB database, the backup constantly crashes with an error:

          Write on "\\backup-dir\db-full.bak" failed: 59(An unexpected network error occurred.)
          BACKUP DATABASE is terminating abnormally.

You can see what is happening very well on the graph.

Image Image

In older versions of blobfuse2, a backup was made and then uploaded to Azure Storage. On newer versions, a file with a final size of 200 Gb is created in Azure Storage. It immediately starts to load the network interface completely, which causes the backup to fail with the error: An unexpected network error occurred.

Please share logs if available.

blobfuse2-logs.zip

syeleti-msft commented 3 days ago

Hi @olegshv , In older versions of blobfuse2, a backup was made and then uploaded to Azure Storage. On newer versions, a file with a final size of 200 Gb is created in Azure Storage. It immediately starts to load the network interface completely, which causes the backup to fail with the error: An unexpected network error occurred.

Yes this is true, there is a recent change after 2.0.3 where some logic was modified, which is causing the filecache to upload the entire file(i.e., Empty file) after a truncate operation(Which is used to modify the file size). According to your DB backup logs, tool makes a truncate operation with size 204.8GB this is causing the blobfuse to upload the entire file with null bytes before starting any write on the file.

We will fix this in our next release 2.4.0