search

restic / restic

Fast, secure, efficient backup program
https://restic.net
BSD 2-Clause "Simplified" License
26.51k stars 1.56k forks source link

Temporary pack files are still written to disk on Windows #4702

Open n1fi opened 8 months ago

n1fi commented 8 months ago

Output of restic version

restic 0.16.4 compiled with go1.21.6 on windows/amd64

What backend/service did you use to store the repository?

A local REST Server.

Problem description / Steps to reproduce

I back up drives from Windows 10 to a local REST Server. I used the option --pack-size 64 but it’s the same with the default. The pack files are written to disk. I have plenty of free RAM available. I confirmed that they are written to disk by checking the total host writes on the SSD.

restic.exe -v -r rest:http://192.168... backup "Folder1" "Folder2" --exclude-if-present .restic_exclude

From the documentation:

Note that larger pack files increase the chance that the temporary pack files are written to disk. An operating system usually caches file write operations in memory and writes them to disk after a short delay. As larger pack files take longer to upload, this increases the chance of these files being written to disk. This can increase disk wear for SSDs.

Related forum topic: Avoid writing temporary pack files to disk?

Expected behavior

Temporary pack files are not written to disk according to #3610

In addition, the flag FILE_ATTRIBUTE_TEMPORARY is added to avoid unnecessary writes of the temporary files to disk.

Actual behavior

For some reason Windows still writes the files to disk.

Do you have any idea what may have caused this?

Unfortunately, no.

Did restic help you today? Did it make you happy in any way?

It's my favorite backup tool. The new compression feature is great, it saved me an additional 230 GiB (12%) of storage space.

MichaelEischer commented 8 months ago

If someone with more experience with the Windows API has an idea why the FILE_ATTRIBUTE_TEMPORARY flag is apparently ignored, that would be really helpful.

Dobatymo commented 8 months ago

According to msdn https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilew

Specifying the FILE_ATTRIBUTE_TEMPORARY attribute causes file systems to avoid writing data back to mass storage if sufficient cache memory is available, because an application deletes a temporary file after a handle is closed. In that case, the system can entirely avoid writing the data. Although it does not directly control data caching in the same way as the previously mentioned flags, the FILE_ATTRIBUTE_TEMPORARY attribute does tell the system to hold as much as possible in the system cache without writing and therefore may be of concern for certain applications.

they should only be written to disk if there is not enough memory. Maybe the handles are not closed quickly enough so Windows still writes them to disk.