Restoring paths which are not valid on windows

[andrey-infinity]

Made a backup on MacOs M2 with following command rustic -r mac-data backup /Volumes/Data using rustic v0.7.0

Later tried to restore it with rustic v0.8.0 using command: Restore: PS R:\> .\rustic.exe -r mac-data restore latest R:\ Got a following crash report.

Also tried to restore it with restic - also got bunch of errors that some files cannot be read. Total size is 100k files and 309GB.

report-b21314ad-079a-403c-a7c7-f72247063914.txt

[aawsome]

Thanks for reporting @andrey-infinity.

set_length in https://github.com/rustic-rs/rustic_core/blob/main/crates/core/src/backend/local_destination.rs#L466 is failing here on Windows. While the create_dir_all did not return an error, the OpenOptions::new().create(true).truncate(false).write(true).open(filename) returned the NotFound, message: "The system cannot find the path specified." error...

So, either the path was not created by the first command, or the second command fails even so the parent path exists..?!? This is all standard Rust functionality, don't know why this shouldn't work...

Can you please check if that dir was correctly created? Also, can you check if this works on MacOS / Linux - I actually wouldn't expect this kind of error there..

[andrey-infinity]

Doing it now, however process is rather slow and without constant update, is it realistic that in 30min, only 927MB were restored? Sounds like an error in calculation or it would really take me 7d to restore 300GB:

[kapitainsky]

I might be wrong but I think that the issue is with trailing space after 2024-04 Sergei visit dir name:

Volumes\Data\0 - Photos\2024-04 Sergei visit \IMG_5423.HEIC

Windows most likely saves it without the space:

https://learn.microsoft.com/en-us/troubleshoot/windows-client/shell-experience/file-folder-name-whitespace-characters

and it breaks rustic logic in some way.

Windows is disaster here as its filesystem allows it but user space tools not. So with a little bit trickery it is possible to create file or directory no standard Windows tool can delete... and other similar nonsense situations unheard in normal OS:)

[andrey-infinity]

Okay, restore worked fine on macos (even tho a bit confusing reporting). But still fails on windows

[andrey-infinity]

@kapitainsky, this does sound like it, since I remember some other files failed to restore using restic, and I've checked they also do have trailing spaces after.

Would that be in project guideline, after making a backup to output a report with found incompatibilities. I would be happy to get report such as: "Found 10 files that will have problems restoring on windows"

Having said that, when the problem is explained, it doesn't block me.

[kapitainsky]

@kapitainsky, this does sound like it, since I remember some other files failed to restore using restic, and I've checked they also do have trailing spaces after.

Good if this is the case as at least we know what is causing it. And for sure this is a bug that rustic panics - it should be handled gracefully with error message and file/directory restoration failure. Not abandoning all task.

In general it is something to pay attention too - how illegal characters are handled. For example ? is perfectly fine in Linux or macOS but not on Windows. But this should result with failure to create item containing it. Leading or trailing spaces are special as no error is returned by OS but effectively different item with different name created.

[aawsome]

@kapitainsky Thanks for your answer! I also suspected that this is something like this.

I just wonder what is the best way to handle this. Leading/trailing whitespaces are allowed on *nix and are of course supported by the repository format. So yet, most likely the best way to handle this is to display an error and proceed - and ideally show at the end "x error occured during restore" with an non-zero exit code.

This kind of error handling is anyway open.

Another possibility would be to add an (windows-only?) option like trim-names which trims names to match windows specifications. In this case that would mean that it would trim the trailing space. This could however cause problems/irritation if both the trimmed and non-trimmed name exists in the snapshot, so I would make this non-default behavior.

[kapitainsky]

Another possibility would be to add an (windows-only?) option like trim-names

I would be strongly against it... as it opens can of worms with all exceptions and edge cases.

I think logic should be clean - something is supported then it is restores. If not then error is produced and restoration job continues.

It is user responsibility to ensure cross platform compatibility - path lengths, characters used etc.

The key is error handling and logging. In this particular case I can imagine that dir without trailing space is created but error is logged that restoration of it had problem maybe with a hint to check cross platform compatibility (length, characters used).

[aawsome]

Actually there is the possibility to always restore to individual paths/names:

In this case something like

rustic restore <SNAP>:\Volumes\Data\0 - Photos\2024-04 Sergei visit \ Volumes\Data\0 - Photos\2024-04 Sergei visit\

(note the trailing space is included in the snapshot path and excluded in the target, might however need some escaping depending on the shell you are calling this) should actually work .

[kapitainsky]

One more thought.

This is absolutely not Windows specific problem as suggested wrongly in the title now . The real issue here is that rustic does not handle exceptions well.

I can be on Linux restoring locally created backup but my destination can be different filesystem (NFS/SMB/FUSE mounted for example) with very different restrictions.

Two things can happen in general I think:

1. Hard error - for example character not supported, name too long, no free space left etc. and restored item creation fails with error - rustic should log it and continue.

2. "Soft error" - example like in orginal post - there is no error returned by system call rustic used to write restored item but it was transformed or even not created at all (all weird things can happen in real life). It should be noticed, logged and all process should continue.

There should be flag to stop on error if user wishes but by default IMO all process should continue regardless of errors. Good example is lack of space error - It can be just one item too big to fit into remaining space but actually all other items can be restored.

Above outlined behaviour should always work regardless of given situation restrictions. I would be totally against trying to handle specific cases like e.g. trailing spaces in Windows. It would be never ending quest to get it right.

In ideal world rustic could support external script invocation for names transformation - it would be up to users to handle their specific cases and apply characters substitutions or even wild things like names shortening. In similar fashion like today rusitc site provides examples of toml configs it could provide few most common script needed. In time community should provide some useful stuff here.

[simonsan]

I agree with the (soft) error handling debate and think this should be handled gracefully and restoring should be continuing, and an error being logged.

I do think, that changing the filename could lead to problems down the line - so I agree we shouldn't go down this rabbit hole.

IMHO we should make it easy to see and understand what is the error and then even give some advice how to restore that file, e.g. give a composited command to copy and paste from/to terminal to restore this file. This would make it easy for the user to handle this error, as they could just use the printed command and copy them to their terminal.