Closed bdillahu closed 1 year ago
That error is rather strange, apparently macOS now returns ENOTTY
(inappropriate ioctl for device) when getting told to actually write the data to disk. Can you test whether https://github.com/restic/rest-server/pull/199 "fixes" (rather works around) the problem?
Which filesystem does the external drive use? Note that without support for fsync, you might loose data if the disk is unplugged or looses power unexpectedly.
Sorry, doesn’t seem to change things...
On Oct 29, 2022, at 17:03, Michael Eischer @.***> wrote:
That error is rather strange, apparently macOS now returns ENOTTY (inappropriate ioctl for device) when getting told to actually write the data to disk. Can you test whether #199 https://github.com/restic/rest-server/pull/199 "fixes" (rather works around) the problem?
Which filesystem does the external drive use? Note that without support for fsync, you'll likely loose data if the disk is unplugged or looses power unexpectedly.
— Reply to this email directly, view it on GitHub https://github.com/restic/rest-server/issues/198#issuecomment-1295974405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPWSRW675RBEP7AYQZLYODWFWGJVANCNFSM6AAAAAARPARB6A. You are receiving this because you authored the thread.
Can you try to run rest-server like sudo dtruss rest-server ...
to get a trace of all the system calls?
Also, as Michael asked, what filesystem are you using on the external disk?
Is the problem reproducible when storing data to APFS on your main disk?
Sorry, drive filesystem is ExFat
Seems to run fine with a repository on my internal APFS (encrypted) drive.
I figured out that I can recreate it without rest-server involved - so I guess you guys are off the hook, but if anybody has any ideas, let me know :-)
✗ restic --repo "/Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry" snapshots enter password for repository: repository 65d1f988 opened (repository version 2) successfully, password is correct Save(<lock/e885a83278>) returned error, retrying after 552.330144ms: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2477433873: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.080381816s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-1565657784: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.31013006s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-1035312087: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.582392691s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2265441250: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 2.340488664s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2386770608: inappropriate ioctl for device signal interrupt received, cleaning up
Tried the dtruss - not sure that got anything good, but…
✗ sudo dtruss restic --repo "/Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry" snapshots dtrace: system integrity protection is on, some features will not be available
SYSCALL(args) = return munmap(0x741D000, 0x98000) = 0 0 munmap(0x74B5000, 0x8000) = 0 0 munmap(0x74BD000, 0x4000) = 0 0 munmap(0x74C1000, 0x4000) = 0 0 munmap(0x74C5000, 0x50000) = 0 0 fsgetpath(0x7FF7BFEFE530, 0x400, 0x7FF7BFEFE518) = 43 0 fsgetpath(0x7FF7BFEFE530, 0x400, 0x7FF7BFEFE518) = 14 0 csrctl(0x0, 0x7FF7BFEFE93C, 0x4) = -1 1 __mac_syscall(0x7FF80E0EA08A, 0x2, 0x7FF7BFEFE860) = 0 0 csrctl(0x0, 0x7FF7BFEFE94C, 0x4) = -1 1 __mac_syscall(0x7FF80E0E7437, 0x5A, 0x7FF7BFEFE8E0) = 0 0 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 open("/\0", 0x20100000, 0x0) = 3 0 openat(0x3, "System/Cryptexes/OS\0", 0x100000, 0x0) = 4 0 dup(0x4, 0x0, 0x0) = 5 0 fstatat64(0x4, 0x7FF7BFEFD681, 0x7FF7BFEFDA80) = 0 0 openat(0x4, "System/Library/dyld/\0", 0x100000, 0x0) = 6 0 fcntl(0x6, 0x32, 0x7FF7BFEFD710) = 0 0 dup(0x6, 0x0, 0x0) = 7 0 unknown command "Data/Restic-Rest-Server/restic-backup-masonry" for "restic" dup(0x5, 0x0, 0x0) = 8 0 close(0x3) = 0 0 close(0x5) = 0 0 close(0x4) = 0 0 close(0x6) = 0 0
The dstruss output is not useful. Can you edit the message and remove it? It's very long.
The only similar issue I found is https://github.com/realm/realm-core/issues/5789
They mention F_BARRIERFSYNC
as causing the problem. From the macOS Monterey (have not upgraded yet) fcntl
man page:
F_BARRIERFSYNC
Does the same thing as fsync(2) then issues a barrier command to the drive (arg is
ignored). The barrier applies to I/O that have been flushed with fsync(2) on the
same device before. These operations are guaranteed to be persisted before any
other I/O that would follow the barrier, although no assumption should be made on
what has been persisted or not when this call returns. After the barrier has been
issued, operations on other FDs that have been fsync'd before can still be re-
ordered by the device, but not after the barrier. This is typically useful to
guarantee valid state on disk when ordering is a concern but durability is not. A
barrier can be used to order two phases of operations on a set of file descriptors
and ensure that no file can possibly get persisted with the effect of the second
phase without the effect of the first one. To do so, execute operations of phase
one, then fsync(2) each FD and issue a single barrier. Finally execute operations
of phase two. This is currently implemented on HFS and APFS. It requires hardware
support, which Apple SSDs are guaranteed to provide.
I cannot find this flag in the Go source code, so it looks like something that libSystem does when calling fsync. This would make it a macOS Ventura bug.
Go uses calls fcntl(fd.Sysfd, syscall.F_FULLFSYNC, 0)
following for fsync
:
F_BARRIERFSYNC would actually be too weak for the rest-server. My guess would be that exFAT implements neither.
Sorry, doesn’t seem to change things...
@bdillahu Can you provide the exact error message printed by rest-server when using #199? I wonder whether it still contains the filename or just the directory name.
Here's some example error:
Save(<lock/eb538ceb32>) returned error, retrying after 552.330144ms: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-2418844001: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.080381816s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1897201862: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.31013006s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1226160442: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.582392691s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1275874318: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 2.340488664s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-3964501932: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 4.506218855s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-3990782944: inappropriate ioctl for device
Upgraded working setup on Mac OS to Ventura 13.0 - now when rest-server tries to write files, I get "inapporpriate ioctl for device" - thought it might have been a disk permissions thing, but gave "Full Disk" rights.
Created a new repository and "init" fails when writing files - it creates the directories OK.
This is on an external drive (USB).
ERROR: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-enigma/locks/b4a7f249a92b886734753398e5ad261ec11ec47dac073cea349965e97a6328cd.rest-server-temp6129484611666145821: inappropriate ioctl for device Save(<lock/b4a7f249a9>) returned error, retrying after 1.31013006s: server response unexpected: 500 Internal Server Error (500)