Mac OS Ventura 13.0 - fails to write files

[bdillahu]

Upgraded working setup on Mac OS to Ventura 13.0 - now when rest-server tries to write files, I get "inapporpriate ioctl for device" - thought it might have been a disk permissions thing, but gave "Full Disk" rights.

Created a new repository and "init" fails when writing files - it creates the directories OK.

This is on an external drive (USB).

ERROR: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-enigma/locks/b4a7f249a92b886734753398e5ad261ec11ec47dac073cea349965e97a6328cd.rest-server-temp6129484611666145821: inappropriate ioctl for device Save(<lock/b4a7f249a9>) returned error, retrying after 1.31013006s: server response unexpected: 500 Internal Server Error (500)

[MichaelEischer]

That error is rather strange, apparently macOS now returns ENOTTY (inappropriate ioctl for device) when getting told to actually write the data to disk. Can you test whether https://github.com/restic/rest-server/pull/199 "fixes" (rather works around) the problem?

Which filesystem does the external drive use? Note that without support for fsync, you might loose data if the disk is unplugged or looses power unexpectedly.

[bdillahu]

Sorry, doesn’t seem to change things...

On Oct 29, 2022, at 17:03, Michael Eischer @.***> wrote:

That error is rather strange, apparently macOS now returns ENOTTY (inappropriate ioctl for device) when getting told to actually write the data to disk. Can you test whether #199 https://github.com/restic/rest-server/pull/199 "fixes" (rather works around) the problem?

Which filesystem does the external drive use? Note that without support for fsync, you'll likely loose data if the disk is unplugged or looses power unexpectedly.

— Reply to this email directly, view it on GitHub https://github.com/restic/rest-server/issues/198#issuecomment-1295974405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPWSRW675RBEP7AYQZLYODWFWGJVANCNFSM6AAAAAARPARB6A. You are receiving this because you authored the thread.

[wojas]

Can you try to run rest-server like sudo dtruss rest-server ... to get a trace of all the system calls?

[wojas]

Also, as Michael asked, what filesystem are you using on the external disk?

Is the problem reproducible when storing data to APFS on your main disk?

[bdillahu]

Sorry, drive filesystem is ExFat

Seems to run fine with a repository on my internal APFS (encrypted) drive.

I figured out that I can recreate it without rest-server involved - so I guess you guys are off the hook, but if anybody has any ideas, let me know :-)

✗ restic --repo "/Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry" snapshots enter password for repository: repository 65d1f988 opened (repository version 2) successfully, password is correct Save(<lock/e885a83278>) returned error, retrying after 552.330144ms: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2477433873: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.080381816s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-1565657784: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.31013006s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-1035312087: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 1.582392691s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2265441250: inappropriate ioctl for device Save(<lock/e885a83278>) returned error, retrying after 2.340488664s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/e885a832789389d06156f9f8901f3089b555abe52e2c8fdd9b1ec2fd8f743354-tmp-2386770608: inappropriate ioctl for device signal interrupt received, cleaning up

Tried the dtruss - not sure that got anything good, but…

✗ sudo dtruss restic --repo "/Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry" snapshots dtrace: system integrity protection is on, some features will not be available

SYSCALL(args) = return munmap(0x741D000, 0x98000) = 0 0 munmap(0x74B5000, 0x8000) = 0 0 munmap(0x74BD000, 0x4000) = 0 0 munmap(0x74C1000, 0x4000) = 0 0 munmap(0x74C5000, 0x50000) = 0 0 fsgetpath(0x7FF7BFEFE530, 0x400, 0x7FF7BFEFE518) = 43 0 fsgetpath(0x7FF7BFEFE530, 0x400, 0x7FF7BFEFE518) = 14 0 csrctl(0x0, 0x7FF7BFEFE93C, 0x4) = -1 1 __mac_syscall(0x7FF80E0EA08A, 0x2, 0x7FF7BFEFE860) = 0 0 csrctl(0x0, 0x7FF7BFEFE94C, 0x4) = -1 1 __mac_syscall(0x7FF80E0E7437, 0x5A, 0x7FF7BFEFE8E0) = 0 0 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 dtrace: error on enabled probe ID 1741 (ID 571: syscall::sysctl:return): invalid kernel access in action #10 at DIF offset 28 open("/\0", 0x20100000, 0x0) = 3 0 openat(0x3, "System/Cryptexes/OS\0", 0x100000, 0x0) = 4 0 dup(0x4, 0x0, 0x0) = 5 0 fstatat64(0x4, 0x7FF7BFEFD681, 0x7FF7BFEFDA80) = 0 0 openat(0x4, "System/Library/dyld/\0", 0x100000, 0x0) = 6 0 fcntl(0x6, 0x32, 0x7FF7BFEFD710) = 0 0 dup(0x6, 0x0, 0x0) = 7 0 unknown command "Data/Restic-Rest-Server/restic-backup-masonry" for "restic" dup(0x5, 0x0, 0x0) = 8 0 close(0x3) = 0 0 close(0x5) = 0 0 close(0x4) = 0 0 close(0x6) = 0 0

[wojas]

The dstruss output is not useful. Can you edit the message and remove it? It's very long.

The only similar issue I found is https://github.com/realm/realm-core/issues/5789

They mention F_BARRIERFSYNC as causing the problem. From the macOS Monterey (have not upgraded yet) fcntl man page:

 F_BARRIERFSYNC        
                            Does the same thing as fsync(2) then issues a barrier command to the drive (arg is
                            ignored).  The barrier applies to I/O that have been flushed with fsync(2) on the
                            same device before.  These operations are guaranteed to be persisted before any
                            other I/O that would follow the barrier, although no assumption should be made on
                            what has been persisted or not when this call returns.  After the barrier has been
                            issued, operations on other FDs that have been fsync'd before can still be re-
                            ordered by the device, but not after the barrier.  This is typically useful to
                            guarantee valid state on disk when ordering is a concern but durability is not.  A
                            barrier can be used to order two phases of operations on a set of file descriptors
                            and ensure that no file can possibly get persisted with the effect of the second
                            phase without the effect of the first one. To do so, execute operations of phase
                            one, then fsync(2) each FD and issue a single barrier.  Finally execute operations
                            of phase two.  This is currently implemented on HFS and APFS. It requires hardware
                            support, which Apple SSDs are guaranteed to provide.

I cannot find this flag in the Go source code, so it looks like something that libSystem does when calling fsync. This would make it a macOS Ventura bug.

[MichaelEischer]

Go uses calls fcntl(fd.Sysfd, syscall.F_FULLFSYNC, 0) following for fsync:

https://github.com/golang/go/blob/9dcc873413aa390bd9392d1ed6ae4198593e4c5d/src/internal/poll/fd_fsync_darwin.go#L18

F_BARRIERFSYNC would actually be too weak for the rest-server. My guess would be that exFAT implements neither.

[MichaelEischer]

Sorry, doesn’t seem to change things...

@bdillahu Can you provide the exact error message printed by rest-server when using #199? I wonder whether it still contains the filename or just the directory name.

[bdillahu]

Here's some example error:

Save(<lock/eb538ceb32>) returned error, retrying after 552.330144ms: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-2418844001: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.080381816s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1897201862: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.31013006s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1226160442: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 1.582392691s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-1275874318: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 2.340488664s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-3964501932: inappropriate ioctl for device Save(<lock/eb538ceb32>) returned error, retrying after 4.506218855s: sync /Volumes/Expansion/Backup Data/Restic-Rest-Server/restic-backup-masonry/locks/eb538ceb323f2cf380f002d8c89d81fd4bea7cabf1f87018b761c6c734f52ccc-tmp-3990782944: inappropriate ioctl for device