Add a timeout on opening/reading from UNIX pipes when --read-special option is active

[fallen]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

Question / Suggestion

System information. For client/server mode post info for both machines.

This question is linked to my previous (closed) ticket: #5420

To prevent people from going down the same road/issue I did, it might be interesting to add a timeout mechanism for when borg is in --read-special mode and it is opening/reading from a named UNIX pipe. Since this can block, it could help to quickly see:

1. that there is an issue
2. what the issue is

I don't know whether this could also be a good idea for character devices ... Or maybe just show a warning like "reading from character device XXX, are you sure this is what you wanted to do"?

The timeout, when hit, would print an error message and bail out. Timeout could be if open() blocks for N minutes, and also if N minutes pass without read() returning. This timeout could also be for instance configurable on cmdline.

[ThomasWaldmann]

Maybe that needs to be configurable? I can imagine that the other process on the other end of the pipe (legitimately) takes some time before producing output borg can read.

[fallen]

Maybe that needs to be configurable? I can imagine that the other process on the other end of the pipe (legitimately) takes some time before producing output borg can read.

Yes, indeed that's a good idea.

[enkore]

SIGALRM should work for this, setting it before the read, and resetting it immediately afterwards. Alternatively, non-blocking mode, which does avoid blocking when opening an unestablished pipe, at least on Linux..

[Gu1nness]

Hum even though SIGALRM seems appealing, I'm not sure this solution is universal enough: There can be many places where we might want to have a timeout IMHO and SIGALRM does not exists on WIndows. So I would rather go for opening a thread that just waits the right amount of time before exiting, and joining this thread. When the thread joins, we have reached the timeout and hence we can raise an exception. What do you think ?

[enkore]

Joining a thread is a blocking operation in itself that you can't use to interrupt a blocking syscall.

[fallen]

If on a libc that supports it, you can do pthread_cancel() on the thread. open() and read() are listed in man 7 pthreads as required to be cancellation points by POSIX.1-2001 and/or POSIX.1-2008. I don't know about Windows support for pthreads and their cancellations.

[enkore]

I see what you mean now. That should work, though the interaction of thread cancellation and the CPython runtime may be hairy (probably easier to side-step that can of worms by writing the thread in C and not touching anything Python from it). From what I've heard this is a somewhat hairy corner of POSIX, though it may be manageable for a simple, single-purpose thread.

[fallen]

Another track maybe worth a try:

#!/usr/bin/env python3

import os
import time

def main():
    fd = os.open("myfifo", os.O_RDONLY | os.O_NONBLOCK)
    os.set_blocking(fd, True) # so that os.read() does not raise exception when read is starving because write end would be too slow
    buffer = os.read(fd, 1024)
    while buffer == b'': # first loop is to wait for writer to open the fifo
        buffer = os.read(fd, 1024)
        time.sleep(1)
    print("buffer: {}".format(buffer))
    # handle(buffer) , do something with the first data we got

    while buffer != b'':
        buffer = os.read(fd, 1024)
        print("buffer: {}".format(buffer))
        # handle(buffer) , do something with the data

if __name__ == "__main__":
    main()

if I create a fifo in one terminal: mkfifo myfifo then run the above script in another one, it seems to work OK. i.e. I could run cat /dev/random > myfifo (which does block a lot since it wants truly random data) or cat /dev/urandom > myfifo (which never blocks and sends looots of data) in the first terminal. The first loop could have any kind of timeout since it never blocks. This could run in main thread.

[fallen]

Anyone gave a try to the last idea I posted?

[ThomasWaldmann]

Not yet. Can somebody say something about compatiblity of this? linux, bsd, macOS, openindiana, windows?