Deno.copyFile triggers the file watcher infinitely

[oscarotero]

I have the following code:

const watcher = Deno.watchFs("./src");

for await (const event of watcher) {
    console.log(event);

    event.paths.forEach((path) => {
        Deno.copyFileSync(path, get_destination(path));
    });
}

Every time a file is modified, the watcher detects the change, so the file is copied to the destination folder. But seems like Deno.copyFileSync somehow triggers a new "modify" event to the original file in the watcher, so it's an infinite loop.

As a workaround, I can read and write the file content

const watcher = Deno.watchFs("./src");

for await (const event of watcher) {
    console.log(event);

    event.paths.forEach((path) => {
        Deno.writeFileSync(get_destination(path), Deno.readFileSync(path));
    });
}

I think this is a bug, because copyFile shouldn't modify the original file.

My Deno version:

deno 1.34.1 (release, x86_64-apple-darwin)
v8 11.5.150.2
typescript 5.0.4

[sigmaSd]

I can't reproduce, please provide a minimal example with reproduction steps

tested with

a.ts

const watcher = Deno.watchFs(".");

for await (const event of watcher) {
  console.log(event);

  event.paths.forEach((path) => {
    Deno.copyFileSync(path, "/dev/null"); // without this line I get the same events
  });
}

terminal 1 run: deno run -A a.ts
terminal 2 run: touch hello

result

[Object: null prototype] {
  kind: "create",
  paths: [ "/home/mrcool/dev/deno/lab/tmp/XRm/./hello" ],
  flag: null
}
[Object: null prototype] {
  kind: "modify",
  paths: [ "/home/mrcool/dev/deno/lab/tmp/XRm/./hello" ],
  flag: null
}
[Object: null prototype] {
  kind: "access",
  paths: [ "/home/mrcool/dev/deno/lab/tmp/XRm/./hello" ],
  flag: null
}

[oscarotero]

I have reproduced it in this zip file:

• Run deno run -A main.ts
• Copy the image from dest to src (override it).

watcher.zip

[sigmaSd]

My guess is this is due to the large file size, and maybe the os copying it in chunks and each chunk triggers a modify event (its not infinite loop, its just a lot of events)

I imagine you get the same things without deno, by using inotify directly in linux

[sigmaSd]

I tried inotify directly (with C) and its the same number of events, so it seems this is just how it works

[oscarotero]

Just clarify my OS is no Linux but macOS, and in my case it never stops (or at least it takes many seconds).

[sigmaSd]

you can try with this file (courtesy of chatgpt)

a.c

clang a.c
./a.out

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/inotify.h>

#define EVENT_SIZE  (sizeof(struct inotify_event))
#define BUF_LEN     (1024 * (EVENT_SIZE + 16))

int main() {
    int fd, wd, length;
    char buffer[BUF_LEN];

    fd = inotify_init();

    if (fd < 0) {
        perror("inotify_init");
        exit(EXIT_FAILURE);
    }

    wd = inotify_add_watch(fd, "./src", IN_ALL_EVENTS);

    if (wd < 0) {
        perror("inotify_add_watch");
        exit(EXIT_FAILURE);
    }

    printf("Watching current directory for events...\n");

    while (1) {
        length = read(fd, buffer, BUF_LEN);

        if (length < 0) {
            perror("read");
            exit(EXIT_FAILURE);
        }

        int i = 0;
        while (i < length) {
            struct inotify_event *event = (struct inotify_event *) &buffer[i];

            if (event->len) {
                if (event->mask & IN_CREATE)
                    printf("File/directory created: %s\n", event->name);

                if (event->mask & IN_DELETE)
                    printf("File/directory deleted: %s\n", event->name);

                if (event->mask & IN_MODIFY)
                    printf("File modified: %s\n", event->name);

                if (event->mask & IN_ATTRIB)
                    printf("File attributes modified: %s\n", event->name);

                if (event->mask & IN_MOVE)
                    printf("File/directory moved: %s\n", event->name);
            }

            i += EVENT_SIZE + event->len;
        }
    }

    inotify_rm_watch(fd, wd);
    close(fd);

    return 0;
}

[sigmaSd]

oh actually mac doesn't use inotify probably

[sigmaSd]

I can't test this but here is what chatgpt says for macos (you can alwyas fix the code if needed)

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/event.h>

int main() {
    int kq, dirfd, num_events, i;
    struct kevent event;
    struct timespec timeout = {0, 0};

    // Open the current directory
    dirfd = open("./src", O_RDONLY);
    if (dirfd == -1) {
        perror("open");
        exit(EXIT_FAILURE);
    }

    // Create a new kernel event queue
    kq = kqueue();
    if (kq == -1) {
        perror("kqueue");
        exit(EXIT_FAILURE);
    }

    // Register a new event in the event queue
    EV_SET(&event, dirfd, EVFILT_VNODE, EV_ADD | EV_ENABLE | EV_CLEAR, NOTE_WRITE | NOTE_DELETE | NOTE_RENAME, 0, NULL);

    // Attach the event to the event queue
    if (kevent(kq, &event, 1, NULL, 0, &timeout) == -1) {
        perror("kevent");
        exit(EXIT_FAILURE);
    }

    printf("Watching current directory for events...\n");

    while (1) {
        // Wait for events to occur
        num_events = kevent(kq, NULL, 0, &event, 1, &timeout);

        if (num_events == -1) {
            perror("kevent");
            exit(EXIT_FAILURE);
        } else if (num_events > 0) {
            // Process the event
            if (event.fflags & NOTE_WRITE)
                printf("File modified: %s\n", event.udata);

            if (event.fflags & NOTE_DELETE)
                printf("File/directory deleted: %s\n", event.udata);

            if (event.fflags & NOTE_RENAME)
                printf("File/directory renamed: %s\n", event.udata);
        }
    }

    close(kq);
    close(dirfd);

    return 0;
}

[oscarotero]

Thanks, but I'm not familiarized with C code.

I don't know if the bug is in the OS or Deno, but I think Deno should fix these issues and try to work the same in all operating systems. The way it works now is: on modify a file, copy it to other folder, which modify the file again, so copy the file again...

According to the documentation, the available event types are "any", "access", "create", "modify", "remove" and "other". Copying a file shouldn't modify the original file, but only access to it. This would allow to filter out the "access" events in the watcher and break the infinite loop.

[sigmaSd]

But it not modifying the original file, it's modifying the file being created (by writing to it in chunks)

Just to make sure , is what you get an infinite loop or a lot of events (as in does the logging stops after a while or doesn't?)

[oscarotero]

In this video, you can see what happens in my computer.

• The script watches changes in the src folder to copy the changes to dest.
• First, I move the file from dest to src (overriding the existing file in src) and it works fine.
• But if the file is copied instead of moved (using alt key) it starts an infinite loop of create/modify events.

https://github.com/denoland/deno/assets/377873/c791a202-9eb2-42b4-bc4a-8b03a9a9cdc0

[sigmaSd]

Seems to be a bug in the mac implementation @dsherret maybe you can give it a look (or at least I think issue should be labeled)

to reproduce

wget https://github.com/denoland/deno/files/11691835/watcher.zip

unzip watcher.zip

cd watcher

deno run -A main.ts

In another terminal

cp dest/neom-bhKqZNZeAR0-unsplash.jpg src/neom-bhKqZNZeAR0-unsplash.jpg

Now the terminal show an infinite loop with modify events, doesn't happen on linux, but happens on macos

[sigmaSd]

seems like Deno.watch is just a simple wrapper over notify crate https://github.com/denoland/deno/blob/e1be2bb80d2c9e7f56221d53349cdcea6d76ddde/runtime/ops/fs_events.rs#L96

which had deadlocks with macos before https://github.com/notify-rs/notify/issues/118

@oscarotero maybe you can give a shot to upgrading to the latest stable version https://github.com/denoland/deno/blob/main/Cargo.toml#L109 changing it to 6

[oscarotero]

@sigmaSd Thanks for your time researching on this issue. I'm glad to help but I don't know how do that (I'm not an engineer).