Watcher not working in a docker container if user is not root

Hello, I am running a task with a watcher inside a docker container to reload te server when any file changes. While everything was working when the user was root, now that I change the user inside my dockerfile, the watcher doesn't seem to work anymore. Moreover, when nothing has changed and I run the container, the task says that everything is up to date and won't run the cmd. Here's my task:

  run-server:
    watch: true
    deps: [build]
    sources:
      - '**/*.go'
      - 'web/templates/**/*.templ'
      - exclude: 'web/templates/**/*_templ.go'
      - exclude: 'internal/pkg/repositories/*.gen.go'
      - exclude: 'internal/pkg/repositories/*.sql.go'
    cmds:
      - ./out/server

My dockerfile:

FROM golang:1.22.1-alpine

ARG USER_ID
ARG GROUP_ID

RUN apk add --no-cache curl git
#
# RUN addgroup --gid ${GROUP_ID} app &&\
#     adduser --uid ${USER_ID} --ingroup app --disabled-password app
#
# USER app

WORKDIR /app

RUN go install github.com/go-task/task/v3/cmd/task@latest

RUN git config --global --add safe.directory /app

# COPY --chown=app:app go.mod go.sum ./
COPY go.mod go.sum ./

RUN go mod download && go mod verify

# COPY --chown=app:app . .
COPY . .

EXPOSE 8080

CMD ["/go/bin/task", "-s", "run-server"]

And my docker compose

services:
  tt-app:
    container_name: tt-app
    build:
      context: .
      dockerfile: Dockerfile
      args:
          USER_ID: ${USER_ID:-0}
          GROUP_ID: ${GROUP_ID:-0}
    ports:
      - "8080:8080"
    volumes:
      - ./:/app

In this configuration, everything works fine, but as soon as I uncomment/replace the lines in my dockerfile, rebuild and run, the watcher doesn't work anymore. I need to be able to run the container as the same user as the host so when I generate files from the container I can edit them (without having to chown each file generated). Thank you!

Task version: v3.35.1 (h1:zjQ3tLv+LIStDDTzOQx8F97NE/8FSTanjZuwgy/hwro=)
Operating system: Arch Linux x86_64 / Kernel: 6.8.5-arch1-1
Experiments enabled: No

go-task / task

Watcher not working in a docker container if user is not root #1592