Symfony hanging every now & then

[toby-griffiths]

I'm using this Docker set up for a few of my projects and every now and then they just start locking up, and timing out.

I've done a little debugging and it looks like this this line in the \Symfony\Component\HttpKernel\Kernel::initializeContainer() method that's causing the problem…

        try {
            is_dir($buildDir) ?: mkdir($buildDir, 0777, true);

            if ($lock = fopen($cachePath.'.lock', 'w+')) {
                if (!flock($lock, \LOCK_EX | \LOCK_NB, $wouldBlock) && !flock($lock, $wouldBlock ? \LOCK_SH : \LOCK_EX)) {
                                                                      # ^^^ THIS flock() CALL HERE

I can guess that this is used to prevent 2 processes attempting to update the container at the same time, but I don't know how to determine the cause of the lock not being released, and whether this is a Symfony issue, or a Docker (for Mac) issue.

Here's my Docker info…

Client:
 Version:    26.1.4
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.1-desktop.1
    Path:     /Users/toby/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.1-desktop.1
    Path:     /Users/toby/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/toby/.docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/toby/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.24
    Path:     /Users/toby/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/toby/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/toby/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/toby/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.9.3
    Path:     /Users/toby/.docker/cli-plugins/docker-scout

Server:
 Containers: 27
  Running: 19
  Paused: 0
  Stopped: 8
 Images: 84
 Server Version: 26.1.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d2d58213f83a351ca8f528a95fbd145f5654e957
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.31-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.657GiB
 Name: docker-desktop
 ID: a4d0d51e-5982-4409-b1de-b5a2437a1e5b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/toby/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

… and my Docker compose config…

name: myproject
services:
  blackfire:
    environment:
      BLACKFIRE_CLIENT_ID: REDACTED
      BLACKFIRE_CLIENT_TOKEN: REDACTED
      BLACKFIRE_SERVER_ID: REDACTED
      BLACKFIRE_SERVER_TOKEN: REDACTED
    image: blackfire/blackfire:2
    networks:
      default: null
  database:
    environment:
      POSTGRES_DB: app
      POSTGRES_PASSWORD: '!ChangeMe!'
      POSTGRES_USER: app
    image: postgres:16-alpine
    networks:
      default: null
    ports:
      - mode: ingress
        target: 5432
        published: "54660"
        protocol: tcp
    volumes:
      - type: volume
        source: database_data
        target: /var/lib/postgresql/data
        volume: {}
  mailer:
    environment:
      MP_SMTP_AUTH_ACCEPT_ANY: "1"
      MP_SMTP_AUTH_ALLOW_INSECURE: "1"
    image: axllent/mailpit
    labels:
      traefik.docker.network: traefik
      traefik.enable: "true"
      traefik.http.routers.myproject-mail.entrypoints: https
      traefik.http.routers.myproject-mail.rule: Host(`mail.myproject.localhost`)
      traefik.http.routers.myproject-mail.tls: "true"
      traefik.http.services.myproject-mail.loadbalancer.server.port: "8025"
    networks:
      default: null
      traefik: null
  php:
    build:
      context: .
      dockerfile: Dockerfile
      target: frankenphp_dev
    environment:
      DATABASE_URL: postgresql://app:!ChangeMe!@database:5432/app?serverVersion=15&charset=utf8
      MERCURE_EXTRA_DIRECTIVES: demo
      MERCURE_JWT_SECRET: '!ChangeThisMercureHubJWTSecretKey!'
      MERCURE_PUBLIC_URL: https://localhost/.well-known/mercure
      MERCURE_PUBLISHER_JWT_KEY: '!ChangeThisMercureHubJWTSecretKey!'
      MERCURE_SUBSCRIBER_JWT_KEY: '!ChangeThisMercureHubJWTSecretKey!'
      MERCURE_URL: http://php/.well-known/mercure
      PHP_IDE_CONFIG: serverName=myproject.localhost
      SERVER_NAME: myproject.localhost:80 myproject.local:80
      STABILITY: stable
      SYMFONY_VERSION: ""
      TRUSTED_HOSTS: ^example\.com|localhost|php$$
      TRUSTED_PROXIES: 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
      XDEBUG_CONFIG: start_with_request=trigger client_host=host.docker.internal
      XDEBUG_MODE: "off"
    extra_hosts:
      - host.docker.internal=host-gateway
    image: app-php
    labels:
      traefik.docker.network: traefik
      traefik.enable: "true"
      traefik.http.routers.myproject.entrypoints: https
      traefik.http.routers.myproject.rule: Host(`myproject.localhost`)
      traefik.http.routers.myproject.tls: "true"
      traefik.http.services.myproject.loadbalancer.server.port: "80"
      traefik.udp.routers.myproject.entrypoints: http3
      traefik.udp.services.myproject.loadbalancer.server.port: "443"
    networks:
      default: null
      traefik:
        aliases:
          - myproject.local
    restart: unless-stopped
    tty: true
    volumes:
      - type: volume
        source: caddy_data
        target: /data
        volume: {}
      - type: volume
        source: caddy_config
        target: /config
        volume: {}
      - type: bind
        source: .
        target: /app
        bind:
          create_host_path: true
      - type: volume
        target: /app/var
        volume: {}
      - type: volume
        source: tailwind_var
        target: /app/var/tailwind
        volume: {}
      - type: bind
        source: ./frankenphp/Caddyfile
        target: /etc/caddy/Caddyfile
        read_only: true
        bind:
          create_host_path: true
      - type: bind
        source: ./frankenphp/conf.d/app.dev.ini
        target: /usr/local/etc/php/conf.d/app.dev.ini
        read_only: true
        bind:
          create_host_path: true
  tailwind:
    build:
      context: .
      dockerfile: Dockerfile
      target: frankenphp_dev
    command:
      - bin/console
      - tailwind:build
      - --watch
    environment:
      MERCURE_EXTRA_DIRECTIVES: demo
      PHP_IDE_CONFIG: serverName=myproject.localhost
      SERVER_NAME: myproject.localhost:80 myproject.local:80
      XDEBUG_CONFIG: start_with_request=trigger client_host=host.docker.internal
      XDEBUG_MODE: "off"
    networks:
      default: null
    volumes:
      - type: bind
        source: .
        target: /app
        bind:
          create_host_path: true
      - type: volume
        source: tailwind_var
        target: /app/var/tailwind
        volume: {}
      - type: bind
        source: ./frankenphp/conf.d/app.dev.ini
        target: /usr/local/etc/php/conf.d/app.dev.ini
        read_only: true
        bind:
          create_host_path: true
networks:
  default: ~
  traefik:
    name: traefik
    external: true
volumes:
  caddy_config:
  caddy_data:
  database_data:
  tailwind_var:

Could someone advise how I might diagnose the issue here?

Please let me know if you need more info.

[toby-griffiths]

It's just happened again now, and this is the output of lslocks…

docker compose exec php lslocks
COMMAND         PID   TYPE SIZE MODE  M START END PATH
frankenphp        1  FLOCK  16K WRITE 0     0   0 /data/mercure.db
(undefined)      -1 OFDLCK      READ  0     0   0
(undefined)      -1 OFDLCK      READ  0     0   0
frankenphp        1  POSIX      READ  0     1   1 /memfd:opcache_lock (deleted)
frankenphp        1  FLOCK      READ* 0     0   0 /app/var/cache/dev/App_KernelDevDebugContainer.php.lock
frankenphp        1  FLOCK      WRITE 0     0   0 /app/var/cache/dev/App_KernelDevDebugContainer.php.lock

… in case it helps.

[toby-griffiths]

Should I perhaps move this issue to https://github.com/dunglas/frankenphp?

[aratinau]

Same here In the meantime, I have rolled back the PHP version in the Dockerfile to 8.3.7

[dunglas]

FrankenPHP issue: https://github.com/dunglas/frankenphp/issues/886

@aratinau are you using Blackfire?

[aratinau]

I don't use blackfire But it's only after a project modification that there's a problem. If I down and up the project, everything works again and the modification is taken into account.

[dunglas]

Thanks! Could you tell me the versions of FrankenPHP you use please (the one that works and the one that doesn't)?

[aratinau]

Sure!

The version that works FROM dunglas/frankenphp:1-php8.3.7 AS frankenphp_upstream

The version that doesn't work (but it was 8.3.8) FROM dunglas/frankenphp:1-php8.3 AS frankenphp_upstream

[dunglas]

Thanks. Could you check the exact version of FrankenPHP itself please? I'm trying to figure if it's a problem introduced by FrankenPHP or by PHP.

[aratinau]

Ok

The version that works FrankenPHP v1.2.0 PHP 8.3.7 Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

The version that doesn't work FrankenPHP v1.2.1 PHP 8.3.8 Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

[toby-griffiths]

@dunglas I may be able to get Blackfire working to profile, if that helps? Although I see you've linked to https://github.com/docker/for-mac/issues/7004 in https://github.com/dunglas/frankenphp/issues/886, so is this still helpful?

[dunglas]

@aratinau thanks! I can see what could cause the issue in 1.2.1 (https://github.com/dunglas/frankenphp/compare/v1.2.0...v1.2.1), so it's likely a PHP issue (https://github.com/php/php-src/compare/php-8.3.7...php-8.3.8).

Would you mind reporting the problem to PHP, please?

@toby-griffiths no, as it doesn't look related to Blackfire.

Could you try to not share the cache directory as a volume (to not use VirtioFS) to see if this works as a workaround?

[dunglas]

This seems to be this PHP bug: https://github.com/php/php-src/issues/14592

[toby-griffiths]

@toby-griffiths no, as it doesn't look related to Blackfire.

Could you try to not share the cache directory as a volume (to not use VirtioFS) to see if this works as a workaround?

@dunglas I already have the following in my compose.override.yaml file…

# …
services:
  php:
    # …
    volumes:
      - ./:/app
      - /app/var

… and I'm still experiencing the issue.

[dunglas]

Could someone does that please? https://github.com/php/php-src/issues/14592#issuecomment-2192553039

[dunglas]

I don't manage to reproduce the issue (latest version of Docker for Mac, FrankenPHP, and PHP).

Could someone provide me with a minimal reproducer (a Git repo would be perfect)? Thanks.

[toby-griffiths]

@dunglas I might be able to next week, but looking at this comment it looks like the issue might be with Docker for Mac, or MacOS, perhaps?

[sjonkeesse]

To reproduce on a Mac:

1. Clone https://github.com/dunglas/symfony-docker
2. Instal API Platform
docker exec -it symfony-docker-php-1 composer require api
3. Restart docker
4. Instal MakerBundle
docker exec -it symfony-docker-php-1 composer require --dev symfony/maker-bundle
5. Create an entity
docker exec -it symfony-docker-php-1 bin/console make:entity
 ApiResource: yes name: Address
 new property: label
 type: string
 length: 255
 nullable: yes
6. Update database
docker exec -it symfony-docker-php-1 bin/console doctrine:schema:update --force
7. Check localhost/api -> should still work
8. Add (any) filter to the entity
#[ApiFilter(SearchFilter::class, properties: ["label" => "partial"])]
9. Flock issue..
10. Restart docker
11. Check localhost/api -> works again
12. Change a project file
13. Flock issue...

https://github.com/sjonkeesse/flock