search

Baldinof / roadrunner-bundle

A RoadRunner worker integrated in your Symfony app
MIT License
270 stars 48 forks source link

vertex got an error #52

Open ntnchq opened 3 years ago

ntnchq commented 3 years ago

Sometimes we have error, and api response is 502, what can we do?

Error log:

web    | github.com/spiral/endure/pkg/container.(*Endure).poll.func1
web    |        github.com/spiral/endure@v1.0.2/pkg/container/poller.go:16
web    | error occurred: http_plugin_serve: WorkerAllocate:
web    |        server_plugin_new_worker_pool:
web    |        static_pool_initialize:
web    |        allocate workers: context deadline exceeded, plugin: http.Plugin
web    | handle_serve_command: http_plugin_serve: WorkerAllocate:
web    |        server_plugin_new_worker_pool:
web    |        static_pool_initialize:
web    |        allocate workers: context deadline exceeded; fsm_recognizer: can't transition from state: Stopped by event Stop

.rr.yaml

rpc:
  listen: tcp://127.0.0.1:6001

server:
  command: "php -dopcache.enable_cli=1 bin/console baldinof:roadrunner:worker"
  user: "www-data"
  group: "www-data"
  env:
    - APP_RUNTIME: Baldinof\RoadRunnerBundle\Runtime\Runtime
  relay: "tcp://localhost:7000"
  relay_timeout: 60s

logs:
  mode: production
  level: error
  encoding: json
  output: stderr
  err_output: stderr

http:
  address: 0.0.0.0:80
  max_request_size: 10
  middleware: [ "headers", "static", "gzip" ]
  trusted_subnets: [
      "10.0.0.0/8",
      "127.0.0.0/8",
      "172.16.0.0/12",
      "192.168.0.0/16",
      "::1/128",
      "fc00::/7",
      "fe80::/10",
  ]

  uploads:
    dir: "/tmp"
    forbid: [ ".php", ".exe", ".bat", ".sh" ]

  static:
    dir: "public"
    forbid: []
  pool:
    num_workers: 0
    max_jobs: 20
    allocate_timeout: 60s
    destroy_timeout: 60s
    supervisor:
      watch_tick: 1s
      ttl: 60s
      idle_ttl: 10s
      max_worker_memory: 256
      exec_ttl: 60s

  http2:
    h2c: false
    max_concurrent_streams: 15

endure:
  grace_period: 30s
  print_graph: false
  log_level: error

config/packages/baldinof_road_runner.yaml

baldinof_road_runner:
    kernel_reboot:
        strategy: on_exception
        allowed_exceptions:
            - Symfony\Component\HttpKernel\Exception\HttpExceptionInterface
            - Symfony\Component\Serializer\Exception\ExceptionInterface
            - Symfony\Contracts\HttpClient\Exception\ExceptionInterface

    metrics:
        enabled: false

Dockerfile

FROM core.harbor.k8s.devim.team/proxy/library/php:8-cli

ENV COMPOSER_ALLOW_SUPERUSER=1
ENV COMPOSER_CACHE_DIR=/var/www/.cache
ENV SUDO_FORCE_REMOVE=yes
WORKDIR /var/www/web
EXPOSE 80
ENTRYPOINT ["/entrypoint.sh"]

RUN usermod -u 1000 www-data \
    && groupmod -g 1000 www-data \
    && addgroup nobody www-data \
    && chown -R www-data:www-data /var/www

RUN apt-get update && apt-get install -y \
    sudo \
    curl \
    git \
    zip \
    wget \
    libzip-dev \
    libgmp-dev \
    libffi-dev \
    libssl-dev \
    gnupg2 \
    librabbitmq-dev \
    libicu-dev \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

RUN docker-php-ext-install -j$(nproc) \
    sockets \
    zip \
    gmp \
    pcntl \
    bcmath \
    ffi \
    intl \
    opcache \
    pdo \
    pdo_pgsql \
    pgsql

RUN docker-php-source extract \
    && mkdir /usr/src/php/ext/amqp \
    && curl -L https://github.com/php-amqp/php-amqp/archive/master.tar.gz | tar -xzC /usr/src/php/ext/amqp --strip-components=1 \
    && docker-php-ext-install amqp \
    && docker-php-ext-enable amqp

RUN pecl install \
    redis \
    && docker-php-ext-enable \
    redis

ADD docker/crypto/linux-amd64_deb.tgz /tmp
RUN bash /tmp/linux-amd64_deb/install.sh && rm -r /tmp/linux-amd64_deb

RUN curl -sS https://getcomposer.org/installer | php -- --version="2.0.8" --install-dir=/usr/local/bin --filename=composer
COPY --chown=www-data:www-data composer.* ./
RUN sudo -u www-data composer install --optimize-autoloader --no-interaction

COPY --chown=www-data:www-data . /var/www/web

RUN set -eux; \
    sudo -u www-data mkdir -p var/cache var/log; \
    sudo -u www-data composer dump-autoload --no-dev --classmap-authoritative; \
    sudo -u www-data composer dump-env prod --no-cache --profile; \
    sudo -u www-data php bin/console cache:clear

COPY docker/entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh; \
    apt-get remove sudo -y

RUN ./vendor/bin/rr get-binary --location /usr/local/bin

COPY docker/prod/php-cli/php.ini /usr/local/etc/php/php.ini
ntnchq commented 3 years ago

also logs in "debug" mode

web    | 2021-08-31T23:26:46.021Z       DEBUG   container/serve.go:20   called Serve on the vrtx        {"vrtx id": "reload.Plugin"}
web    | 2021-08-31T23:27:46.022Z       ERROR   container/poller.go:16  vertex got an error     {"vertex id": "http.Plugin", "error": "http_plugin_serve: WorkerAllocate:\n\tserver_plugin_new_worker_pool:\n\tstatic_pool_initialize:\n\tallocate workers: context deadline exceeded"}
web    | github.com/spiral/endure/pkg/container.(*Endure).poll.func1
web    |        github.com/spiral/endure@v1.0.2/pkg/container/poller.go:16
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/poller.go:49  processing error in the main thread     {"vertex id": "http.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:54  retry is turned off, sending exit signal to every vertex in the graph
web    | 2021-08-31T23:27:46.022Z       INFO    container/endure.go:441 exiting from the Endure
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "informer.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "gzip.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "config.Viper"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "server.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "config.Viper"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "resetter.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "redis.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "memory.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "http.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "reload.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "server.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "memory.Plugin"}
web    | 2021-08-31T23:27:46.022Z       DEBUG   container/stop.go:34    calling internal_stop function on the vrtx      {"vrtx id": "rpc.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "memory.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "redis.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "config.Viper"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "rpc.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "redis.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "http.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "server.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "logger.ZapLogger"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/poller.go:26  vertex got exit signal, exiting from poller     {"vertex id": "reload.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "http.Plugin"}
web    | 2021-08-31T23:27:46.022Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "rpc.Plugin"}
web    | error occurred: http_plugin_serve: WorkerAllocate:
web    |        server_plugin_new_worker_pool:
web    |        static_pool_initialize:
web    |        allocate workers: context deadline exceeded, plugin: http.Plugin
web    | 2021-08-31T23:27:46.095Z       INFO    container/stop.go:117   vertex stopped  {"vertex id": "reload.Plugin"}
web    | handle_serve_command: http_plugin_serve: WorkerAllocate:
web    |        server_plugin_new_worker_pool:
web    |        static_pool_initialize:
web    |        allocate workers: context deadline exceeded; fsm_recognizer: can't transition from state: Stopped by event Stop
ntnchq commented 3 years ago

resolved by replacing RUN ./vendor/bin/rr get-binary --location /usr/local/bin to static binary in my repository

but its strange...

Baldinof commented 3 years ago

Hi!

So Instead of running ./vendor/bin/rr get-binary --location /usr/local/bin you manually download the binary?

ntnchq commented 3 years ago

yes, and manual builded binary same good

patie commented 2 years ago

same problem probably here 🤔 

[2022-05-31 18:30:42] [INFO] RoadRunner server started; version: 2.10.2, buildtime: 2022-05-26T12:10:38+0000
[2022-05-31 18:31:42] 2022-05-31T18:31:42.808Z  ERROR   container/poller.go:16  vertex got an error     {"id": "http.Plugin", "error": "static_pool_allocate_workers: WorkerAllocate: context deadline exceeded"}
[2022-05-31 18:31:42] github.com/roadrunner-server/endure/pkg/container.(*Endure).poll.func1
[2022-05-31 18:31:42]   github.com/roadrunner-server/endure@v1.3.0/pkg/container/poller.go:16
[2022-05-31 18:31:43] error occurred: static_pool_allocate_workers: WorkerAllocate: context deadline exceeded, plugin: http.Plugin, stopping execution

EDIT: SOLVED, in my case there was error with redis cache in symfony - problem with connection, after resolve problem with connection on redis, problem is gone 🎉