search

rudderlabs / rudder-server

Privacy and Security focused Segment-alternative, in Golang and React
https://www.rudderstack.com/
Other
4.09k stars 317 forks source link

Connection refused running control-plane-lite via docker-compose #2894

Closed mkraft closed 1 year ago

mkraft commented 1 year ago

I'm getting "connection refused" errors following the control-plane-lite Docker setup steps and any guidance would be much appreciated please.

rudder-docker.yml

version: "3.7"

services:

  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=rudder
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=jobsdb
    ports:
      - "6432:5432"

  backend:
    depends_on:
      - db
      - metrics-exporter
      - d-transformer
    image: rudderlabs/rudder-server:latest # latest version that matches 0.*.*
    entrypoint: sh -c '/wait-for db:5432 -- /rudder-server'
    ports:
      - "8080:8080"
    environment:
      - JOBS_DB_HOST=db
      - JOBS_DB_USER=rudder
      - JOBS_DB_PORT=5432
      - JOBS_DB_DB_NAME=jobsdb
      - JOBS_DB_PASSWORD=password
      - DEST_TRANSFORM_URL=http://d-transformer:9090
      - CONFIG_BACKEND_URL=https://api.rudderstack.com
      - WORKSPACE_TOKEN=<your_workspace_token> # Must be written without quotes e.g. WORKSPACE_TOKEN=20TokEO34NWitou2t3quO8teS7
      - STATSD_SERVER_URL=metrics-exporter:9125
      - RSERVER_BACKEND_CONFIG_CONFIG_FROM_FILE=true
      - RSERVER_BACKEND_CONFIG_CONFIG_JSONPATH=/etc/rudderstack/workspaceConfig.json
    volumes:
      - /Users/astrophage/Desktop/rudderindocker/workspaceConfig.json:/etc/rudderstack/workspaceConfig.json

  d-transformer:
    depends_on:
      - metrics-exporter
    image: rudderlabs/rudder-transformer:latest
    ports:
      - "9090:9090"
    environment:
      - STATSD_SERVER_HOST=metrics-exporter
      - STATSD_SERVER_PORT="9125"

  minio:
    image: minio/minio
    ports:
      - "9000:9000"
    command: server /data

  metrics-exporter:
    image: prom/statsd-exporter:v0.22.4
    ports:
      - "9102:9102"

workspaceConfig.json

{
    "source": {
        "config": {},
        "id": "1im8yfXQsocRBGcQLXAaq5M8dYV",
        "name": "test-JS",
        "writeKey": "1im8yk2sz8oyHVCGVqViSNfKVDF",
        "enabled": true,
        "sourceDefinitionId": "1TW48i2bIzEl1HPf825cEznfIM8",
        "deleted": false,
        "createdAt": "Mon Oct 12 2020 16:51:54 GMT+0530 (India Standard Time)",
        "updatedAt": "Mon Oct 12 2020 16:51:54 GMT+0530 (India Standard Time)",
        "sourceDefinition": {
            "id": "1TW48i2bIzEl1HPf825cEznfIM8",
            "name": "JavaScript",
            "displayName": "JavaScript",
            "category": null,
            "createdAt": "2019-11-12T12:39:19.885Z",
            "updatedAt": "2020-06-18T11:54:06.114Z"
        }
    },
    "metadata": {
        "version": "1.0.2"
    }
}

starting up with

docker-compose -f rudder-docker.yml up

logs

[+] Running 5/0
 ⠿ Container rudderindocker-minio-1             Created                                                                                                                                                                                                                        0.0s
 ⠿ Container rudderindocker-db-1                Created                                                                                                                                                                                                                        0.0s
 ⠿ Container rudderindocker-metrics-exporter-1  Created                                                                                                                                                                                                                        0.0s
 ⠿ Container rudderindocker-d-transformer-1     Created                                                                                                                                                                                                                        0.0s
 ⠿ Container rudderindocker-backend-1           Created                                                                                                                                                                                                                        0.0s
Attaching to rudderindocker-backend-1, rudderindocker-d-transformer-1, rudderindocker-db-1, rudderindocker-metrics-exporter-1, rudderindocker-minio-1
rudderindocker-metrics-exporter-1  | level=info ts=2023-01-21T11:31:44.670Z caller=main.go:291 msg="Starting StatsD -> Prometheus Exporter" version="(version=0.22.4, branch=HEAD, revision=7e2fe6c2e61552062f03c6a5dddc219512456832)"
rudderindocker-metrics-exporter-1  | level=info ts=2023-01-21T11:31:44.670Z caller=main.go:292 msg="Build context" context="(go=go1.17.3, user=root@f597e9e3e4dd, date=20211126-15:23:50)"
rudderindocker-metrics-exporter-1  | level=info ts=2023-01-21T11:31:44.670Z caller=main.go:341 msg="Accepting StatsD Traffic" udp=:9125 tcp=:9125 unixgram=
rudderindocker-metrics-exporter-1  | level=info ts=2023-01-21T11:31:44.670Z caller=main.go:342 msg="Accepting Prometheus Requests" addr=:9102
rudderindocker-db-1                |
rudderindocker-db-1                | PostgreSQL Database directory appears to contain a database; Skipping initialization
rudderindocker-db-1                |
rudderindocker-db-1                | 2023-01-21 11:31:44.697 UTC [1] LOG:  starting PostgreSQL 15.1 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
rudderindocker-db-1                | 2023-01-21 11:31:44.697 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
rudderindocker-db-1                | 2023-01-21 11:31:44.697 UTC [1] LOG:  listening on IPv6 address "::", port 5432
rudderindocker-db-1                | 2023-01-21 11:31:44.698 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
rudderindocker-db-1                | 2023-01-21 11:31:44.704 UTC [24] LOG:  database system was shut down at 2023-01-21 11:30:53 UTC
rudderindocker-db-1                | 2023-01-21 11:31:44.708 UTC [1] LOG:  database system is ready to accept connections
rudderindocker-minio-1             | WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
rudderindocker-minio-1             | MinIO Object Storage Server
rudderindocker-minio-1             | Copyright: 2015-2023 MinIO, Inc.
rudderindocker-minio-1             | License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
rudderindocker-minio-1             | Version: RELEASE.2023-01-20T02-05-44Z (go1.19.4 linux/arm64)
rudderindocker-minio-1             |
rudderindocker-minio-1             | Status:         1 Online, 0 Offline.
rudderindocker-minio-1             | API: http://172.21.0.2:9000  http://127.0.0.1:9000
rudderindocker-minio-1             | Console: http://172.21.0.2:40093 http://127.0.0.1:40093
rudderindocker-minio-1             |
rudderindocker-minio-1             | Documentation: https://min.io/docs/minio/linux/index.html
rudderindocker-minio-1             | Warning: The standard parity is set to 0. This can lead to data loss.
rudderindocker-backend-1           | 2023/01/21 11:31:45 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
rudderindocker-backend-1           | INFO: No .env file found.
rudderindocker-backend-1           | [Config] :: Failed to parse config file from path "./config/config.yaml", using default values: open ./config/config.yaml: no such file or directory
rudderindocker-backend-1           | 2023-01-21T11:31:45.630Z   WARN    event-schema    event-schema/event_schema.go:202    [EventSchemas] You are using default password. Please change it by setting env variable RUDDER_ADMIN_PASSWORD
rudderindocker-backend-1           | 2023-01-21T11:31:45.638Z   INFO    app app/app.go:55   Open source version of rudder-server
rudderindocker-backend-1           | 2023-01-21T11:31:45.663Z   ERROR   processor.transformer   transformer/transformer.go:177  Unable to make a transfomer build version call with error : Get "http://d-transformer:9090/transformerBuildVersion": dial tcp 172.21.0.5:9090: connect: connection refused
rudderindocker-backend-1           | 2023-01-21T11:31:45.667Z   ERROR   utils.misc  misc/misc.go:1165   Unable to make a warehouse databricks build version call with error : Get "http://localhost:8082/databricksVersion": dial tcp 127.0.0.1:8082: connect: connection refused
rudderindocker-backend-1           | 2023-01-21T11:31:45.748Z   INFO    runner  runner/runner.go:411    warehousemode embedded
rudderindocker-backend-1           | 2023-01-21T11:31:45.752Z   INFO    backend-config  backend-config/single_workspace.go:127  Read workspace config from JSON file
rudderindocker-backend-1           | 2023-01-21T11:31:45.753Z   INFO    backend-config  backend-config/backend-config.go:213    Workspace Config changed
rudderindocker-backend-1           | 2023-01-21T11:31:45.756Z   INFO    db.recovery db/embeddedRecovery.go:42   Starting in normal mode
rudderindocker-db-1                | 2023-01-21 11:31:45.779 UTC [30] FATAL:  terminating connection due to administrator command
rudderindocker-db-1                | 2023-01-21 11:31:45.779 UTC [29] FATAL:  terminating connection due to administrator command
rudderindocker-backend-1           | 2023-01-21T11:31:45.782Z   WARN    validators.envValidator validators/envValidator.go:181  Terminated 2 dangling connection(s)
rudderindocker-backend-1           | 2023-01-21T11:31:45.783Z   WARN    validators.envValidator validators/envValidator.go:183  dangling connection #1: {pid:29 queryStart:0xc00051d240 waitEventType:Client waitEvent:ClientRead state:idle query:SELECT pg_advisory_unlock($1) terminated:true}
rudderindocker-backend-1           | 2023-01-21T11:31:45.783Z   WARN    validators.envValidator validators/envValidator.go:183  dangling connection #2: {pid:30 queryStart:0xc00051d3e0 waitEventType:Client waitEvent:ClientRead state:idle query:INSERT INTO config_cache (key, config) VALUES ($1, $2)
rudderindocker-backend-1           |        on conflict (key)
rudderindocker-backend-1           |        do update set
rudderindocker-backend-1           |        config = $2,
rudderindocker-backend-1           |        updated_at = NOW() terminated:true}
rudderindocker-backend-1           | 2023-01-21T11:31:45.869Z   INFO    runner  runner/runner.go:411    warehousemode embedded
rudderindocker-backend-1           | 2023-01-21T11:31:45.869Z   INFO    admin   profiler/profiler.go:82 Starting server on port 7777
rudderindocker-backend-1           | 2023-01-21T11:31:45.869Z   INFO    apphandlers.EMBEDDED    apphandlers/embeddedAppHandler.go:91    Embedded mode: Starting Rudder Core
rudderindocker-backend-1           | 2023-01-21T11:31:45.874Z   INFO    warehouse   warehouse/warehouse.go:2040 WH: Starting Warehouse service...
rudderindocker-backend-1           | 2023-01-21T11:31:45.875Z   INFO    warehouse.pgnotifier    pgnotifier/pgnotifier.go:128    PgNotifier: Initializing PgNotifier...
rudderindocker-backend-1           | 2023-01-21T11:31:45.875Z   INFO    warehouse.pgnotifier    pgnotifier/pgnotifier.go:621    PgNotifier: Creating Job Queue Tables
rudderindocker-backend-1           | 2023-01-21T11:31:45.893Z   INFO    utils.misc  misc/misc.go:412    RUDDER_TMPDIR not found, falling back to /tmp
rudderindocker-backend-1           |
rudderindocker-backend-1           | 2023-01-21T11:31:45.893Z   INFO    admin   admin/admin.go:272  Serving on admin interface @ /tmp/rudder-server.sock
rudderindocker-backend-1           | 2023-01-21T11:31:45.908Z   INFO    jobsdb.readonly-gw  jobsdb/readonly_jobsdb.go:107   Readonly user connected to gw DB
rudderindocker-backend-1           | 2023-01-21T11:31:45.932Z   INFO    warehouse   warehouse/warehouse.go:2108 WH: Starting warehouse slave...
rudderindocker-backend-1           | 2023-01-21T11:31:45.935Z   INFO    jobsdb.readonly-rt  jobsdb/readonly_jobsdb.go:107   Readonly user connected to rt DB
rudderindocker-backend-1           | 2023-01-21T11:31:45.936Z   INFO    warehouse   warehouse/warehouse.go:2115 [WH]: Starting warehouse master...
rudderindocker-backend-1           | 2023-01-21T11:31:45.939Z   INFO    warehouse.pgnotifier    pgnotifier/pgnotifier.go:172    PgNotifier: Deleting all jobs for workspace: ::77bf0b886fd262a44b244144a05fb412
rudderindocker-backend-1           | 2023-01-21T11:31:45.954Z   INFO    warehouse   warehouse/warehouse.go:1921 WH: Warehouse master service waiting for BackendConfig before starting on 8082
rudderindocker-backend-1           | 2023-01-21T11:31:45.961Z   INFO    warehouse   warehouse/warehouse.go:1944 WH: Starting warehouse master service in 8082
rudderindocker-backend-1           | 2023-01-21T11:31:45.962Z   INFO    asyncjob    jobs/runner.go:120  [WH-Jobs]: Initializing async job runner
rudderindocker-backend-1           | 2023-01-21T11:31:45.962Z   INFO    asyncjob    jobs/runner.go:152  [WH-Jobs]: Cleaning up the zombie asyncjobs
rudderindocker-backend-1           | 2023-01-21T11:31:45.964Z   INFO    jobsdb.readonly-batch_rt    jobsdb/readonly_jobsdb.go:107   Readonly user connected to batch_rt DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.005Z   INFO    asyncjob    jobs/runner.go:129  [WH-Jobs]: successfully cleanup async table with error
rudderindocker-backend-1           | 2023-01-21T11:31:46.005Z   INFO    asyncjob    jobs/runner.go:171  [WH-Jobs]: Starting async job runner
rudderindocker-backend-1           | 2023-01-21T11:31:46.010Z   INFO    jobsdb.readonly-proc_error  jobsdb/readonly_jobsdb.go:107   Readonly user connected to proc_error DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.010Z   INFO    apphandlers.EMBEDDED    apphandlers/embeddedAppHandler.go:104   Configured deployment type: "DEDICATED"
rudderindocker-backend-1           | 2023-01-21T11:31:46.010Z   INFO    apphandlers.EMBEDDED    apphandlers/embeddedAppHandler.go:113   Clearing DB false
rudderindocker-db-1                | 2023-01-21 11:31:46.037 UTC [44] ERROR:  relation "rsources_failed_keys" already exists
rudderindocker-db-1                | 2023-01-21 11:31:46.037 UTC [44] STATEMENT:  create table "rsources_failed_keys" (
rudderindocker-db-1                |            id BIGSERIAL,
rudderindocker-db-1                |            db_name text not null default 'db',
rudderindocker-db-1                |            job_run_id text not null,
rudderindocker-db-1                |            task_run_id text not null,
rudderindocker-db-1                |            source_id text not null,
rudderindocker-db-1                |            destination_id text not null,
rudderindocker-db-1                |            record_id jsonb not null,
rudderindocker-db-1                |            ts timestamp not null default NOW(),
rudderindocker-db-1                |            primary key (job_run_id, task_run_id, source_id, destination_id, db_name, id)
rudderindocker-db-1                |        )
rudderindocker-db-1                | 2023-01-21 11:31:46.039 UTC [44] ERROR:  relation "rsources_stats" already exists
rudderindocker-db-1                | 2023-01-21 11:31:46.039 UTC [44] STATEMENT:  create table "rsources_stats" (
rudderindocker-db-1                |            db_name text not null default 'db',
rudderindocker-db-1                |            job_run_id text not null,
rudderindocker-db-1                |            task_run_id text not null,
rudderindocker-db-1                |            source_id text not null,
rudderindocker-db-1                |            destination_id text not null,
rudderindocker-db-1                |            in_count integer not null default 0,
rudderindocker-db-1                |            out_count integer not null default 0,
rudderindocker-db-1                |            failed_count integer not null default 0,
rudderindocker-db-1                |            ts timestamp not null default NOW(),
rudderindocker-db-1                |            primary key (db_name, job_run_id, task_run_id, source_id, destination_id)
rudderindocker-db-1                |        )
rudderindocker-backend-1           | 2023-01-21T11:31:46.059Z   INFO    jobsdb.gw   jobsdb/jobsdb.go:858    Connected to gw DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.145Z   INFO    jobsdb.rt   jobsdb/jobsdb.go:858    Connected to rt DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.169Z   ERROR   runner  runner/runner.go:258    error sending server features: non retriable: unexpected status code 404: Not Found
rudderindocker-backend-1           | 2023-01-21T11:31:46.248Z   INFO    jobsdb.batch_rt jobsdb/jobsdb.go:858    Connected to batch_rt DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.352Z   INFO    jobsdb.proc_error   jobsdb/jobsdb.go:858    Connected to proc_error DB
rudderindocker-d-transformer-1     |
rudderindocker-d-transformer-1     | > rudder-transformer@1.11.1 start /home/node/app
rudderindocker-d-transformer-1     | > node src/index.js
rudderindocker-d-transformer-1     |
rudderindocker-backend-1           | 2023-01-21T11:31:46.439Z   INFO    apphandlers.EMBEDDED    apphandlers/embeddedAppHandler.go:194   using Static Cluster Manager
rudderindocker-backend-1           | 2023-01-21T11:31:46.459Z   INFO    jobsdb.gw   jobsdb/jobsdb.go:858    Connected to gw DB
rudderindocker-backend-1           | 2023-01-21T11:31:46.554Z   INFO    enterprise.suppress-user    suppress-user/factory.go:30 Suppress User feature is enterprise only
rudderindocker-backend-1           | 2023-01-21T11:31:46.560Z   INFO    gateway gateway/gateway.go:1244 AdminHandler waiting for BackendConfig before starting on 8089
rudderindocker-backend-1           | 2023-01-21T11:31:46.561Z   INFO    gateway gateway/gateway.go:1170 WebHandler waiting for BackendConfig before starting on 8080
rudderindocker-backend-1           | 2023-01-21T11:31:46.561Z   INFO    gateway gateway/gateway.go:1172 WebHandler Starting on 8080
rudderindocker-backend-1           | 2023-01-21T11:31:46.561Z   INFO    gateway gateway/gateway.go:1246 AdminHandler starting on 8089
rudderindocker-backend-1           | 2023-01-21T11:31:46.562Z   WARN    rsources    rsources/handler.go:419 shared database is not configured, skipping logical replication monitoring
rudderindocker-backend-1           | 2023-01-21T11:31:46.563Z   INFO    cluster cluster/dynamic.go:110  Got trigger to change the mode, new mode: NORMAL, old mode: DEGRADED
rudderindocker-backend-1           | 2023-01-21T11:31:46.564Z   INFO    cluster cluster/dynamic.go:238  Transiting the server from DegradedMode to NormalMode
rudderindocker-backend-1           | 2023-01-21T11:31:46.564Z   INFO    cluster cluster/dynamic.go:150  Starting the server
rudderindocker-backend-1           | 2023-01-21T11:31:46.575Z   INFO    jobsdb.proc_error   jobsdb/backup.go:42 BackupDS loop is running
rudderindocker-backend-1           | 2023-01-21T11:31:46.585Z   INFO    jobsdb.gw   jobsdb/backup.go:42 BackupDS loop is running
rudderindocker-backend-1           | 2023-01-21T11:31:46.595Z   INFO    jobsdb.rt   jobsdb/backup.go:42 BackupDS loop is running
rudderindocker-backend-1           | 2023-01-21T11:31:46.605Z   INFO    jobsdb.batch_rt jobsdb/backup.go:42 BackupDS loop is running
rudderindocker-backend-1           | 2023-01-21T11:31:46.612Z   ERROR   processor   processor/processor.go:556  error sending request - %sGet "http://d-transformer:9090/features": dial tcp 172.21.0.5:9090: connect: connection refused
rudderindocker-backend-1           | 2023-01-21T11:31:46.614Z   INFO    processor   processor/processor.go:2402 Processor mainPipeline started, subJobSize=2000 pipelineBufferedItems=0
rudderindocker-backend-1           | 2023-01-21T11:31:46.616Z   INFO    processor.stash stash/stash.go:296  Processor errors stash loop started
rudderindocker-backend-1           | 2023-01-21T11:31:46.814Z   ERROR   processor   processor/processor.go:556  error sending request - %sGet "http://d-transformer:9090/features": dial tcp 172.21.0.5:9090: connect: connection refused
...

The backend container has network access to the transformer as I'm able to do

docker exec -it rudderindocker-backend-1 bash -c 'curl http://d-transformer:9090/transformerBuildVersion'
19012023.101129%
mkraft commented 1 year ago

@achettyiitr Do you happen to know if this is the best place to ask this question?

atzoum commented 1 year ago

Hey @mkraft, I wasn't able to reproduce your issue. Would it be possible to pull the latest versions of the docker images first, and then try again?

docker pull rudderlabs/rudder-transformer:latest 
docker pull rudderlabs/rudder-server:latest
gitcommitshow commented 1 year ago

As Control Plane Lite is deprecated now and it will fail with the latest rudder-server versions so I am going to close this issue for now. Likely the change needed will be at the Control Plane Lite side so there's no action for rudder-server team. If you plan to fix Control Plane Lite on your own and need support, feel free to ask in slack community.