Redash server exited with code 139 with docker

[andresorrego94]

Hi all

I am facing issues trying to run docker-compose up -d I tried from the following branches versions : master and release/10.0.x

Error: redash_server_1 exited with code 139

I tried from ubuntu 20.04, Mac BigSur, and windows 10. Does anyone facing the same issue ? There is no more details in the error than the exited code 139. Also I run the create db commands but after that the issue still there with the server.

Need help to solve ths issue. Thanks !

[MathiasLantean]

Hi @andresorrego94!

I've had a hard time installing redash or running it using docker. Currently, I have the same issue, running the image in macOS Big Sur 11.4 using docker engine 20.10.8 and docker compose 1.29.2.

ml@192 redash % docker-compose up        
Starting redash_redis_1    ... done
Starting redash_email_1    ... done
Starting redash_postgres_1 ... done
Starting redash_server_1   ... done
Starting redash_worker_1    ... done
Starting redash_scheduler_1 ... done
Attaching to redash_postgres_1, redash_redis_1, redash_email_1, redash_server_1, redash_scheduler_1, redash_worker_1
email_1      | MailDev webapp running at http://0.0.0.0:80
email_1      | MailDev SMTP Server running at 0.0.0.0:25
postgres_1   | 
postgres_1   | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1   | 
postgres_1   | LOG:  database system was interrupted; last known up at 2021-09-19 21:08:11 UTC
postgres_1   | LOG:  database system was not properly shut down; automatic recovery in progress
postgres_1   | LOG:  invalid record length at 0/1701F70
postgres_1   | LOG:  redo is not required
postgres_1   | LOG:  MultiXact member wraparound protections are now enabled
postgres_1   | LOG:  database system is ready to accept connections
postgres_1   | LOG:  autovacuum launcher started
scheduler_1  | Starting dev RQ scheduler...
redis_1      | 1:C 19 Sep 21:23:41.039 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1      |                 _._                                                  
redis_1      |            _.-``__ ''-._                                             
redis_1      |       _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
redis_1      |   .-`` .-```.  ```\/    _.,_ ''-._                                   
redis_1      |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1      |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1      |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1      |   `-._    `-._  `-./  _.-'    _.-'                                   
redis_1      |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1      |  |    `-._`-._        _.-'_.-'    |           http://redis.io        
redis_1      |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1      |  |`-._`-._    `-.__.-'    _.-'_.-'|                                  
redis_1      |  |    `-._`-._        _.-'_.-'    |                                  
redis_1      |   `-._    `-._`-.__.-'_.-'    _.-'                                   
redis_1      |       `-._    `-.__.-'    _.-'                                       
redis_1      |           `-._        _.-'                                           
redis_1      |               `-.__.-'                                               
redis_1      | 
worker_1     | Starting dev RQ worker...
redis_1      | 1:M 19 Sep 21:23:41.041 # Server started, Redis version 3.2.12
redis_1      | 1:M 19 Sep 21:23:41.041 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1      | 1:M 19 Sep 21:23:41.041 * The server is now ready to accept connections on port 6379
redash_server_1 exited with code 139

Please, if you find any solution, comment it here. It's almost impossible to install Redash in a local environment. The only way I see to have a Redash instance working is to fix the Docker issue.

Thanks!

[susodapop]

I think the problem is our image not your environment. The release/10.x.x branch hasn't incorporated https://github.com/getredash/redash/pull/5570 yet, so it won't build. This will be fixed in a couple days as we prepare the V10 final release.

[schoettler]

I'm having this issue, even with #5570 incorporated. 100% replicable.

Worth noting I could run docker-compose up since June, and this error started appearing just a couple days ago, after removing all images & images cache (with docker system prune -af).

[susodapop]

Thanks @schoettler. Will see if I can reproduce here.

[schoettler]

@susodapop updated above message to include detail about image & cache removal for replication.

I checked the source, betting on a dependency having a patched update breaking the build, to no avail. pip also outputs:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

httplib2 0.19.1 requires pyparsing<3,>=2.4.2, but you'll have pyparsing 2.1.4 which is incompatible.
botocore 1.13.50 requires urllib3<1.26,>=1.20; python_version >= "3.4", but you'll have urllib3 1.26.7 which is incompatible.
dql 0.5.26 requires python-dateutil<2.7.0, but you'll have python-dateutil 2.8.2 which is incompatible.
memsql 3.0.0 requires mysqlclient==1.3.13, but you'll have mysqlclient 1.3.14 which is incompatible.
pymapd 0.19.0 requires thrift==0.11.0, but you'll have thrift 0.13.0 which is incompatible.
pyopenssl 20.0.1 requires cryptography>=3.2, but you'll have cryptography 2.9.2 which is incompatible.
snowflake-connector-python 2.1.3 requires certifi<2021.0.0, but you'll have certifi 2021.5.30 which is incompatible.
snowflake-connector-python 2.1.3 requires pytz<2021.0, but you'll have pytz 2021.1 which is incompatible.
snowflake-connector-python 2.1.3 requires requests<2.23.0, but you'll have requests 2.26.0 which is incompatible.
snowflake-connector-python 2.1.3 requires urllib3<1.26.0,>=1.20, but you'll have urllib3 1.26.7 which is incompatible.

Not sure if it can be related.

[schoettler]

Were you able to find anything? @susodapop

[susodapop]

Hi @schoettler I'm not able to reproduce this. From a completely bare system I can build master on macOS Catalina. I wonder if there is an issue with Docker in Big Sur?

[jbreiding]

I got passed this by adding this environment override REDASH_ENABLED_QUERY_RUNNERS: "redash.query_runner.mysql,redash.query_runner.pg"

[YuanHongde]

I have the same issue too. When I execute docker-compose run --rm server create_db on macOS Big Sur, I got the error:

Creating redash_server_run ... done
ERROR: 139

[lpong]

Has this isuess solved?

[susodapop]

@lea21st @YuanHongde We on the core team cannot reproduce this behavior. master builds fine for us -- plus our CircleCI builds are fine. So this is probably an issue on your system. Will either of you provide more information about your environment?

Offhand I wonder if you're trying this on an Apple computer with an M1 processor?

[emilianosantucci]

@susodapop i'm link to this issue for the same error.

My working machine as O.S. Windows 10 but the build is launched on WSL2 Debian GNU/Linux 10.

I've try:

• Build with base docker-compose configuration -> FAILED
• Run with Docker Hub built images and volumes attached on -> FAILED
• Run with Docker Hub built images without volumes attached on -> OK

All the errors occur only on the server service.

[susodapop]

Thanks @emilianosantucci . We don't test Redash on WSL (maybe we should?). Do you have any feedback on what step causes the error? Any stack traces? Error messages?

[edit] Consult this SO question for resources to debug further. You're facing an issue with Docker, not with Redash AFAICT

[jbreiding]

Thanks @emilianosantucci . We don't test Redash on WSL (maybe we should?). Do you have any feedback on what step causes the error? Any stack traces? Error messages?

[edit] Consult this SO question for resources to debug further. You're facing an issue with Docker, not with Redash AFAICT

In my instance this was not true. There was log output that stopped after the snowflake provider. I was running on a Mac (2021), not M1.

I was able to move passed it by enabling only the providers I knew I needed. This was building locally from master, not using dockerhub.

Unfortunately no amount of verbose logging gave any details into the error, but reducing the providers did move passed it.

[emilianosantucci]

I've tried to debug it without too much success, anyway this is my inspect:

{
    "Id": "b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00",
    "Created": "2021-09-27T16:42:14.8211196Z",
    "Path": "/app/bin/docker-entrypoint",
    "Args": [
        "dev_server"
    ],
    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 139,
        "Error": "",
        "StartedAt": "2021-09-27T16:42:21.1461658Z",
        "FinishedAt": "2021-09-27T16:42:24.6585156Z"
    },
    "Image": "sha256:1c3d39cc83861ce23476bedbc573dddba1956d0ff8da8c7e646666b36f51e578",
    "ResolvConfPath": "/var/lib/docker/containers/b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00/hostname",
    "HostsPath": "/var/lib/docker/containers/b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00/hosts",
    "LogPath": "/var/lib/docker/containers/b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00/b76257886e4443d932df5c3d2175fbe810454171089e19de8e4306c123a9cf00-json.log",
    "Name": "/redash_server_1",
    "RestartCount": 0,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": [
            "/home/emiliano/workspaces/redash:/app:rw"
        ],
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "NetworkMode": "redash_default",
        "PortBindings": {
            "5000/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "5000"
                }
            ],
            "5678/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "5678"
                }
            ]
        },
        "RestartPolicy": {
            "Name": "",
            "MaximumRetryCount": 0
        },
        "AutoRemove": false,
        "VolumeDriver": "",
        "VolumesFrom": [],
        "CapAdd": null,
        "CapDrop": null,
        "CgroupnsMode": "host",
        "Dns": null,
        "DnsOptions": null,
        "DnsSearch": null,
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "private",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": null,
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": null,
        "DeviceCgroupRules": null,
        "DeviceRequests": null,
        "KernelMemory": 0,
        "KernelMemoryTCP": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": null,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/asound",
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/8ca85adf0650187c6fb360211ff6e46fffae869ec0e0eafb5dd8bf0bb4cdd1d7-init/diff:/var/lib/docker/overlay2/ln28trbbk774k76numd7ybk0h/diff:/var/lib/docker/overlay2/m7f6bjbky3ibths9w8eo35w95/diff:/var/lib/docker/overlay2/54yobwldmzy2rhotzkzt9gman/diff:/var/lib/docker/overlay2/ugxmhxaqzyimxpgzzzsxg9qf9/diff:/var/lib/docker/overlay2/yvoca82ha3uq5au3gpqk0jtn5/diff:/var/lib/docker/overlay2/q29jr419bw4d28ydeqxpi4f8h/diff:/var/lib/docker/overlay2/z6tp57ue0pq5uuebkwzwjzbt3/diff:/var/lib/docker/overlay2/d9i6fqfe0sahjxgzn235tgyq5/diff:/var/lib/docker/overlay2/i4cdidrv2t27jnb1qc4y5u1c3/diff:/var/lib/docker/overlay2/jz2rsucv4861o55k301z28588/diff:/var/lib/docker/overlay2/er8ac4hawm3zc69xx7ibb15c6/diff:/var/lib/docker/overlay2/yg3qyckpg3s5dm5so328tox15/diff:/var/lib/docker/overlay2/dp7l5iljhiwl5qu2na2bvm8ai/diff:/var/lib/docker/overlay2/nkjbk9vpuh78j59d3qhh04lv9/diff:/var/lib/docker/overlay2/16bad7c9298740ba215f491a567096face32bcf628b73ec54fb7804a1eb70ebe/diff:/var/lib/docker/overlay2/f90a0bd23ae22807a7d445a6bd74500b7fdd4f633e1645959a468b18f8f9a299/diff:/var/lib/docker/overlay2/fb7d2acc6f933a4031bd7c1d7b1816612ff83f7291acda55e5d8205c1ede92df/diff:/var/lib/docker/overlay2/2a2ad7df19cd57af09af8febe17952ad8c8acbf40528deedc99eb114cdd3f8c7/diff:/var/lib/docker/overlay2/f37872cf14ff3242b61b3143b62ff8eeec69ad37c1a438f0dfbc9fb19c1dc62c/diff",
            "MergedDir": "/var/lib/docker/overlay2/8ca85adf0650187c6fb360211ff6e46fffae869ec0e0eafb5dd8bf0bb4cdd1d7/merged",
            "UpperDir": "/var/lib/docker/overlay2/8ca85adf0650187c6fb360211ff6e46fffae869ec0e0eafb5dd8bf0bb4cdd1d7/diff",
            "WorkDir": "/var/lib/docker/overlay2/8ca85adf0650187c6fb360211ff6e46fffae869ec0e0eafb5dd8bf0bb4cdd1d7/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [
        {
            "Type": "bind",
            "Source": "/home/emiliano/workspaces/redash",
            "Destination": "/app",
            "Mode": "rw",
            "RW": true,
            "Propagation": "rprivate"
        }
    ],
    "Config": {
        "Hostname": "b76257886e44",
        "Domainname": "",
        "User": "redash",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "ExposedPorts": {
            "5000/tcp": {},
            "5678/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "REDASH_MAIL_SERVER=email",
            "REDASH_ENFORCE_CSRF=true",
            "PYTHONUNBUFFERED=0",
            "REDASH_LOG_LEVEL=INFO",
            "REDASH_REDIS_URL=redis://redis:6379/0",
            "REDASH_DATABASE_URL=postgresql://postgres@postgres/postgres",
            "REDASH_RATELIMIT_ENABLED=false",
            "REDASH_MAIL_DEFAULT_SENDER=redash@example.com",
            "PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "LANG=C.UTF-8",
            "GPG_KEY=0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D",
            "PYTHON_VERSION=3.7.12",
            "PYTHON_PIP_VERSION=21.2.4",
            "PYTHON_SETUPTOOLS_VERSION=57.5.0",
            "PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/c20b0cfd643cd4a19246ccf204e2997af70f6b21/public/get-pip.py",
            "PYTHON_GET_PIP_SHA256=fa6f3fb93cce234cd4e8dd2beb54a51ab9c247653b52855a48dd44e6b21ff28b",
            "PIP_DISABLE_PIP_VERSION_CHECK=1",
            "PIP_NO_CACHE_DIR=1"
        ],
        "Cmd": [
            "dev_server"
        ],
        "Image": "redash_server",
        "Volumes": {
            "/app": {}
        },
        "WorkingDir": "/app",
        "Entrypoint": [
            "/app/bin/docker-entrypoint"
        ],
        "OnBuild": null,
        "Labels": {
            "com.docker.compose.config-hash": "37aee4728ee0efdfda0f09d60156acbd801c9c5889c35034863bfc1c055811b1",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.image": "sha256:1c3d39cc83861ce23476bedbc573dddba1956d0ff8da8c7e646666b36f51e578",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "redash",
            "com.docker.compose.project.config_files": "/home/emiliano/workspaces/redash/docker-compose.yml",
            "com.docker.compose.project.working_dir": "/home/emiliano/workspaces/redash",
            "com.docker.compose.service": "server",
            "com.docker.compose.version": "2.0.0",
            "desktop.docker.io/wsl-distro": "Debian"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "bd3103a51ccb5157ef31b4b68dd0e1d744da720becc2dc078d217fd1a2b9216a",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {},
        "SandboxKey": "/var/run/docker/netns/bd3103a51ccb",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "MacAddress": "",
        "Networks": {
            "redash_default": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": [
                    "redash_server_1",
                    "server",
                    "b76257886e44"
                ],
                "NetworkID": "d1f3e33c638f391d208faeb51d99e5a51c92fe7c8bea9df9635e89e7e97ad640",
                "EndpointID": "",
                "Gateway": "",
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "",
                "DriverOpts": null
            }
        }
    },
    "CreatedTime": 1632760934821
}

[susodapop]

Thanks for the inspect output @emilianosantucci . I agree there's nothing obvious out of place there. This seems like an environmental issue. Closing since we cannot reproduce.

For further help/discussion, I'd suggest the Docker Community Forums, the Docker Community Slack, or Stack Overflow.

[susodapop]

As some additional data in case it helps: I've seen issues like this when building Redash on M1 macs before Docker had a stable release for that CPU. The only fix was to exclude query runner dependencies that were broken on the M1 architecture, like @jbreiding described. However this was a workaround to an environmental problem. There was no way to fix it within Redash.

[jbreiding]

One last tidbit, this is the docker-compose file that ended up working for me

version: "3"
x-redash-service: &redash-service
  image: redash/redash:preview
  depends_on:
    - postgres
    - redis
  restart: always
x-redash-environment: &redash-environment
  REDASH_LOG_LEVEL: "DEBUG"
  REDASH_REDIS_URL: "redis://redis:6379/0"
  REDASH_DATABASE_URL: "postgresql://postgres@postgres/postgres"
  REDASH_RATELIMIT_ENABLED: "false"
  REDASH_ENABLED_QUERY_RUNNERS: "redash.query_runner.mysql,redash.query_runner.pg"
services:
  redash:
    <<: *redash-service
    command: server
    ports:
      - "5000:5000"
      - "5678:5678"
    environment:
      <<: *redash-environment
      REDASH_WEB_WORKERS: 4
      PYTHONUNBUFFERED: 0
    networks:
        - redash-network
  scheduler:
    <<: *redash-service
    command: scheduler
    environment:
      <<: *redash-environment
    networks:
        - redash-network
  adhoc_worker:
    <<: *redash-service
    command: worker
    environment:
      <<: *redash-environment
    networks:
        - redash-network
  redis:
    image: redis:5.0-alpine
    restart: always
    networks:
        - redash-network
  postgres:
    image: postgres:9.6-alpine
    ports:
      - "15432:5432"
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
    volumes:
      - data-volume:/var/lib/postgresql
    networks:
        - redash-network
    restart: always
  nginx:
    image: redash/nginx:latest
    ports:
      - "8000:80"
    depends_on:
      - redash
    networks:
        - redash-network
    restart: always
networks:
  redash-network:
    driver: bridge
    name: redash-network
volumes:
  data-volume:

[susodapop]

Reopening because at last I can reproduce! Thanks @zoomdot for the fix in #5608.

[susodapop]

The fix has been merged and passes CircleCI.

[kurianbenoy-aot]

Thanks @zoomdot for the fix

[zachliu]

in case someone is on the same boat, exit code 139 means exactly Segmentation fault took me some time to connect my Segmentation fault with this issue :sweat_smile:

however, why protobuf > 3.17.3 causes Segmentation fault is totally beyond me. something deep in their c code maybe?