Elasticsearch 6: pmap is not working

mortyccp commented 5 years ago

Bug Description

Docker image used: docker.elastic.co/elasticsearch/elasticsearch:6.0.1
Operating System: Darwin 18.0.0 Darwin Kernel Version 18.0.0

docker info output

Containers: 19
Running: 19
Paused: 0
Stopped: 0
Images: 19
Server Version: 18.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.93-linuxkit-aufs
Operating System: Docker for Mac
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: linuxkit-025000000001
ID: UQ4B:IEQI:AHS6:RMY2:6R3A:QRJU:4OO2:V4E2:IZ5D:DWXV:ZXOM:27TY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 131
Goroutines: 142
System Time: 2018-10-25T16:30:54.552498237Z
EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

docker inspect NAME|ID output

[
{
    "Id": "da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7",
    "Created": "2018-10-25T16:26:13.778847387Z",
    "Path": "/usr/local/bin/docker-entrypoint.sh",
    "Args": [
        "eswrapper"
    ],
    "State": {
        "Status": "running",
        "Running": true,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 58020,
        "ExitCode": 0,
        "Error": "",
        "StartedAt": "2018-10-25T16:26:14.577445017Z",
        "FinishedAt": "0001-01-01T00:00:00Z"
    },
    "Image": "sha256:28259852697e3d82125bec210b29e06bbd950c8395eb5b0a253dc22349ba3011",
    "ResolvConfPath": "/var/lib/docker/containers/da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7/resolv.conf",
    "HostnamePath": "/var/lib/docker/containers/da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7/hostname",
    "HostsPath": "/var/lib/docker/containers/da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7/hosts",
    "LogPath": "/var/lib/docker/containers/da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7/da95961f9150ec13f1d7542bcc98062b9bc87df3234a0629832da99ff37ca4e7-json.log",
    "Name": "/es",
    "RestartCount": 0,
    "Driver": "overlay2",
    "Platform": "linux",
    "MountLabel": "",
    "ProcessLabel": "",
    "AppArmorProfile": "",
    "ExecIDs": null,
    "HostConfig": {
        "Binds": null,
        "ContainerIDFile": "",
        "LogConfig": {
            "Type": "json-file",
            "Config": {}
        },
        "NetworkMode": "default",
        "PortBindings": {
            "9200/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "9200"
                }
            ],
            "9300/tcp": [
                {
                    "HostIp": "",
                    "HostPort": "9300"
                }
            ]
        },
        "RestartPolicy": {
            "Name": "no",
            "MaximumRetryCount": 0
        },
        "AutoRemove": true,
        "VolumeDriver": "",
        "VolumesFrom": null,
        "CapAdd": null,
        "CapDrop": null,
        "Dns": [],
        "DnsOptions": [],
        "DnsSearch": [],
        "ExtraHosts": null,
        "GroupAdd": null,
        "IpcMode": "shareable",
        "Cgroup": "",
        "Links": null,
        "OomScoreAdj": 0,
        "PidMode": "",
        "Privileged": false,
        "PublishAllPorts": false,
        "ReadonlyRootfs": false,
        "SecurityOpt": null,
        "UTSMode": "",
        "UsernsMode": "",
        "ShmSize": 67108864,
        "Runtime": "runc",
        "ConsoleSize": [
            0,
            0
        ],
        "Isolation": "",
        "CpuShares": 0,
        "Memory": 0,
        "NanoCpus": 0,
        "CgroupParent": "",
        "BlkioWeight": 0,
        "BlkioWeightDevice": [],
        "BlkioDeviceReadBps": null,
        "BlkioDeviceWriteBps": null,
        "BlkioDeviceReadIOps": null,
        "BlkioDeviceWriteIOps": null,
        "CpuPeriod": 0,
        "CpuQuota": 0,
        "CpuRealtimePeriod": 0,
        "CpuRealtimeRuntime": 0,
        "CpusetCpus": "",
        "CpusetMems": "",
        "Devices": [],
        "DeviceCgroupRules": null,
        "DiskQuota": 0,
        "KernelMemory": 0,
        "MemoryReservation": 0,
        "MemorySwap": 0,
        "MemorySwappiness": null,
        "OomKillDisable": false,
        "PidsLimit": 0,
        "Ulimits": null,
        "CpuCount": 0,
        "CpuPercent": 0,
        "IOMaximumIOps": 0,
        "IOMaximumBandwidth": 0,
        "MaskedPaths": [
            "/proc/acpi",
            "/proc/kcore",
            "/proc/keys",
            "/proc/latency_stats",
            "/proc/timer_list",
            "/proc/timer_stats",
            "/proc/sched_debug",
            "/proc/scsi",
            "/sys/firmware"
        ],
        "ReadonlyPaths": [
            "/proc/asound",
            "/proc/bus",
            "/proc/fs",
            "/proc/irq",
            "/proc/sys",
            "/proc/sysrq-trigger"
        ]
    },
    "GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/b9de29ada8b94801153eae111813074b16e5005d88516311a6cd4d67fd7494ae-init/diff:/var/lib/docker/overlay2/2e7e7b5182a42830e5673c2dcc38b4e4872d0027d38697a4ebe43c2c86d8a270/diff:/var/lib/docker/overlay2/8361b7f8ce78c35b66d11cfa3b4432c1b12c778d02c2381f1b0894b10466789e/diff:/var/lib/docker/overlay2/3ec2ef7c07a9b9508227f1bed005ff1fa3a6294396c64b348bf956f5dd5da986/diff:/var/lib/docker/overlay2/b5bbfc4d452009336af55dbcf4543ec39b4abaf10f6315c26a64d36dc0e9e7a6/diff:/var/lib/docker/overlay2/afff32cd9e86e6a1daf6995c3260fc044d6b9e16c3640e9bc0cfb969b0f31fcf/diff:/var/lib/docker/overlay2/a7dfec933703213407af829fa72ad5936de417115008906d552edbfc18d2ca4f/diff",
            "MergedDir": "/var/lib/docker/overlay2/b9de29ada8b94801153eae111813074b16e5005d88516311a6cd4d67fd7494ae/merged",
            "UpperDir": "/var/lib/docker/overlay2/b9de29ada8b94801153eae111813074b16e5005d88516311a6cd4d67fd7494ae/diff",
            "WorkDir": "/var/lib/docker/overlay2/b9de29ada8b94801153eae111813074b16e5005d88516311a6cd4d67fd7494ae/work"
        },
        "Name": "overlay2"
    },
    "Mounts": [],
    "Config": {
        "Hostname": "da95961f9150",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": true,
        "AttachStderr": true,
        "ExposedPorts": {
            "9200/tcp": {},
            "9300/tcp": {}
        },
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "discovery.type=single-node",
            "PATH=/usr/share/elasticsearch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
            "ELASTIC_CONTAINER=true",
            "JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk"
        ],
        "Cmd": [
            "eswrapper"
        ],
        "ArgsEscaped": true,
        "Image": "docker.elastic.co/elasticsearch/elasticsearch:6.0.1",
        "Volumes": null,
        "WorkingDir": "/usr/share/elasticsearch",
        "Entrypoint": [
            "/usr/local/bin/docker-entrypoint.sh"
        ],
        "OnBuild": null,
        "Labels": {
            "build-date": "20171128",
            "license": "GPLv2",
            "maintainer": "Elastic Docker Team <docker@elastic.co>",
            "name": "CentOS Base Image",
            "vendor": "CentOS"
        }
    },
    "NetworkSettings": {
        "Bridge": "",
        "SandboxID": "1eeb0787f091f3ad38c44e1b188f21cf01ee7d9cefbbb466128d51e73de5e9c8",
        "HairpinMode": false,
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "Ports": {
            "9200/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "9200"
                }
            ],
            "9300/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "9300"
                }
            ]
        },
        "SandboxKey": "/var/run/docker/netns/1eeb0787f091",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null,
        "EndpointID": "5634c59bb214c6c57577e79d513c6f541318d30ccce4bb6e060bce1efbf8cdf6",
        "Gateway": "172.17.0.1",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "IPAddress": "172.17.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "MacAddress": "02:42:ac:11:00:02",
        "Networks": {
            "bridge": {
                "IPAMConfig": null,
                "Links": null,
                "Aliases": null,
                "NetworkID": "38cb4ae4cc5c869f7c63f3df8445dbe1bf217e5281063ada895b655268a6a1dc",
                "EndpointID": "5634c59bb214c6c57577e79d513c6f541318d30ccce4bb6e060bce1efbf8cdf6",
                "Gateway": "172.17.0.1",
                "IPAddress": "172.17.0.2",
                "IPPrefixLen": 16,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "02:42:ac:11:00:02",
                "DriverOpts": null
            }
        }
    }
}
]

Steps to reproduce
1. docker container run -it --rm --name es -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.0.1
2. docker container exec es pmap 1
Expected result pmap 1 should return valid result just like elasticsearch 5

jarpy commented 5 years ago

I don't know why yet, but adding the SYS_PTRACE capability restores the desired behaviour.

$ docker container run -d --name es -e "discovery.type=single-node" --cap-add SYS_PTRACE docker.elastic.co/elasticsearch/elasticsearch:6.0.1
96a3d6c064187d6a850820f9b6aee9d9817b974ff25f10c424f9d979ab498c0d

$ docker container exec es pmap 1 | head -n 10
1:   /usr/lib/jvm/jre-1.8.0-openjdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -XX:+HeapDumpOnOutOfMemoryError -Des.cgroups.hierarchy.override=/
0000000000400000      4K r-x-- java
0000000000600000      4K r---- java
0000000000601000      4K rw--- java
00000000015cd000    132K rw---   [ anon ]
00000000c0000000 1056292K rw---   [ anon ]
0000000100789000 1040860K -----   [ anon ]
00007fa4f0000000    132K rw---   [ anon ]
00007fa4f0021000  65404K -----   [ anon ]
00007fa4f8000000    132K rw---   [ anon ]

rjernst commented 5 years ago

@mortyccp As this issue was with Elasticsearch 6, can you confirm the issue still exists for you with Elasticsearch 7?

Additionally, maintenance of the docker files for elasticsearch has moved to the elasticsearch repo. As we will be archiving this repository, I am going to close this issue. If you can confirm the issue still exists, please open a new issue there.

elastic / elasticsearch-docker

Elasticsearch 6: pmap is not working #208

Bug Description