search

budtmo / docker-android

Android in docker solution with noVNC supported and video recording
Other
9.29k stars 1.24k forks source link

Long running emulator becomes unhealthy - :glCheckFramebufferStatus:733 error 0x500 #205

Open mcopjan opened 4 years ago

mcopjan commented 4 years ago

🐛 Bug Report

Operating System: Linux 3.10.0-1062.1.1.el7.x86_64 x86_64

Docker compose:

version: '2'

services:
  selenium_hub:
    image: selenium/hub
    restart: unless-stopped
    ports:
      - "4444:4444"

  emulator:
    image: budtmo/docker-android-x86-8.1
    restart: unless-stopped
    privileged: true
    depends_on:
      - selenium_hub
    ports:
      - 6080
      - 5554
      - 5555
    environment:
      - DEVICE=Samsung Galaxy S10
      - CONNECT_TO_GRID=True
      - APPIUM=true
      - SELENIUM_HOST=selenium_hub
      - MOBILE_WEB_TEST=true

Docker Version:
Docker version 19.03.2, build 6a30dfc

Expected Behavior

When I leave my selenium grid with emulators running for longer period they should still be in 'healthy' state

Actual Behavior

I left my emulators up and running overnight (without using them for anything). After 12 hours I can see one of the containers became unhealthy with the following error image

[root@uiplinfdoc019 martinco]# docker exec -it docker_emulator_2 tail -f /var/log/supervisor/docker-android.stderr.log
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500
/buildbot/src/android/emu-master-dev/external/qemu/android/android-emugl/host/libs/Translator/GLES_V2/GLESv2Imp.cpp:glCheckFramebufferStatus:733 error 0x500

System resources does not seem to be fully depleted. image

un000 commented 4 years ago

Got another unhealthy status for my long-running container:

docker-android.stderr.log

xterm: cannot load font "-Misc-Fixed-bold-R-*-*-13-120-75-75-C-60-ISO8859-1"
Warning: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' ((null):0, (null))
emulator: ERROR: AdbHostServer.cpp:93: Unable to connect to adb daemon on port: 5037
emulator: ERROR: AdbHostServer.cpp:93: Unable to connect to adb daemon on port: 5037
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
Warning: libpng warning: iCCP: known incorrect sRGB profile ((null):0, (null))
Warning: This program is an suid-root program or is being run by the root user.
The full text of the error or warning message cannot be safely formatted
in this environment. You may get a more descriptive message by running the
program as a non-root user or by removing the suid bit on the executable.
xterm: Xt error: Can't open display: %s
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/src/app.py", line 238, in <module>
    run()
  File "/root/src/app.py", line 232, in run
    appium_run(avd_name)
  File "/root/src/app.py", line 157, in appium_run
    subprocess.check_call('xterm -T "{title}" -n "{title}" -e \"{cmd}\"'.format(title=title, cmd=cmd), shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'xterm -T "Appium Server" -n "Appium Server" -e "appium --log /var/log/supervisor/appium.log --chromedriver-executable /root/chromedriver"' returned non-zero exit status 1.
xterm: cannot load font "-Misc-Fixed-bold-R-*-*-13-120-75-75-C-60-ISO8859-1"
Warning: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root' ((null):0, (null))
xvfb.stderr.log

(EE) Server is already active for display 0
    If this server is no longer running, remove /tmp/.X0-lock
    and start again.
(EE)
(EE)
Fatal server error:
(EE) Server is already active for display 0
    If this server is no longer running, remove /tmp/.X0-lock
    and start again.
(EE)
supervisord.log

2019-11-03 11:24:47,528 CRIT Supervisor running as root (no user in config file)
2019-11-03 11:24:47,531 INFO supervisord started with pid 6
2019-11-03 11:24:48,534 INFO spawned: 'xvfb' with pid 9
2019-11-03 11:24:48,536 INFO spawned: 'port-forward' with pid 10
2019-11-03 11:24:48,538 INFO spawned: 'novnc' with pid 11
2019-11-03 11:24:48,540 INFO spawned: 'openbox' with pid 12
2019-11-03 11:24:48,542 INFO spawned: 'x11vnc' with pid 13
2019-11-03 11:24:48,544 INFO spawned: 'android-screen-mirror' with pid 14
2019-11-03 11:24:48,546 INFO spawned: 'docker-appium' with pid 20
2019-11-03 11:24:48,549 INFO spawned: 'atd' with pid 23
2019-11-03 11:24:48,551 INFO spawned: 'auto-recording' with pid 27
2019-11-03 11:24:48,553 INFO spawned: 'adb-utils' with pid 29
2019-11-03 11:24:48,554 INFO exited: xvfb (exit status 1; not expected)
2019-11-03 11:24:48,554 INFO exited: android-screen-mirror (exit status 0; expected)
2019-11-03 11:24:48,557 INFO exited: atd (exit status 0; not expected)
2019-11-03 11:24:48,560 INFO exited: x11vnc (exit status 1; not expected)
2019-11-03 11:24:48,569 INFO exited: openbox (exit status 1; not expected)
2019-11-03 11:24:48,951 INFO exited: docker-appium (exit status 1; not expected)
2019-11-03 11:24:49,577 INFO spawned: 'xvfb' with pid 81
2019-11-03 11:24:49,577 INFO success: port-forward entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-03 11:24:49,577 INFO success: novnc entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-03 11:24:49,579 INFO spawned: 'openbox' with pid 83
2019-11-03 11:24:49,580 INFO spawned: 'x11vnc' with pid 84
2019-11-03 11:24:49,582 INFO spawned: 'atd' with pid 85
2019-11-03 11:24:49,582 INFO success: auto-recording entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-03 11:24:49,582 INFO success: adb-utils entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-03 11:24:49,590 INFO exited: xvfb (exit status 1; not expected)
2019-11-03 11:24:49,590 INFO exited: atd (exit status 0; not expected)
2019-11-03 11:24:49,597 INFO exited: x11vnc (exit status 1; not expected)
2019-11-03 11:24:49,608 INFO exited: openbox (exit status 1; not expected)
2019-11-03 11:24:50,610 INFO spawned: 'docker-appium' with pid 91
2019-11-03 11:24:51,608 INFO spawned: 'xvfb' with pid 102
2019-11-03 11:24:51,611 INFO spawned: 'openbox' with pid 103
2019-11-03 11:24:51,613 INFO spawned: 'x11vnc' with pid 105
2019-11-03 11:24:51,613 INFO success: docker-appium entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2019-11-03 11:24:51,615 INFO spawned: 'atd' with pid 106
2019-11-03 11:24:51,622 INFO exited: xvfb (exit status 1; not expected)
2019-11-03 11:24:51,624 INFO exited: atd (exit status 0; not expected)
2019-11-03 11:24:51,631 INFO exited: x11vnc (exit status 1; not expected)
2019-11-03 11:24:51,640 INFO exited: openbox (exit status 1; not expected)
2019-11-03 11:24:51,748 INFO exited: docker-appium (exit status 1; not expected)
2019-11-03 11:24:54,563 INFO exited: auto-recording (exit status 0; expected)
2019-11-03 11:24:55,566 INFO spawned: 'xvfb' with pid 122
2019-11-03 11:24:55,568 INFO spawned: 'openbox' with pid 123
2019-11-03 11:24:55,570 INFO spawned: 'x11vnc' with pid 124
2019-11-03 11:24:55,572 INFO spawned: 'atd' with pid 125
2019-11-03 11:24:55,580 INFO exited: xvfb (exit status 1; not expected)
2019-11-03 11:24:55,581 INFO gave up: xvfb entered FATAL state, too many start retries too quickly
2019-11-03 11:24:55,581 INFO exited: atd (exit status 0; not expected)
2019-11-03 11:24:55,587 INFO gave up: atd entered FATAL state, too many start retries too quickly
2019-11-03 11:24:55,589 INFO exited: x11vnc (exit status 1; not expected)
2019-11-03 11:24:55,595 INFO gave up: x11vnc entered FATAL state, too many start retries too quickly
2019-11-03 11:24:55,596 INFO exited: openbox (exit status 1; not expected)
2019-11-03 11:24:56,598 INFO gave up: openbox entered FATAL state, too many start retries too quickly

openbox.stderr.log

ERROR: openbox-xdg-autostart requires PyXDG to be installed
ERROR: openbox-xdg-autostart requires PyXDG to be installed
ERROR: openbox-xdg-autostart requires PyXDG to be installed
ERROR: openbox-xdg-autostart requires PyXDG to be installed
ERROR: openbox-xdg-autostart requires PyXDG to be installed
openbox.stderr.log

ERROR: openbox-xdg-autostart requires PyXDG to be installed
ERROR: openbox-xdg-autostart requires PyXDG to be installed
/tmp/adb.0.log

adb server killed by remote request
--- adb starting (pid 8139) ---
adb I 11-03 14:18:17  8139  8139 main.cpp:60] Android Debug Bridge version 1.0.41
adb I 11-03 14:18:17  8139  8139 main.cpp:60] Version 29.0.1-5644136
adb I 11-03 14:18:17  8139  8139 main.cpp:60] Installed as /root/platform-tools/adb
adb I 11-03 14:18:17  8139  8139 main.cpp:60]
adb I 11-03 14:18:17  8139  8139 auth.cpp:436] adb_auth_init...
adb I 11-03 14:18:17  8139  8139 auth.cpp:411] adb_auth_inotify_init...
adb server killed by remote request
--- adb starting (pid 8160) ---
adb I 11-03 14:18:21  8160  8160 main.cpp:60] Android Debug Bridge version 1.0.41
adb I 11-03 14:18:21  8160  8160 main.cpp:60] Version 29.0.1-5644136
adb I 11-03 14:18:21  8160  8160 main.cpp:60] Installed as /root/platform-tools/adb
adb I 11-03 14:18:21  8160  8160 main.cpp:60]
adb I 11-03 14:18:21  8160  8160 auth.cpp:436] adb_auth_init...
adb I 11-03 14:18:21  8160  8160 auth.cpp:411] adb_auth_inotify_init...
adb server killed by remote request
--- adb starting (pid 8181) ---
adb I 11-03 14:18:24  8181  8181 main.cpp:60] Android Debug Bridge version 1.0.41
adb I 11-03 14:18:24  8181  8181 main.cpp:60] Version 29.0.1-5644136
adb I 11-03 14:18:24  8181  8181 main.cpp:60] Installed as /root/platform-tools/adb
adb I 11-03 14:18:24  8181  8181 main.cpp:60]
adb I 11-03 14:18:24  8181  8181 auth.cpp:436] adb_auth_init...
adb I 11-03 14:18:24  8181  8181 auth.cpp:411] adb_auth_inotify_init...

Then I recreate a container with the docker-compose and it continues working with a previous state:

image: budtmo/docker-android-x86-9.0
volumes:
      - /opt/emulator/data/.android:/root/.android
      - /opt/emulator/data/android_emulator:/root/android_emulator
    environment:
      - EMULATOR_ARGS=-gpu off -no-boot-anim -partition-size 8192
      - DEVICE=Samsung Galaxy S10
      - APPIUM=true

After some hours dies.

un000 commented 4 years ago 
root@5dc98862fc1c:~# ps afx
   PID TTY      STAT   TIME COMMAND
  1253 pts/1    Ss     0:00 bash
  4095 pts/1    R+     0:00  \_ ps afx
     1 ?        Ss     0:00 /bin/sh -c /usr/bin/supervisord --configuration supervisord.conf
     6 ?        S      0:00 /usr/bin/python /usr/bin/supervisord --configuration supervisord.conf
     9 ?        Rl     0:01  \_ /usr/bin/Xvfb :0 -screen 0 1600x900x24+32
    10 ?        S      0:00  \_ /bin/bash ./src/port_forward.sh
    19 ?        S      0:00  |   \_ socat tcp-listen:5554,bind=172.20.0.2,fork tcp:127.0.0.1:5554
    20 ?        S      0:00  |   \_ socat tcp-listen:5555,bind=172.20.0.2,fork tcp:127.0.0.1:5555
    11 ?        S      0:00  \_ bash ./noVNC/utils/launch.sh --vnc localhost:5900 --listen 6080
    45 ?        S      0:00  |   \_ python /root/noVNC/utils/websockify/run --web /root/noVNC/utils/../ 6080 localhost:5900
    12 ?        S      0:00  \_ /usr/bin/openbox --startup /usr/lib/x86_64-linux-gnu/openbox-autostart OPENBOX
    18 ?        S      0:00  \_ /usr/bin/x11vnc -display :0 -nopw -forever -shared
    28 ?        S      0:00  \_ /bin/bash ./src/appium.sh
    78 ?        S      0:00      \_ python3 -m src.app
    88 ?        Sl    28:58          \_ /root/emulator/qemu/linux-x86_64/qemu-system-x86_64 @samsung_galaxy_s10_9.0 -verbose -writable-system -gpu off -no-boot-anim -par
    99 ?        Sl     0:00          |   \_ /root/emulator/emulator64-crash-service -pipe 4 -ppid 88 -data-dir /tmp/android-unknown/111b0981-d601-4097-963c-f99fe8abd7fe
    92 ?        S      0:00          \_ /bin/sh -c xterm -T "Appium Server" -n "Appium Server" -e "appium --log /var/log/supervisor/appium.log --chromedriver-executable
    94 ?        S      0:00              \_ xterm -T Appium Server -n Appium Server -e appium --log /var/log/supervisor/appium.log --chromedriver-executable /root/chrome
    96 pts/0    Ssl+   0:12                  \_ node /usr/bin/appium --log /var/log/supervisor/appium.log --chromedriver-executable /root/chromedriver
  3201 pts/0    S+     0:00                      \_ /root/platform-tools/adb -P 5037 -s emulator-5554 logcat -v threadtime
  3985 pts/0    S+     0:00                      \_ /root/platform-tools/adb -P 5037 -s emulator-5554 logcat -v threadtime
  4093 pts/0    Sl+    0:00                      \_ /root/platform-tools/adb -P 5037 -s emulator-5554 shell am instrument -w --no-window-animation io.appium.uiautomator2
   178 ?        Ssl    0:02 adb -L tcp:5037 fork-server server --reply-fd 4

kill -9 28 which kills /bin/bash ./src/appium.sh helps