search

chromedp / chromedp

A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
MIT License
10.82k stars 783 forks source link

{"method":"Target.targetCrashed","params":{"targetId":"XXX","status":"killed","errorCode":9} when running in Kubernetes #1368

Open seanprince opened 11 months ago

seanprince commented 11 months ago

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.9.2
$ google-chrome --version
chromium-112.0.5615.165-r0
$ go version
go version go1.20.1 linux/amd64

What did you do? Include clear steps.

Attempt to navigate to a page. Code is running in a Docker container within a Kubernetes cluster.

func (us *uiService) Login(ctx context.Context, pulseStudy *domain.Study) error {
    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.IgnoreCertErrors,
        chromedp.DisableGPU,
        chromedp.NoSandbox,
        chromedp.Headless,
    )

    ctx, cancel := chromedp.NewExecAllocator(ctx, opts...)
    defer cancel()

    ctx, cancel = context.WithTimeout(ctx, 2*time.Minute)
    defer cancel()

    ctx, cancel = chromedp.NewContext(ctx,
        chromedp.WithDebugf(us.browserDebugf),
        chromedp.WithLogf(us.browserLogf),
        chromedp.WithErrorf(us.browserErrorf),
    )
    defer cancel()

    url := "https://" + us.host
    err := us.cdp.Run(ctx,
        chromedp.Navigate(url),
    )
    if err != nil {
        return fmt.Errorf("failed to enter user name and password due to error: %w", err)
    }
        return nil
}

What did you expect to see?

Nil error returned from Run().

What did you see instead?

'context deadline exceeded' error after 2 minute timeout period. The last few lines of debug output is:

<- [{"method":"Target.targetCrashed","params":{"targetId":"1730AB1333F46FC8470755FA4DFC1AB1","status":"killed","errorCode":9},"sessionId":"973E3A6D83F4406301876E26B6FD967A"}]
<- [{"method":"Target.targetCrashed","params":{"targetId":"1730AB1333F46FC8470755FA4DFC1AB1","status":"killed","errorCode":9}}]
<- [{"method":"Inspector.targetCrashed","params":{},"sessionId":"973E3A6D83F4406301876E26B6FD967A"}]
<- [{"method":"Network.responseReceivedExtraInfo","params":{"requestId":"738.24","blockedCookies":[],"headers":{"accept-ranges":"bytes","access-control-allow-origin":"*","access-control-expose-headers":"Content-Length, Content-Type, Date, Server, Transfer-Encoding, X-GUploader-UploadID, X-Google-Trace","cache-control":"public, max-age=3600","content-length":"41144","content-type":"font/woff2","date":"Fri, 06 Oct 2023 17:03:31 GMT","etag":"\"5527312710fb1fbb6a8170020be4b187\"","expires":"Fri, 06 Oct 2023 18:03:31 GMT","last-modified":"Tue, 04 Apr 2023 13:47:17 GMT","server":"UploadServer","x-goog-generation":"1680616037082932","x-goog-hash":"crc32c=mA5v9w==\nmd5=VScxJxD7H7tqgXACC+Sxhw==","x-goog-metageneration":"3","x-goog-storage-class":"MULTI_REGIONAL","x-goog-stored-content-encoding":"identity","x-goog-stored-content-length":"41144","x-guploader-uploadid":"ADPycdstaYK_LNSaPRHPG13cJTVrdYjP4e-3o01_WEGeC7S3CxpJXOtZ_E-VHF4iFA0YJlIL720QL3o7JkkU6ln9X9iMfg"},"resourceIPAddressSpace":"Public","statusCode":200},"sessionId":"973E3A6D83F4406301876E26B6FD967A"}]
<- [{"method":"Network.loadingFinished","params":{"requestId":"738.95","timestamp":19156.504732,"encodedDataLength":73527,"shouldReportCorbBlocking":false},"sessionId":"973E3A6D83F4406301876E26B6FD967A"}]
<- [{"method":"Network.dataReceived","params":{"requestId":"738.95","timestamp":19156.705291,"dataLength":110664,"encodedDataLength":73217},"sessionId":"973E3A6D83F4406301876E26B6FD967A"}]

If I run the same Docker image outside Kubernetes, then this works fine.

ZekeLu commented 11 months ago

Can you share the content of the Dockerfile?

Please try to use the chromedp.CombinedOutput option to capture the error log of the browser. Hopefully that the log contains why the tab is crashed.

seanprince commented 10 months ago

Hi there - thanks for helping out. Dockerfile is a combination of:

FROM alpine:3.17.3

# add root certificates, mime types and required c libraries
RUN apk add --update --no-cache \
    ca-certificates=20220614-r4 \
    mailcap=2.1.53-r0 && \
    openssl=3.0.8-r4 && \
    libssl3=3.0.8-r4 && \
    mkdir /lib64 && \
    ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

RUN mkdir /app
RUN addgroup -S bumblebee && adduser -S -g bumblebee bumblebee
RUN chown bumblebee:bumblebee /app
USER bumblebee

and

FROM eu.gcr.io/cybertron-164209/andellor/base-images.bumblebee-go:1.0.24

USER root
# Install chrome depedencies
RUN set -ex && \
    apk update && \
    apk add --update bash chromium && \
    rm -rf /var/cache/apk/*

USER bumblebee
EXPOSE 8080

USER bumblebee

COPY ./bumblebee-pulse-server /app/
COPY ./P04 /app/
CMD /app/bumblebee-pulse-server --port 8080 --host 0.0.0.0

I'm working on using chromedp.CombinedOutput to get the browser error log - will update you when I have that information.

seanprince commented 10 months ago

Hi there, could you share an example of how to use chromedp.CombinedOutput, please? I'm struggling to know how to use this.

ZekeLu commented 10 months ago

Hi, here is one of the examples: https://github.com/chromedp/chromedp/issues/1216#issuecomment-1341901485.

seanprince commented 10 months ago

Hi there, I can see the chromedp.CombinedOutput when running locally. Unfortunately this output is not visible when running inside Kubernetes (maybe for the same reason as the browser tab crashing?), so I'm no further forward there.

Can you see any issue with the Dockerfile? Is there any missing software when using alpine as the base image?

ZekeLu commented 10 months ago

The first Dockerfile has some errors (The first two && should be removed). But since you have built a docker image successfully and you can run the docker image outside Kubernetes, I assume that you were using a correct Dockerfile.

My guess is that the node does not have enough resource to run the browser. But we have to check the error log of the browser to confirm that. Maybe you could execute the following command in the pod to see if it reports any error:

chromium-browser --headless --no-sandbox

If possible, please provide a detailed instruction to guide me to reproduce the issue myself. Thank you!

seanprince commented 10 months ago

The browser does not start when I run it from within the pod:

[1101/171823.985658:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
Fontconfig error: No writable cache directories
Fontconfig error: No writable cache directories
[1101/171824.089308:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1101/171824.089596:ERROR:bus.cc(399)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory
[1101/171824.089879:WARNING:dns_config_service_linux.cc(429)] Failed to read DnsConfig.
[1101/171824.093110:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 10979: Permission denied (13)
[1101/171824.187907:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1101/171824.188255:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1101/171824.188514:ERROR:angle_platform_impl.cc(43)] Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
ERR: Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.188693:ERROR:gl_display.cc(504)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.188884:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 10981: Permission denied (13)
[1101/171824.188899:ERROR:gl_display.cc(793)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1101/171824.189070:ERROR:gl_display.cc(819)] Initialization of all EGL display types failed.
[1101/171824.189191:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1101/171824.283496:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
[1101/171824.283800:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1101/171824.284027:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1101/171824.284225:ERROR:angle_platform_impl.cc(43)] Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
ERR: Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.284374:ERROR:gl_display.cc(504)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.284589:ERROR:gl_display.cc(793)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1101/171824.284717:ERROR:gl_display.cc(819)] Initialization of all EGL display types failed.
[1101/171824.284892:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1101/171824.287090:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization
[1101/171824.483778:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 10994: Permission denied (13)
[1101/171824.587279:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1101/171824.587553:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1101/171824.587789:ERROR:angle_platform_impl.cc(43)] Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
ERR: Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.587965:ERROR:gl_display.cc(504)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.588105:ERROR:gl_display.cc(793)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1101/171824.588223:ERROR:gl_display.cc(819)] Initialization of all EGL display types failed.
[1101/171824.588320:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1101/171824.590335:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_surface
[1101/171824.590545:ERROR:angle_platform_impl.cc(43)] RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
ERR: RendererVk.cpp:157 (VerifyExtensionsPresent): Extension not supported: VK_KHR_xcb_surface
[1101/171824.590688:ERROR:angle_platform_impl.cc(43)] Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
ERR: Display.cpp:1019 (initialize): ANGLE Display::initialize error 0: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.590829:ERROR:gl_display.cc(504)] EGL Driver message (Critical) eglInitialize: Internal Vulkan error (-7): A requested extension is not supported, in ../../third_party/angle/src/libANGLE/renderer/vulkan/RendererVk.cpp, enableInstanceExtensions:1639.
[1101/171824.591004:ERROR:gl_display.cc(793)] eglInitialize SwANGLE failed with error EGL_NOT_INITIALIZED
[1101/171824.591110:ERROR:gl_display.cc(819)] Initialization of all EGL display types failed.
[1101/171824.591215:ERROR:gl_ozone_egl.cc(26)] GLDisplayEGL::Initialize failed.
[1101/171824.684604:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization
[1101/171824.783766:ERROR:zygote_host_impl_linux.cc(273)] Failed to adjust OOM score of renderer with pid 11005: Permission denied (13)
[1101/171825.090536:WARNING:dns_config_service_linux.cc(429)] Failed to read DnsConfig.

Looks like a writeable folder required does not exist?

ZekeLu commented 10 months ago

Looks like the browser can not start at all. There is very little we can do unless you tell us how to reproduce the issue ourselves.