search

chromedp / chromedp

A faster, simpler way to drive browsers supporting the Chrome DevTools Protocol.
MIT License
11.07k stars 789 forks source link

Problem with aws lambda #1074

Closed Andiedie closed 2 years ago

Andiedie commented 2 years ago

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.8.2

$ docker pull chromedp/headless-shell:latest
latest: Pulling from chromedp/headless-shell
Digest: sha256:8fe1cbf5aba60474079fedf56867a3a3576340ab21b6e222674edcb6610a584d
Status: Image is up to date for chromedp/headless-shell:latest
docker.io/chromedp/headless-shell:latest

$ docker run --rm golang:alpine go version
go version go1.18.2 linux/amd64

What did you do? Include clear steps.

Here is an example repo. I want to deploy it to aws lambda.

Code snippet

func Handler(_ context.Context, _ json.RawMessage) error {
    opts := []chromedp.ExecAllocatorOption{
        chromedp.NoFirstRun,
        chromedp.NoDefaultBrowserCheck,
        chromedp.NoSandbox,
        chromedp.DisableGPU,
        chromedp.Headless,
        chromedp.Flag("no-zygote", true),
        chromedp.Flag("user-data-dir", "/tmp/chrome-user-data-dir"),
        chromedp.Flag("homedir", "/tmp/chrome-home"),
        chromedp.Flag("data-path", "/tmp/chrome-data-path"),
        chromedp.Flag("disk-cache-dir", "/tmp/chrome-disk-cache-dir"),
        chromedp.Flag("remote-debugging-port", "9222"),
        chromedp.Flag("remote-debugging-address", "0.0.0.0"),
        chromedp.Flag("disable-dev-shm-usage", true),
        chromedp.Flag("enable-features", "NetworkService,NetworkServiceInProcess"),
    }
    ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
    defer cancel()

    ctx, cancel = chromedp.NewContext(ctx, chromedp.WithDebugf(log.Printf))
    defer cancel()

    var content string
    if err := chromedp.Run(ctx, chromedp.Tasks{
        chromedp.Navigate("https://example.com/"),
        chromedp.Text("body > div > p:nth-child(2)", &content),
    }); err != nil {
        log.Fatal(err)
    }
    fmt.Println(content)
    return nil
}
FROM golang:alpine AS builder

RUN apk add git

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o main

FROM chromedp/headless-shell

RUN apt-get update && \
    apt-get install -y dumb-init musl-dev && \
    ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 &&\
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/main .

ENTRYPOINT [ "dumb-init", "--" ]
CMD [ "./main" ]

This code (without docker) run perfectly on my M1 macbook, and built docker container run perfectly on AWS EC2.

But when I deploy it to aws lambda, it just stuck after last log 2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}.

What did you expect to see?

I can run it on aws lambda.

Log on ec2 running perfectly

2022/06/01 05:13:03 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:03 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/06/01 05:13:03 <- {"id":1,"result":{}}
2022/06/01 05:13:03 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"412DC8003B59C78196B600AFBB7C0231","flatten":true}}
2022/06/01 05:13:03 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:03 <- {"method":"Target.attachedToTarget","params":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"},"waitingForDebugger":false}}
2022/06/01 05:13:03 <- {"id":2,"result":{"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}}
2022/06/01 05:13:04 -> {"id":3,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.enable"}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":1,"origin":"://","name":"","uniqueId":"5877140344713506672.6727992831272504814","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":3,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":4,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.evaluate","params":{"expression":"self"}}
2022/06/01 05:13:04 <- {"id":4,"result":{"result":{"type":"object","className":"Window","description":"Window","objectId":"5836470857787727263.1.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":5,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Log.enable"}
2022/06/01 05:13:04 <- {"id":5,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":6,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Network.enable","params":{}}
2022/06/01 05:13:04 <- {"id":6,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":7,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Inspector.enable"}
2022/06/01 05:13:04 <- {"id":7,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":8,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.enable"}
2022/06/01 05:13:04 <- {"id":8,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":9,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.enable","params":{}}
2022/06/01 05:13:04 <- {"id":9,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":10,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"CSS.enable"}
2022/06/01 05:13:04 <- {"id":10,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":11,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"11278bc8-3ff8-436d-901f-a6404ecb61d1","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"about:blank","url":"about:blank","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e42b9ee5-9df7-49b6-b6f2-0314165479f6","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":11,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":12,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Target.setAutoAttach","params":{"autoAttach":true,"waitForDebuggerOnStart":false,"flatten":true}}
2022/06/01 05:13:04 <- {"id":12,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":13,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.setLifecycleEventsEnabled","params":{"enabled":true}}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"commit","timestamp":55583.941934},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"DOMContentLoaded","timestamp":55583.942021},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"load","timestamp":55583.942425},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":13,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":14,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Page.navigate","params":{"url":"https://example.com/"}}
2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSent","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","documentURL":"https://example.com/","request":{"url":"https://example.com/","method":"GET","headers":{"Upgrade-Insecure-Requests":"1","User-Agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"mixedContentType":"none","initialPriority":"VeryHigh","referrerPolicy":"strict-origin-when-cross-origin","isSameSite":true},"timestamp":55583.963522,"wallTime":1654060384.050588,"initiator":{"type":"other"},"redirectHasExtraInfo":false,"type":"Document","frameId":"412DC8003B59C78196B600AFBB7C0231","hasUserGesture":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.requestWillBeSentExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","associatedCookies":[],"headers":{":authority":"example.com",":method":"GET",":path":"/",":scheme":"https","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","accept-encoding":"gzip, deflate, br","accept-language":"en-US@posix","sec-fetch-dest":"document","sec-fetch-mode":"navigate","sec-fetch-site":"none","sec-fetch-user":"?1","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36"},"connectTiming":{"requestTime":55583.965293}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.responseReceivedExtraInfo","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","blockedCookies":[],"headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"resourceIPAddressSpace":"Public","statusCode":200},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.responseReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.700562,"type":"Document","response":{"url":"https://example.com/","status":200,"statusText":"","headers":{"accept-ranges":"bytes","age":"267163","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Wed, 01 Jun 2022 05:13:04 GMT","etag":"\"3147526947\"","expires":"Wed, 08 Jun 2022 05:13:04 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (oxr/8370)","vary":"Accept-Encoding","x-cache":"HIT"},"mimeType":"text/html","connectionReused":false,"connectionId":13,"remoteIPAddress":"93.184.216.34","remotePort":443,"fromDiskCache":false,"fromServiceWorker":false,"fromPrefetchCache":false,"encodedDataLength":193,"timing":{"requestTime":55583.965293,"proxyStart":-1,"proxyEnd":-1,"dnsStart":23.354,"dnsEnd":24.929,"connectStart":24.929,"connectEnd":562.419,"sslStart":194.584,"sslEnd":562.39,"workerStart":-1,"workerReady":-1,"workerFetchStart":-1,"workerRespondWithSettled":-1,"sendStart":562.684,"sendEnd":562.847,"pushStart":0,"pushEnd":0,"receiveHeadersEnd":733.232},"responseTime":1.654060384785279e+12,"protocol":"h2","securityState":"secure","securityDetails":{"protocol":"TLS 1.3","keyExchange":"","keyExchangeGroup":"P-256","cipher":"AES_256_GCM","certificateId":0,"subjectName":"www.example.org","sanList":["www.example.org","example.net","example.edu","example.com","example.org","www.example.com","www.example.edu","www.example.net"],"issuer":"DigiCert TLS RSA SHA256 2020 CA1","validFrom":1647216000,"validTo":1678838399,"signedCertificateTimestampList":[],"certificateTransparencyCompliance":"unknown"}},"hasExtraInfo":true,"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":14,"result":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkAlmostIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"8E275FC88E861B68C971049B227D2C13","name":"networkIdle","timestamp":55583.942883},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameStartedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"init","timestamp":55584.704893},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}}}
2022/06/01 05:13:04 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"412DC8003B59C78196B600AFBB7C0231","type":"page","title":"https://example.com","url":"https://example.com/","attached":true,"canAccessOpener":false,"browserContextId":"1488447107EADF5939A51B3C61EA1F3B"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextsCleared","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameNavigated","params":{"frame":{"id":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","url":"https://example.com/","domainAndRegistry":"example.com","securityOrigin":"https://example.com","mimeType":"text/html","adFrameStatus":{"adFrameType":"none"},"secureContextType":"Secure","crossOriginIsolatedContextType":"NotIsolated","gatedAPIFeatures":[]},"type":"Navigation"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Network.dataReceived","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.707376,"dataLength":1256,"encodedDataLength":0},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Runtime.executionContextCreated","params":{"context":{"id":2,"origin":"https://example.com","name":"","uniqueId":"964267974370550981.7417314755448916594","auxData":{"isDefault":true,"type":"default","frameId":"412DC8003B59C78196B600AFBB7C0231"}}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":15,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
2022/06/01 05:13:04 <- {"method":"Network.loadingFinished","params":{"requestId":"C9980CD6274A43FBE9998E7B5E95629B","timestamp":55584.699583,"encodedDataLength":850,"shouldReportCorbBlocking":false},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"CSS.styleSheetAdded","params":{"header":{"styleSheetId":"38.0","frameId":"412DC8003B59C78196B600AFBB7C0231","sourceURL":"https://example.com/","origin":"regular","title":"","ownerNode":9,"disabled":false,"isInline":true,"isMutable":false,"isConstructed":false,"startLine":8,"startColumn":27,"length":650,"endLine":34,"endColumn":4}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":15,"result":{"root":{"nodeId":1,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":2,"parentId":1,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":3,"parentId":1,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":4,"parentId":3,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":5,"parentId":3,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.loadEventFired","params":{"timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"load","timestamp":55584.726037},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.frameStoppedLoading","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231"},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.documentUpdated","params":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.domContentEventFired","params":{"timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"Page.lifecycleEvent","params":{"frameId":"412DC8003B59C78196B600AFBB7C0231","loaderId":"C9980CD6274A43FBE9998E7B5E95629B","name":"DOMContentLoaded","timestamp":55584.726372},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":16,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getDocument","params":{}}
2022/06/01 05:13:04 <- {"id":16,"result":{"root":{"nodeId":6,"backendNodeId":3,"nodeType":9,"nodeName":"#document","localName":"","nodeValue":"","childNodeCount":2,"children":[{"nodeId":7,"parentId":6,"backendNodeId":10,"nodeType":10,"nodeName":"html","localName":"","nodeValue":"","publicId":"","systemId":""},{"nodeId":8,"parentId":6,"backendNodeId":4,"nodeType":1,"nodeName":"HTML","localName":"html","nodeValue":"","childNodeCount":2,"children":[{"nodeId":9,"parentId":8,"backendNodeId":11,"nodeType":1,"nodeName":"HEAD","localName":"head","nodeValue":"","childNodeCount":5,"attributes":[]},{"nodeId":10,"parentId":8,"backendNodeId":12,"nodeType":1,"nodeName":"BODY","localName":"body","nodeValue":"","childNodeCount":1,"attributes":[]}],"attributes":[],"frameId":"412DC8003B59C78196B600AFBB7C0231"}],"documentURL":"https://example.com/","baseURL":"https://example.com/","xmlVersion":"","compatibilityMode":"NoQuirksMode"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":17,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
2022/06/01 05:13:04 <- {"id":17,"result":{"searchId":"38.1","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":18,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.1","fromIndex":0,"toIndex":1}}
2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":10,"nodes":[{"nodeId":11,"parentId":10,"backendNodeId":5,"nodeType":1,"nodeName":"DIV","localName":"div","nodeValue":"","childNodeCount":3,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"method":"DOM.setChildNodes","params":{"parentId":11,"nodes":[{"nodeId":12,"parentId":11,"backendNodeId":13,"nodeType":1,"nodeName":"H1","localName":"h1","nodeValue":"","childNodeCount":1,"children":[{"nodeId":13,"parentId":12,"backendNodeId":6,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"Example Domain"}],"attributes":[]},{"nodeId":14,"parentId":11,"backendNodeId":14,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"children":[{"nodeId":15,"parentId":14,"backendNodeId":7,"nodeType":3,"nodeName":"#text","localName":"","nodeValue":"This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission."}],"attributes":[]},{"nodeId":16,"parentId":11,"backendNodeId":15,"nodeType":1,"nodeName":"P","localName":"p","nodeValue":"","childNodeCount":1,"attributes":[]}]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 <- {"id":18,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":19,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.performSearch","params":{"query":"body \u003e div \u003e p:nth-child(2)"}}
2022/06/01 05:13:04 <- {"id":19,"result":{"searchId":"38.2","resultCount":1},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":20,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.getSearchResults","params":{"searchId":"38.2","fromIndex":0,"toIndex":1}}
2022/06/01 05:13:04 <- {"id":20,"result":{"nodeIds":[14]},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":21,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"DOM.resolveNode","params":{"nodeId":14}}
2022/06/01 05:13:04 <- {"id":21,"result":{"object":{"type":"object","subtype":"node","className":"HTMLParagraphElement","description":"p","objectId":"5836470857787727263.2.1"}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":22,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.callFunctionOn","params":{"functionDeclaration":"function text() {\n    if (this.offsetWidth || this.offsetHeight || this.getClientRects().length) {\n        return this.innerText;\n    }\n    return '';\n}\n","objectId":"5836470857787727263.2.1","silent":true,"returnByValue":true}}
2022/06/01 05:13:04 <- {"id":22,"result":{"result":{"type":"string","value":"This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission."}},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
2022/06/01 05:13:04 -> {"id":23,"sessionId":"0F3F699532F154F976242DBF4AC0DFD2","method":"Runtime.releaseObject","params":{"objectId":"5836470857787727263.2.1"}}
2022/06/01 05:13:04 <- {"id":23,"result":{},"sessionId":"0F3F699532F154F976242DBF4AC0DFD2"}
This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

What did you see instead?

It stuck and timeout (aws lambda timeout).

Log on aws lambda.

START RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Version: $LATEST
2022/06/01 04:13:38 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"b08ebc85-843e-44d0-bcb3-42ea7bd5a146","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/06/01 04:13:38 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"edd4fc71-a027-494d-8a43-eba9239cd0bb","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/06/01 04:13:38 <- {"id":1,"result":{}}
2022/06/01 04:13:38 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","flatten":true}}
2022/06/01 04:13:38 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"}}}
2022/06/01 04:13:38 <- {"method":"Target.attachedToTarget","params":{"sessionId":"26082765C593E944A783B371227B397A","targetInfo":{"targetId":"D86C302EE0B44DAED223378F7C17BB78","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"0C22F0D3B02A5878B60E8F3500944994"},"waitingForDebugger":false}}
2022/06/01 04:13:38 <- {"id":2,"result":{"sessionId":"26082765C593E944A783B371227B397A"}}
2022/06/01 04:13:38 -> {"id":3,"sessionId":"26082765C593E944A783B371227B397A","method":"Runtime.enable"}
END RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1
REPORT RequestId: f829c5ca-e956-4bfe-bef0-dae4f36d50d1  Duration: 20019.61 ms   Billed Duration: 20552 ms   Memory Size: 10240 MB   Max Memory Used: 125 MB Init Duration: 532.33 ms    
2022-06-01T04:13:57.470Z f829c5ca-e956-4bfe-bef0-dae4f36d50d1 Task timed out after 20.02 seconds
Andiedie commented 2 years ago

I have seen #298 but did't find anything useful, because I use docker image in aws lambda.

For more infomation, I run chromedp/headless-shell directly on ec2 and aws lambda using the same args in golang code.

FROM chromedp/headless-shell

ENTRYPOINT [ "/headless-shell/headless-shell", "--no-first-run", "--no-default-browser-check", "--no-sandbox", "--disable-gpu", "--headless", "--hide-scrollbars", "--mute-audio", "--no-zygote", "--user-data-dir=/tmp/chrome-user-data-dir", "--homedir=/tmp/chrome-home", "--data-path=/tmp/chrome-data-path", "--disk-cache-dir=/tmp/chrome-disk-cache-dir", "--remote-debugging-port=9222", "--remote-debugging-address=0.0.0.0", "--disable-dev-shm-usage", "--enable-features=NetworkService,NetworkServiceInProcess" ]

Log on ec2 (Works)

$ docker run --rm -it test
[0601/050439.195829:INFO:content_main_runner_impl.cc(1144)] Chrome is running in full browser mode.

DevTools listening on ws://0.0.0.0:9222/devtools/browser/a3478411-e6c4-41f1-bb64-44e0f4b6dbcc
[0601/050439.206209:ERROR:devtools_http_handler.cc(291)] Error writing DevTools active port to file
[0601/050439.213214:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0601/050439.213275:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is angle, ANGLE is swiftshader
[0601/050439.213335:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0601/050439.213357:ERROR:gpu_info_collector.cc(93)] gl::init::CreateGLContext failed
[0601/050439.213363:ERROR:gpu_info_collector.cc(348)] Could not create context for info collection.
[0601/050439.213380:ERROR:gpu_init.cc(86)] CollectGraphicsInfo failed.
[0601/050439.215571:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization
[0601/050439.234087:WARNING:child_process_launcher_helper_posix.cc(83)] Ignoring invalid file snapshot_blob.bin
[0601/050439.243838:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0601/050439.243902:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is angle, ANGLE is swiftshader
[0601/050439.243968:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0601/050439.244002:ERROR:gpu_info_collector.cc(93)] gl::init::CreateGLContext failed
[0601/050439.244020:ERROR:gpu_info_collector.cc(348)] Could not create context for info collection.
[0601/050439.244045:ERROR:gpu_init.cc(86)] CollectGraphicsInfo failed.
[0601/050439.247142:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization
[0601/050439.256784:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is disabled, ANGLE is

Log on aws lambda (does't work)

START RequestId: 201b0ec5-c1c1-4bb8-a055-c2658305d6ea Version: $LATEST
[0601/045722.126366:INFO:content_main_runner_impl.cc(1144)] Chrome is running in full browser mode.
[0601/045722.134623:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.134609:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
prctl(PR_SET_NO_NEW_PRIVS) failed
DevTools listening on ws://0.0.0.0:9222/devtools/browser/8277e70c-1ad8-4465-9fc0-ea1258c28c3d
[0601/045722.148364:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.148388:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 1 time(s)
[0601/045722.161952:WARNING:child_process_launcher_helper_posix.cc(83)] Ignoring invalid file snapshot_blob.bin
prctl(PR_SET_NO_NEW_PRIVS) failed
prctl(PR_SET_NO_NEW_PRIVS) failed
[0601/045722.164563:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.185013:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.185042:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 2 time(s)
prctl(PR_SET_NO_NEW_PRIVS) failed
[0601/045722.186685:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.201714:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.201756:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 3 time(s)
prctl(PR_SET_NO_NEW_PRIVS) failed
[0601/045722.203623:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.218695:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.218727:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 4 time(s)
prctl(PR_SET_NO_NEW_PRIVS) failed
[0601/045722.220732:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.235865:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.235899:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 5 time(s)
prctl(PR_SET_NO_NEW_PRIVS) failed
[0601/045722.237839:ERROR:file_util_posix.cc(932)] fallocate: Operation not permitted (1)
[0601/045722.253183:ERROR:gpu_process_host.cc(973)] GPU process exited unexpectedly: exit_code=133
[0601/045722.253214:WARNING:gpu_process_host.cc(1273)] The GPU process has crashed 6 time(s)
[0601/045722.253239:FATAL:gpu_data_manager_impl_private.cc(450)] GPU process isn't usable. Goodbye.
#0 0x55a64ee43369 (/headless-shell/headless-shell+0x6dbb368)
#1 0x55a64ed77023 (/headless-shell/headless-shell+0x6cef022)
#2 0x55a64ed8db80 (/headless-shell/headless-shell+0x6d05b7f)
#3 0x55a64cefa7a7 (/headless-shell/headless-shell+0x4e727a6)
#4 0x55a64cef712c (/headless-shell/headless-shell+0x4e6f12b)
#5 0x55a64cef58cb (/headless-shell/headless-shell+0x4e6d8ca)
#6 0x55a64cf01829 (/headless-shell/headless-shell+0x4e79828)
#7 0x55a64cf02ff4 (/headless-shell/headless-shell+0x4e7aff3)
#8 0x55a64cd1eb7b (/headless-shell/headless-shell+0x4c96b7a)
#9 0x55a64c9d1ede (/headless-shell/headless-shell+0x4949edd)
#10 0x55a64f414b98 (/headless-shell/headless-shell+0x738cb97)
#11 0x55a64f41ac23 (/headless-shell/headless-shell+0x7392c22)
#12 0x55a64f16c47d (/headless-shell/headless-shell+0x70e447c)
#13 0x55a64f420a2d (/headless-shell/headless-shell+0x7398a2c)
#14 0x55a64f422949 (/headless-shell/headless-shell+0x739a948)
#15 0x55a64f167545 (/headless-shell/headless-shell+0x70df544)
#16 0x55a64f1682c7 (/headless-shell/headless-shell+0x70e02c6)
#17 0x55a64c984d40 (/headless-shell/headless-shell+0x48fcd3f)
#18 0x55a64f191fd0 (/headless-shell/headless-shell+0x7109fcf)
#19 0x55a64ecbcfb2 (/headless-shell/headless-shell+0x6c34fb1)
#20 0x55a64eddf5b9 (/headless-shell/headless-shell+0x6d575b8)
#21 0x55a64ee044b4 (/headless-shell/headless-shell+0x6d7c4b3)
#22 0x55a64ee03b5e (/headless-shell/headless-shell+0x6d7bb5d)
#23 0x55a64ee04cf2 (/headless-shell/headless-shell+0x6d7ccf1)
#24 0x55a64ee7c8f5 (/headless-shell/headless-shell+0x6df48f4)
#25 0x55a64ee05299 (/headless-shell/headless-shell+0x6d7d298)
#26 0x55a64edc3da0 (/headless-shell/headless-shell+0x6d3bd9f)
#27 0x55a64cd3be95 (/headless-shell/headless-shell+0x4cb3e94)
#28 0x55a64cd3d872 (/headless-shell/headless-shell+0x4cb5871)
#29 0x55a64ed6d54b (/headless-shell/headless-shell+0x6ce554a)
#30 0x55a64d689b33 (/headless-shell/headless-shell+0x5601b32)
#31 0x55a64d68b2b4 (/headless-shell/headless-shell+0x56032b3)
#32 0x55a64d68ad30 (/headless-shell/headless-shell+0x5602d2f)
#33 0x55a64d688747 (/headless-shell/headless-shell+0x5600746)
#34 0x55a64d68882e (/headless-shell/headless-shell+0x560082d)
#35 0x55a64ed68e9d (/headless-shell/headless-shell+0x6ce0e9c)
#36 0x55a64ed68cfe (/headless-shell/headless-shell+0x6ce0cfd)
#37 0x55a64ed68a45 (/headless-shell/headless-shell+0x6ce0a44)
#38 0x7fd055406d0a __libc_start_main
#39 0x55a64b5f01ea _start
Task trace:
#0 0x55a64f19237c (/headless-shell/headless-shell+0x710a37b)
END RequestId: 201b0ec5-c1c1-4bb8-a055-c2658305d6ea
REPORT RequestId: 201b0ec5-c1c1-4bb8-a055-c2658305d6ea  Duration: 180.60 ms Billed Duration: 181 ms Memory Size: 10240 MB   Max Memory Used: 84 MB  
RequestId: 201b0ec5-c1c1-4bb8-a055-c2658305d6ea Error: Runtime exited with error: signal: trace/breakpoint trap
Runtime.ExitError
Andiedie commented 2 years ago

For more infomation, aws lambda only allows to write file in /tmp, other path is read-only.

ZekeLu commented 2 years ago

I have tried the following images:

Except for 102.0.5005.63 (which is tagged as latest as of now and is used in this issue report), all the others can't even start (no CDP messages, no output from the headless-shell).

So, unfortunately, chromedp/headless-shell just does not work on AWS Lambda.

Updated: The listed images failed the same way after I increase the timeout of the AWS Lambda function to 20s.

Andiedie commented 2 years ago

Thanks for quick reply.

Do you have any idea about why it fails, or any other way to run chromedp on AWS Lambda?

Andiedie commented 2 years ago

I try to change image of Dockerfile in repo to chromedp/headless-shell:101.0.4951.7

FROM golang:alpine AS builder

RUN apk add git

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o main

FROM chromedp/headless-shell:101.0.4951.7

RUN apt-get update && \
    apt-get install -y dumb-init musl-dev && \
    ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1 &&\
    rm -rf /var/lib/apt/lists/*

COPY --from=builder /app/main .

ENTRYPOINT [ "dumb-init", "--" ]
CMD [ "./main" ]

It behave the same as chromedp/headless-shell:latest (stuck after last log {"id":3,"sessionId":"341D47C727A583D1E232D49953E705FE","method":"Runtime.enable"})

START RequestId: ce42768d-8a0b-4169-a25c-17e4b2317919 Version: $LATEST
2022/06/01 12:00:28 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/06/01 12:00:29 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"3512005a-c14b-41b8-8a74-cf09d1a8a154","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/06/01 12:00:29 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"6189F03BE61B2DD54665C50FF111DA97","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"5C128ACB50B7AFEB7AB7FA9D81BF3777"}}}
2022/06/01 12:00:29 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"c392f8af-ca14-48c9-ac00-46113b2210a8","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/06/01 12:00:29 <- {"id":1,"result":{}}
2022/06/01 12:00:29 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"6189F03BE61B2DD54665C50FF111DA97","flatten":true}}
2022/06/01 12:00:29 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"6189F03BE61B2DD54665C50FF111DA97","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"5C128ACB50B7AFEB7AB7FA9D81BF3777"}}}
2022/06/01 12:00:29 <- {"method":"Target.attachedToTarget","params":{"sessionId":"341D47C727A583D1E232D49953E705FE","targetInfo":{"targetId":"6189F03BE61B2DD54665C50FF111DA97","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"5C128ACB50B7AFEB7AB7FA9D81BF3777"},"waitingForDebugger":false}}
2022/06/01 12:00:29 <- {"id":2,"result":{"sessionId":"341D47C727A583D1E232D49953E705FE"}}
2022/06/01 12:00:29 -> {"id":3,"sessionId":"341D47C727A583D1E232D49953E705FE","method":"Runtime.enable"}
END RequestId: ce42768d-8a0b-4169-a25c-17e4b2317919
REPORT RequestId: ce42768d-8a0b-4169-a25c-17e4b2317919  Duration: 20016.95 ms   Billed Duration: 21390 ms   Memory Size: 10240 MB   Max Memory Used: 124 MB Init Duration: 1372.23 ms   
2022-06-01T12:00:35.069Z ce42768d-8a0b-4169-a25c-17e4b2317919 Task timed out after 20.02 seconds
ZekeLu commented 2 years ago

You're correct. I got the same result after I increase the timeout of the AWS Lambda function.

As the "Log on aws lambda (does't work)" shows, it failed because GPU process isn't usable. Goodbye..

I have tried passing the following options to headless-shell, but it still does not work:

And I just tried chromedp/headless-shell:96.0.4664.93 (just because I have pulled it before). It failed with another reason: [0601/131811.991312:FATAL:viz_main_impl.cc(239)] Check failed: gl::GetGLImplementation() == gl::kGLImplementationDisabled (0 vs. 8).

It seems there are too many issues to run chromedp/headless-shell. I don't have any idea about how to address them.

Andiedie commented 2 years ago

I found that adding --single-process flag will crash the container.

Without --single-process (Works)

$ docker run --rm -it chromedp/headless-shell --no-first-run --no-default-browser-check --no-sandbox --disable-gpu --headless --hide-scrollbars --mute-audio --no-zygote --user-data-dir=/tmp/chrome-user-data-dir --homedir=/tmp/chrome-home --data-path=/tmp/chrome-data-path --disk-cache-dir=/tmp/chrome-disk-cache-dir --remote-debugging-port=9222 --remote-debugging-address=0.0.0.0 --disable-dev-shm-usage --enable-features=NetworkService,NetworkServiceInProcess --disable-gpu-shader-disk-cache disable-gpu-early-init --disable-gpu-compositing  --disable-gpu-sandbox
[0602/025438.019167:INFO:content_main_runner_impl.cc(1144)] Chrome is running in full browser mode.

DevTools listening on ws://0.0.0.0:9222/devtools/browser/bdae72ef-ffee-4600-b6a6-fe1f7d9b3c75
[0602/025438.029201:ERROR:devtools_http_handler.cc(291)] Error writing DevTools active port to file
[0602/025438.036425:WARNING:child_process_launcher_helper_posix.cc(83)] Ignoring invalid file snapshot_blob.bin
[0602/025438.036966:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025438.037037:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is angle, ANGLE is swiftshader
[0602/025438.037140:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025438.037255:ERROR:gpu_info_collector.cc(93)] gl::init::CreateGLContext failed
[0602/025438.037299:ERROR:gpu_info_collector.cc(348)] Could not create context for info collection.
[0602/025438.037461:ERROR:gpu_init.cc(86)] CollectGraphicsInfo failed.
[0602/025438.045644:ERROR:viz_main_impl.cc(186)] Exiting GPU process due to errors during initialization
[0602/025438.048441:WARNING:directory_listing.cc(24)] Missing resource: directory listing header
[0602/025438.068671:ERROR:gpu_init.cc(481)] Passthrough is not supported, GL is disabled, ANGLE is
[0602/025438.086773:INFO:CONSOLE(1)] "Uncaught ReferenceError: start is not defined", source: file:/// (1)
[0602/025438.087195:INFO:CONSOLE(2)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (2)
[0602/025438.087607:INFO:CONSOLE(3)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (3)
[0602/025438.088018:INFO:CONSOLE(4)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (4)
[0602/025438.088105:INFO:CONSOLE(5)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (5)
[0602/025438.088502:INFO:CONSOLE(6)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (6)
[0602/025438.088653:INFO:CONSOLE(7)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (7)
[0602/025438.088985:INFO:CONSOLE(8)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (8)
[0602/025438.089231:INFO:CONSOLE(9)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (9)
[0602/025438.089581:INFO:CONSOLE(10)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (10)
[0602/025438.090077:INFO:CONSOLE(11)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (11)
[0602/025438.090400:INFO:CONSOLE(12)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (12)
[0602/025438.090477:INFO:CONSOLE(13)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (13)
[0602/025438.090822:INFO:CONSOLE(14)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (14)
[0602/025438.091078:INFO:CONSOLE(15)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (15)
[0602/025438.091298:INFO:CONSOLE(16)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (16)
[0602/025438.091549:INFO:CONSOLE(17)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (17)
[0602/025438.091817:INFO:CONSOLE(18)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (18)
[0602/025438.092270:INFO:CONSOLE(19)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (19)
[0602/025438.092487:INFO:CONSOLE(20)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (20)
[0602/025438.092576:INFO:CONSOLE(21)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (21)
[0602/025438.092816:INFO:CONSOLE(22)] "Uncaught ReferenceError: addRow is not defined", source: file:/// (22)

With --single-process (Doesn't work)

$ docker run --rm -it chromedp/headless-shell --no-first-run --no-default-browser-check --no-sandbox --disable-gpu --headless --hide-scrollbars --mute-audio --no-zygote --user-data-dir=/tmp/chrome-user-data-dir --homedir=/tmp/chrome-home --data-path=/tmp/chrome-data-path --disk-cache-dir=/tmp/chrome-disk-cache-dir --remote-debugging-port=9222 --remote-debugging-address=0.0.0.0 --disable-dev-shm-usage --enable-features=NetworkService,NetworkServiceInProcess --disable-gpu-shader-disk-cache disable-gpu-early-init --disable-gpu-compositing  --disable-gpu-sandbox --single-process
[0602/025632.802722:INFO:content_main_runner_impl.cc(1144)] Chrome is running in full browser mode.

DevTools listening on ws://0.0.0.0:9222/devtools/browser/680709ac-91b4-48a5-b9bf-5cba140835b6
[0602/025632.812484:ERROR:devtools_http_handler.cc(291)] Error writing DevTools active port to file
[0602/025632.813066:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025632.813149:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025632.813175:ERROR:gpu_info_collector.cc(93)] gl::init::CreateGLContext failed
[0602/025632.813192:ERROR:gpu_info_collector.cc(348)] Could not create context for info collection.
[0602/025632.813278:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025632.813302:ERROR:gpu_info_collector.cc(93)] gl::init::CreateGLContext failed
[0602/025632.813337:ERROR:gpu_info_collector.cc(348)] Could not create context for info collection.
[0602/025632.824407:ERROR:gl_context_egl.cc(355)] eglCreateContext failed with error EGL_BAD_ATTRIBUTE
[0602/025632.824445:ERROR:gpu_channel_manager.cc(831)] ContextResult::kFatalFailure: Failed to create shared context for virtualization.
[0602/025632.824461:ERROR:shared_image_stub.cc(537)] SharedImageStub: unable to create context
[0602/025632.824489:ERROR:gpu_channel.cc(572)] GpuChannel: Failed to create SharedImageStub
[0602/025632.842847:FATAL:gpu_client.cc(107)] Check failed: channel_handle.is_valid() == status == GpuHostImpl::EstablishChannelStatus::kSuccess (0 vs. 1)
#0 0x55dc66968369 (/headless-shell/headless-shell+0x6dbb368)
#1 0x55dc6689c023 (/headless-shell/headless-shell+0x6cef022)
#2 0x55dc668b2b80 (/headless-shell/headless-shell+0x6d05b7f)
#3 0x55dc668b366e (/headless-shell/headless-shell+0x6d0666d)
#4 0x55dc683aaa73 (/headless-shell/headless-shell+0x87fda72)
#5 0x55dc683abbb1 (/headless-shell/headless-shell+0x87febb0)
#6 0x55dc683abaf9 (/headless-shell/headless-shell+0x87feaf8)
#7 0x55dc683ae143 (/headless-shell/headless-shell+0x8801142)
#8 0x55dc683b046c (/headless-shell/headless-shell+0x880346b)
#9 0x55dc683b03b0 (/headless-shell/headless-shell+0x88033af)
#10 0x55dc645e666a (/headless-shell/headless-shell+0x4a39669)
#11 0x55dc66c8f7f9 (/headless-shell/headless-shell+0x70e27f8)
#12 0x55dc66c97aa9 (/headless-shell/headless-shell+0x70eaaa8)
#13 0x55dc66c9126d (/headless-shell/headless-shell+0x70e426c)
#14 0x55dc66c9d3b7 (/headless-shell/headless-shell+0x70f03b6)
#15 0x55dc66c9c85c (/headless-shell/headless-shell+0x70ef85b)
#16 0x55dc66c97aa9 (/headless-shell/headless-shell+0x70eaaa8)
#17 0x55dc66c8c9fc (/headless-shell/headless-shell+0x70df9fb)
#18 0x55dc66c8d515 (/headless-shell/headless-shell+0x70e0514)
#19 0x55dc66c8d2b9 (/headless-shell/headless-shell+0x70e02b8)
#20 0x55dc644a9d40 (/headless-shell/headless-shell+0x48fcd3f)
#21 0x55dc66cb6fd0 (/headless-shell/headless-shell+0x7109fcf)
#22 0x55dc667e1fb2 (/headless-shell/headless-shell+0x6c34fb1)
#23 0x55dc669045b9 (/headless-shell/headless-shell+0x6d575b8)
#24 0x55dc669294b4 (/headless-shell/headless-shell+0x6d7c4b3)
#25 0x55dc66928b5e (/headless-shell/headless-shell+0x6d7bb5d)
#26 0x55dc66929cf2 (/headless-shell/headless-shell+0x6d7ccf1)
#27 0x55dc669a18f5 (/headless-shell/headless-shell+0x6df48f4)
#28 0x55dc6692a299 (/headless-shell/headless-shell+0x6d7d298)
#29 0x55dc668e8da0 (/headless-shell/headless-shell+0x6d3bd9f)
#30 0x55dc64860e95 (/headless-shell/headless-shell+0x4cb3e94)
#31 0x55dc64862872 (/headless-shell/headless-shell+0x4cb5871)
#32 0x55dc6689254b (/headless-shell/headless-shell+0x6ce554a)
#33 0x55dc651aeb33 (/headless-shell/headless-shell+0x5601b32)
#34 0x55dc651b02b4 (/headless-shell/headless-shell+0x56032b3)
#35 0x55dc651afd30 (/headless-shell/headless-shell+0x5602d2f)
#36 0x55dc651ad747 (/headless-shell/headless-shell+0x5600746)
#37 0x55dc651ad82e (/headless-shell/headless-shell+0x560082d)
#38 0x55dc6688de9d (/headless-shell/headless-shell+0x6ce0e9c)
#39 0x55dc6688dcfe (/headless-shell/headless-shell+0x6ce0cfd)
#40 0x55dc6688da45 (/headless-shell/headless-shell+0x6ce0a44)
#41 0x7fd6878a0d0a __libc_start_main
#42 0x55dc631151ea _start
Task trace:
#0 0x55dc66cb737c (/headless-shell/headless-shell+0x710a37b)
#1 0x55dc667e21a1 (/headless-shell/headless-shell+0x6c351a0)
#2 0x55dc667df1ba (/headless-shell/headless-shell+0x6c321b9)
#3 0x55dc66cb6a49 (/headless-shell/headless-shell+0x7109a48)
#4 0x55dc667dd0fb (/headless-shell/headless-shell+0x6c300fa)
Task trace buffer limit hit, update PendingTask::kTaskBacktraceLength to increase.
Andiedie commented 2 years ago

IT WORKS! Checkout this example repo.

image

I just gave up chromedp/headless-shell and turned into debian with manually installed chromium. Launch chromium with args:

FROM golang AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o main

FROM debian AS runtime

RUN apt-get update && apt-get install -y chromium

COPY --from=builder /app/main .

ENTRYPOINT [ "./main" ]
ZekeLu commented 2 years ago

I'm going to reopen this issue in the hope that we can make chromedp/headless-shell work in aws lambda (since chromium works).

ZekeLu commented 2 years ago

@Andiedie Would you like to give https://hub.docker.com/r/zeke/headless-shell/tags a try? It should work on AWS Lambda now. Please note that two more flags are required:

See chromedp/docker-headless-shell#18 for more information.

Andiedie commented 2 years ago

Thanks for updating.

But sorry I am a little bit busy now, I will try it as soon as I can.

Andiedie commented 2 years ago

Hey there, I have tried running app on lambda with image zeke/headless-shell:102.0.5005.115 and everything works.

FROM golang AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o main

FROM zeke/headless-shell:102.0.5005.115

COPY --from=builder /app/main .

ENTRYPOINT [ "./main" ]
ZekeLu commented 2 years ago

Thank you for confirming! We will fix chromedp/headless-shell later.

celianiu commented 2 years ago

@Andiedie Would you like to give https://hub.docker.com/r/zeke/headless-shell/tags a try? It should work on AWS Lambda now. Please note that two more flags are required:

  • --use-gl=angle
  • --use-angle=swiftshader

See chromedp/docker-headless-shell#18 for more information.

Hey there, I got similar issue when running on aws lambda, which is stuck after {"id":3,"sessionId":"341D47C727A583D1E232D49953E705FE","method":"Runtime.enable"}) My dockerfile is very similar to @Andiedie's one 

FROM golang:1.18-alpine AS builder

WORKDIR /app

RUN apk add go git
RUN go env -w GOPROXY=direct

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN go build -o main

FROM zeke/headless-shell:102.0.5005.115

RUN apt-get update; apt-get install -y dumb-init musl-dev ca-certificates
COPY --from=builder /app/main .
ENTRYPOINT ["./main"]

I'm not sure where to add these two flags you mentioned --use-gl=angle --use-angle=swiftshader

since run on AWS lambda is just build the container, push to the ECR repository and then update and invoke the function?

I'm also curious if this fix has been added to the chromedp/headless-shell:latest?

Thank you very much for your time and help!

ZekeLu commented 2 years ago

The entry point is your app ./main. So the browser is started by your app. You can specify --use-gl=angle and --use-angle=swiftshader with chromedp.Flag in your code:

    opts := append(chromedp.DefaultExecAllocatorOptions[:],
        chromedp.Flag("use-gl", "angle"),
        chromedp.Flag("use-angle", "swiftshader"),
    )

I'm also curious if this fix has been added to the chromedp/headless-shell:latest?

Yes. It seems that there is something wrong with the builder. We are looking into it.

ZekeLu commented 2 years ago

@celianiu The docker images have just been updated. You can try them if you like.

celianiu commented 2 years ago

Thanks for the quick reply! I just tested will the latest version with the following flags:

    opts := []chromedp.ExecAllocatorOption{
        chromedp.Headless,
        chromedp.NoSandbox,
        chromedp.WindowSize(1200, 1000),
        // chromedp.Flag("disable-setuid-sandbox", true),
        chromedp.Flag("disable-dev-shm-usage", true),
        // chromedp.Flag("single-process", true),
        //chromedp.Flag("no-zygote", true),
        chromedp.Flag("use-gl", "angle"),
        chromedp.Flag("use-angle", "swiftshader"),
    }

I noticed that disable-dev-shm-usage seems also necessary, and the no-zygote flag seems also cause the program stuck.(Not sure if I tested correctly) And when I removed that flag, I received the context canceled error. 

2022/07/05 13:58:08 -> {"id":1,"method":"Target.setDiscoverTargets","params":{"discover":true}}
2022/07/05 13:58:08 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"187B4DA22EBCC3C85FA292F447A83B2C","type":"page","title":"","url":"about:blank","attached":false,"canAccessOpener":false,"browserContextId":"7FDE17DD2B4FCF743EE7945F2F63275D"}}}
2022/07/05 13:58:08 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"dd7bf2ba-6750-496b-8d74-e63105a0fcc3","type":"browser","title":"","url":"","attached":true,"canAccessOpener":false}}}
2022/07/05 13:58:08 <- {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"e3acc6d7-c6ee-4841-970f-0d80b8a81823","type":"browser","title":"","url":"","attached":false,"canAccessOpener":false}}}
2022/07/05 13:58:08 <- {"id":1,"result":{}}
2022/07/05 13:58:08 -> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"187B4DA22EBCC3C85FA292F447A83B2C","flatten":true}}
2022/07/05 13:58:08 <- {"method":"Target.targetCrashed","params":{"targetId":"187B4DA22EBCC3C85FA292F447A83B2C","status":"failed to launch","errorCode":1002}}
2022/07/05 13:58:08 <- {"method":"Target.targetInfoChanged","params":{"targetInfo":{"targetId":"187B4DA22EBCC3C85FA292F447A83B2C","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"7FDE17DD2B4FCF743EE7945F2F63275D"}}}
2022/07/05 13:58:08 <- {"method":"Target.attachedToTarget","params":{"sessionId":"6A3B82D2AC85C87D8632529FF8DC2663","targetInfo":{"targetId":"187B4DA22EBCC3C85FA292F447A83B2C","type":"page","title":"","url":"","attached":true,"canAccessOpener":false,"browserContextId":"7FDE17DD2B4FCF743EE7945F2F63275D"},"waitingForDebugger":false}}
2022/07/05 13:58:08 <- {"id":2,"result":{"sessionId":"6A3B82D2AC85C87D8632529FF8DC2663"}}
2022/07/05 13:58:08 -> {"id":3,"sessionId":"6A3B82D2AC85C87D8632529FF8DC2663","method":"Runtime.enable"}
2022/07/05 13:58:09 context canceled
END RequestId: 20f09141-7911-41e9-8ff8-94ef51bdb10f
REPORT RequestId: 20f09141-7911-41e9-8ff8-94ef51bdb10f  Duration: 1160.96 ms    Billed Duration: 2556 ms    Memory Size: 512 MB Max Memory Used: 122 MB Init Duration: 1394.25 ms   
RequestId: 20f09141-7911-41e9-8ff8-94ef51bdb10f Error: Runtime exited with error: exit status 1
Runtime.ExitError

I saw you mentioned this error in FAQ, but I didn't get this when I ran locally, so I was wondering if this error could be caused by lambda? or if there might be anything wrong with my flag setting?

ZekeLu commented 2 years ago

Can you provide a minimal reproducer? You can create one based on https://github.com/Andiedie/chromedp-aws-lambda-example.

celianiu commented 2 years ago

Thanks! I got it fixed! It's the flag-setting issue. It works with following:

    opts := []chromedp.ExecAllocatorOption{
        chromedp.Headless,
        chromedp.NoSandbox,
        chromedp.WindowSize(1200, 1000),
        chromedp.Flag("disable-setuid-sandbox", true),
        chromedp.Flag("disable-dev-shm-usage", true),
        chromedp.Flag("single-process", true),
        chromedp.Flag("no-zygote", true),
        chromedp.Flag("use-gl", "angle"),
        chromedp.Flag("use-angle", "swiftshader"),
    }

Other than the two flags you mentioned, the rest of the flags seems also necessary to run on lambda.

ZekeLu commented 2 years ago

Can you remove the two options (use-gl and use-angle) and try again? Maybe they are not needed. Thank you!

celianiu commented 2 years ago

Sorry for the late reply! It works fine without that two flags (use-gl and use-angle). Only require these:

    opts := []chromedp.ExecAllocatorOption{
        chromedp.Headless,
        chromedp.NoSandbox,
        chromedp.WindowSize(1200, 1000),
        chromedp.Flag("disable-setuid-sandbox", true),
        chromedp.Flag("disable-dev-shm-usage", true),
        chromedp.Flag("single-process", true),
        chromedp.Flag("no-zygote", true),
    }

Again, thank you very much for your time and support!

psmarcin commented 6 months ago

@celianiu thanks so much for providing your solution. It still works!