[BUG] Large WebGL Canvas Screenshot crashes Chromium

deldrid1 commented 3 years ago

Environment

chrome-aws-lambda Version: 5.5.0
puppeteer / puppeteer-core Version: 5.5.0
OS: Mac
Node.js Version: v12.18.3
Lambda / GCF Runtime: nodejs12.x

Expected Behavior

This code attempts to take a very large screenshot of a map using Mapbox GL JS. Everything works fine locally but fails in AWS. I believe it is related to https://github.com/alixaxel/chrome-aws-lambda/issues/61, however, I haven't found any workarounds that actually work (it looks like most of the CLI flags, etc. have all made their way into chromium.args)...

@alixaxel - if this is "asked an answered" feel free to close, but I would love to work with you to figure out if there is a workaround that we can come up with to prevent chrome from using /tmp (assuming that is in fact the issue)!

{
    "statusCode": 200,
    "body": "OK"
}

Current Behavior

{
  "statusCode": 500,
  "body": "Internal Server Error"
}

Steps to Reproduce

Full code for reproduction available at: https://gist.github.com/deldrid1/43e4e0996060331d6b8340f237f418de.

# First go to account.mapbox.com and create a free account to get an API key and update index.html

# Demonstrate that everything works locally
yarn invoke 

# Deploy to lambda (assuming you have AWS env setup)
yarn deploy

# Watch in fail in AWS
npx sls invoke --function test

Logs

Expected Log Output

» yarn invoke                                                                                                                                                                                                                                                                                           
yarn run v1.22.10
warning package.json: No license field
$ yarn install && serverless invoke local --function test
warning package.json: No license field
warning No license field
[1/4] 🔍  Resolving packages...
success Already up-to-date.
Starting up
[0105/115447.589370:ERROR:service_utils.cc(157)] --ignore-gpu-blacklist is deprecated and will be removed in 2020Q4, use --ignore-gpu-blocklist instead.

DevTools listening on ws://127.0.0.1:62797/devtools/browser/a5ba5275-10cd-43cf-80fb-f33b76fa0590
Started up; now navigating to file:///test-lambda/index.html
[0105/115447.956162:INFO:CONSOLE(6)] "Uncaught TypeError: Cannot redefine property: webdriver", source:  (6)
Navigated to file:///test-lambda/index.html.
[0105/115449.343083:INFO:CONSOLE(2)] "Inside evaluate", source: __puppeteer_evaluation_script__ (2)
[0105/115449.343193:INFO:CONSOLE(3)] "RENDERING =  true", source: __puppeteer_evaluation_script__ (3)
[0105/115454.201044:INFO:CONSOLE(9)] "The map is idle", source: __puppeteer_evaluation_script__ (9)
taking screenshot
Screenshot taken!
Screenshot size = 0.7358760833740234 MB
Generated a blob of length 4798614 = 18.305259704589844 MB
{
    "statusCode": 200,
    "body": "OK"
}
✨  Done in 13.94s.

AWS Log Output

e.cc(431)] locale_file_path.empty() for locale 
[0105/163535.955878:ERROR:service_utils.cc(157)] --ignore-gpu-blacklist is deprecated and will be removed in 2020Q4, use --ignore-gpu-blocklist instead.
[0105/163535.956867:ERROR:service_utils.cc(157)] --ignore-gpu-blacklist is deprecated and will be removed in 2020Q4, use --ignore-gpu-blocklist instead.

DevTools listening on ws://127.0.0.1:34557/devtools/browser/79b3d0d0-778e-408f-919b-a48b0d474cd7
2021-01-05T16:35:36.026Z    9ef8809a-248d-42bf-a3b1-316b34464051    INFO    Started up; now navigating to file:///var/task/index.html
[0105/163536.040460:INFO:CONSOLE(6)] "Uncaught TypeError: Cannot redefine property: webdriver", source:  (6)
2021-01-05T16:35:37.909Z    9ef8809a-248d-42bf-a3b1-316b34464051    INFO    Navigated to file:///var/task/index.html.
[0105/163537.916999:INFO:CONSOLE(2)] "Inside evaluate", source: __puppeteer_evaluation_script__ (2)
[0105/163537.918190:INFO:CONSOLE(3)] "RENDERING =  true", source: __puppeteer_evaluation_script__ (3)
Received signal 6
#0 0x55b8c74512a9 (/tmp/chromium+0x4e2b2a8)
#1 0x55b8c73d1513 (/tmp/chromium+0x4dab512)
#2 0x55b8c7450e4b (/tmp/chromium+0x4e2ae4a)
#3 0x7fcfcd3677e0 (/usr/lib64/libpthread-2.26.so+0x117df)
#4 0x7fcfcc644b20 __GI_raise
#5 0x7fcfcc645fc8 __GI_abort
#6 0x55b8c73ff9ea (/tmp/chromium+0x4dd99e9)
#7 0x55b8c73ffa09 (/tmp/chromium+0x4dd9a08)
#8 0x55b8c73ff9f9 (/tmp/chromium+0x4dd99f8)
#9 0x55b8c836f481 (/tmp/chromium+0x5d49480)
#10 0x55b8ca3f8ebd (/tmp/chromium+0x7dd2ebc)
#11 0x55b8ca3f93ca (/tmp/chromium+0x7dd33c9)
#12 0x55b8ca3f91ab (/tmp/chromium+0x7dd31aa)
#13 0x55b8ca3f90e2 (/tmp/chromium+0x7dd30e1)
#14 0x55b8c86c3a06 (/tmp/chromium+0x609da05)
#15 0x55b8c8402fc9 (/tmp/chromium+0x5ddcfc8)
#16 0x55b8c8402cec (/tmp/chromium+0x5ddcceb)
#17 0x55b8c8483c2b (/tmp/chromium+0x5e5dc2a)
#18 0x55b8c848998e (/tmp/chromium+0x5e6398d)
#19 0x55b8c7416a76 (/tmp/chromium+0x4df0a75)
#20 0x55b8c7425cba (/tmp/chromium+0x4dffcb9)
#21 0x55b8c537a818 (/tmp/chromium+0x2d54817)
#22 0x55b8c7416a76 (/tmp/chromium+0x4df0a75)
#23 0x55b8c7427925 (/tmp/chromium+0x4e01924)
#24 0x55b8c7427628 (/tmp/chromium+0x4e01627)
#25 0x55b8c73e716a (/tmp/chromium+0x4dc1169)
#26 0x55b8c74280cd (/tmp/chromium+0x4e020cc)
#27 0x55b8c740316e (/tmp/chromium+0x4ddd16d)
#28 0x55b8c743f774 (/tmp/chromium+0x4e19773)
#29 0x55b8c7460dca (/tmp/chromium+0x4e3adc9)
#30 0x7fcfcd35d40b start_thread
#31 0x7fcfcc6fdeef __GI___clone
  r8: 0000000000000000  r9: 00007fcfbce24dd0 r10: 0000000000000008 r11: 0000000000000246
 r12: 0000000009450c00 r13: 00007fcfbce25300 r14: 00007fcfbce25150 r15: 0000000000000000
  di: 0000000000000002  si: 00007fcfbce24dd0  bp: 00007fcfbce25030  bx: 0000000000000006
  dx: 0000000000000000  ax: 0000000000000000  cx: 00007fcfcc644b20  sp: 00007fcfbce24dd0
  ip: 00007fcfcc644b20 efl: 0000000000000246 cgf: 002b000000000033 erf: 0000000000000000
 trp: 0000000000000000 msk: 0000000000000000 cr2: 0000000000000000
[end of stack trace]
Calling _exit(1). Core file will not be generated.
2021-01-05T16:35:39.974Z    9ef8809a-248d-42bf-a3b1-316b34464051    INFO    IN THIS CATCH
2021-01-05T16:35:39.977Z    9ef8809a-248d-42bf-a3b1-316b34464051    ERROR   Error: Protocol error (Runtime.callFunctionOn): Target closed.
    at /var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:208:63
    at new Promise (<anonymous>)
    at CDPSession.send (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Connection.js:207:16)
    at ExecutionContext._evaluateInternal (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:200:50)
    at ExecutionContext.evaluate (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:106:27)
    at DOMWorld.evaluate (/var/task/node_modules/puppeteer-core/lib/cjs/puppeteer/common/DOMWorld.js:90:24)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
END RequestId: 9ef8809a-248d-42bf-a3b1-316b34464051
REPORT RequestId: 9ef8809a-248d-42bf-a3b1-316b34464051  Duration: 6021.29 ms    Billed Duration: 6022 ms    Memory Size: 3008 MB    Max Memory Used: 1165 MB    Init Duration: 216.16 ms

CheeseNPort commented 3 years ago

I have a similar issue with this, similar use case. I am using puppeteer to navigate to a URL with mapbox-gl. It works fine for images at A4 and A3 but crashes with Target Closed if I try the browser at A2 size. To do this I am setting this window size in the args. This forces the map to be the right size when I navigate to it. Like @deldrid1 it works fine locally or in an EC2 instance for all sizes even up to A0 but on Lambda it only works up to A3.

ahansson89 commented 2 years ago

Should be fixed with this: https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/

deldrid1 commented 2 years ago

@ahansson89 - have you done any testing with this to see how it affects performance?

I'm curious if anyone has done any evaluation on increasing this size vs. memory (and therefore vCPU) for puppeteer? From my testing, I see that /tmp is often HIGHLY utilized (i.e. 99% full) after running puppeteer (see https://medium.com/wix-engineering/how-to-debug-enospc-on-aws-lambda-2a74ae457fcb and https://gist.github.com/yurynix/6beef61ec4189b37fad0c86bf9fe21b7 for some brilliant tools to help with visibility here) so this is a very intriguing new lambda capability!

CheeseNPort commented 2 years ago

Should be fixed with this: https://aws.amazon.com/blogs/aws/aws-lambda-now-supports-up-to-10-gb-ephemeral-storage/

Yeah it does indeed for me.

ahansson89 commented 2 years ago

@deldrid1 I don't see why performance would be affected - certainly not going to do any performance testing

Yes, that is the issue that is causing this. /tmp filling up as images or the WebGL canvas is downloaded to the /tmp folder. Now that you can increase the size, you should be good creating large PDFs or Screenshots.

alixaxel / chrome-aws-lambda