lambda 6.10 fails to run

brianleroux commented 6 years ago

Ideally AWS just upgrades to Node 8 but I thought you maybe would like to know running this inside a Lambda currently bombs here. Will have a look if I get a chance. Any time saved in our CI is super appreciated! 🍻

{
    "errorMessage": "Command failed: cd /tmp/GZmcWNlBJ && node /var/task/node_modules/qdd/index.js\n/var/task/node_modules/qdd/lib/isdir.js:10\nconst statValues = bStatValues || getStatValues();\n                                  ^\n\nTypeError: getStatValues is not a function\n    at Object.<anonymous> (/var/task/node_modules/qdd/lib/isdir.js:10:35)\n    at Module._compile (module.js:570:32)\n    at Object.Module._extensions..js (module.js:579:10)\n    at Module.load (module.js:487:32)\n    at tryModuleLoad (module.js:446:12)\n    at Function.Module._load (module.js:438:3)\n    at Module.require (module.js:497:17)\n    at require (internal/module.js:20:19)\n    at Object.<anonymous> (/var/task/node_modules/qdd/lib/cp.js:5:15)\n    at Module._compile (module.js:570:32)\n",
    "errorType": "Error",
    "stackTrace": [
        "/var/task/node_modules/qdd/lib/isdir.js:10",
        "const statValues = bStatValues || getStatValues();",
        "                                  ^",
        "",
        "TypeError: getStatValues is not a function",
        "Object.<anonymous> (/var/task/node_modules/qdd/lib/isdir.js:10:35)",
        "Module._compile (module.js:570:32)",
        "Object.Module._extensions..js (module.js:579:10)",
        "Module.load (module.js:487:32)",
        "tryModuleLoad (module.js:446:12)",
        "Function.Module._load (module.js:438:3)",
        "Module.require (module.js:497:17)",
        "require (internal/module.js:20:19)",
        "Object.<anonymous> (/var/task/node_modules/qdd/lib/cp.js:5:15)",
        "Module._compile (module.js:570:32)",
        "",
        "ChildProcess.exithandler (child_process.js:204:12)",
        "emitTwo (events.js:106:13)",
        "ChildProcess.emit (events.js:191:7)",
        "maybeClose (internal/child_process.js:886:16)",
        "Process.ChildProcess._handle.onexit (internal/child_process.js:226:5)"
    ]
}

bengl commented 6 years ago

@brianleroux Hi Brian! Thanks for giving qdd a spin!

I'm sorry to say that Node 6 is not supported.

I did make an attempt at porting to Node 6, but there were a few hurdles:

fs.stat is implemented completely differently, so using the binding directly as qdd does requires a complete rewrite of qdd/lib/isdir.js. Bit of a pain, but doable.
qdd/index.js uses Object.entries, so a polyfill for that is needed. No problem.
fs.copyFile isn't available in Node 6, so a more manual process is needed. No big deal here, just pipe one file to the other, right? Nah. That's a great recipe for an Error: EMFILE: too many open files, open, so something like graceful-fs is needed, and qdd already has too many dependencies.

Number 3 here is the real showstopper.

That being said, copyFile is only used when copying from cache. It's my understanding that Lambda has no persistent FS (please correct me if I'm wrong), so perhaps you'd always be running qdd in Lambda with no persistent cache? In that case, a version of qdd could be constructed in which you may be able to just ignore the cache case entirely and just use the innards of qdd that download and extract tarballs to the correct places. As an added bonus, you could skip the also-extract-to-cache bits of that code, to shave off even more precious nanoseconds.

If my assumption above is correct, then this is totally doable, and I can make a one-off qdd-lambda module when I have some time.

Are you currently doing npm i (or npm ci) in Lambda?

bengl commented 6 years ago

I made a branch https://github.com/bengl/qdd/tree/node6 that has my rough-draft of getting it to work in Node 6. Doesn't address hurdle number 3, above.

brianleroux commented 6 years ago

oh fkn eh! sorry just seeing these and ya the stateless qdd-lambda idea should work BUT BIG WARNING! I am guessing we can expect to hear Node 8 on Lambda as soon as Wed for AWS Summit.

So maybe wait a couple of days and this problem gets solved with patience? =P

bengl commented 6 years ago

It might still make sense to make a stateless qdd (or maybe an option in existing qdd) anyway, just to not bother writing to a cache directory.

I can't imagine I'd get a lot done before Wednesday anyway, haha.

bengl commented 6 years ago

Boom: https://aws.amazon.com/about-aws/whats-new/2018/04/aws-lambda-supports-nodejs/

brianleroux commented 6 years ago

damn they must've been listening!

bengl commented 6 years ago

Good news! In qdd@1.1.0, I've added a new no-cache mode. Just set the environment variable QDD_NOCACHE=1, and it will bypass any usage of a cache directory. In my completely unscientific tests, this represents a 30% reduction in total install time. Enjoy!

brianleroux commented 6 years ago

This is huge, works great on the new Node8 and def baking into both our internal CI and arc.codes

Thx so much for this work and pls lmk if I can help in any way. Closing the issue but still feel I owe you a couple 🍻!

bengl commented 6 years ago

I'm glad qdd is useful for you! In terms of helping: please let me know if there are any missing features you need. Also, all PRs are quite welcome.

qddjs / qdd

lambda 6.10 fails to run #3