Arguments =)

[myoung34]

You wrong brah

Outline

• High Latency when spinning up new instances.
  • Only in VPC's because the ENI has to be created and attached. Resolved here
  • Also you can architect for this. read here
  • you can utilize the aws container runtime that hosts lambda to cache a bunch of stuff and end up running your lambdas for basically free by avoiding putting it in the handler. Cant find the doc but this is a tried and true method of running for essentially free and fast at scale
  • lambda@edge?
• Debugging is a nightmare:
  • Logs don't come through CloudWatch immediately
  • This is my biggest complaint is CW, i have a bunch of AWS internal tickets for using kinesis streams/rsyslog for logs
  • No ptrace or bpftrace for production debugging
  • NO
  • NO
  • NO
  • you should be debugging in dev using localstack. NO ONE should ever be using ptrace on production anything
  • ALSO NO. if you have to run this shit in prod youre missing tooling.
  • No atop for understanding resource utilization.
  • Use an APM like xray/datadog etc
• Weird packaging & deployment pattern that isn't used anywhere else.
  • In practice you must use a framework like Serverless or Zappa
  • not true? terraform with zips? apex? aws cli?
  • use a makefile
  • They handle some but not all infra for you -- where should the line be drawn?
    • they dont handle any infra for you and shouldnt. the only thing they handle is the ENI (expected for anything managed) and CW logs which suck
• Is "patching servers" really that hard? Yumcron, anyone?
  • Yum cron aint shit. That doesnt solve SOC2/ISO27001 or change control or package auditing, i can go on here.
  • Patching is extremely hard in places with legacy software. SSM/osquery/etc all need a pipeline/lifecycle of their own.
  • even if you go to full containerization you have to package in 3 places: the host, the container (claire/twistlock container scans), as well as package scans (python, node, whatever). If you scan you have to continuously scan, so you have to continuosly scan 3 lifecycles now. Patching is hard
• Providing secrets means using + paying for AWS Secrets manager
  • You can use pure KMS. Or Vault. Or credstash. All of those are free even at our scale (23k lambdas)
  • Secrets SHOULD use IAM. Permissions are important for secrets
• Network performance is proportional to memory allocation
  • This makes you pay for way more than you need in order to get a responsive app
    • why tf would you rely on a responsive app from lambda? Use something like fargate
    • architect for performance, dont hope for it. lambda@edge?
  • Speculation: this comes from oversubscribing RAM via KSM, can't do same with network
  • No, just how aws wants you to pay more, plain and simple
• Lambda saves money when apps are mostly off, but the developer time will never pay off.
  • Not true. Use something like a cookiecutter if thats a big deal.
  • ironically lambda is a much better dev experience if you tool it right for your devs becaus eyou can 100% isolate your code and tests.
  • tests using localstack for full AWS sandboxing
  • Test using pytest/tox/whatever you want. See earlier makefile

From slides:

https://github.com/robertdfrench/lambda-is-terrible/blob/master/slides.tex#L23

• this is a failure to scale. Scale down
• turn shit off with cloud-custodian (or ironically just a lambda that uses tags)

https://github.com/robertdfrench/lambda-is-terrible/blob/master/slides.tex#L21

• Patching servers is very hard.
  • Immutable? You need an artifact lifecycle for your servers. packer + tests (inspec/testinfra) + tests for your automation (ansible + molecule etc).
  • Non-immutable? You need config management:
    • shit better not be using a shared ssh key (ansible by default etc), now you need to use SSM under ansible etc
    • you need change control for SOC2/ISO27001 compliance you cant just push packages
  • youll need an inventory management for packages to CVE's thats up to date for all servers
• Scaling is hard. By default its usually only cpu. RAM/etc is all internal and youll need your own metrics system (datadog, metricbeat, statsd etc) and some sort of coordinator. this is relative to the app. Web apps are easy'ish but stateful shit is hard. SFTP? Better make sure you dont change the fingerprint if you have outward facing integrations. Things that use TCP sockets (vpn's, healthcare integrations etc) live on a long lived socket so you have a hard time "just scaling" and it needs a special lifecycle

https://github.com/robertdfrench/lambda-is-terrible/blob/master/slides.tex#L33

• gatekeeping at its finest. I dont use any windows machines but i wont tell someone we dont use x thing because "you should just learn unix bro". This wont fly in any company. Linux is fine but doing this shit in production in a company has the same issues no matter what OS. none of my arguments take OS into consideration on purpose. its irrelevant.

https://github.com/robertdfrench/lambda-is-terrible/blob/master/slides.tex#L39

wot?

come at me brah

[robertdfrench]

Thanks @myoung34 I will incorporate this and cite you.

[robertdfrench]

@myoung34 can you elaborate on "23k lambdas"? Is that your peak concurrency per second? Or are you saying you have 23,000 different lambda functions defined?

[myoung34]

we have 23000 separate lambdas.