POC - Scraper on State machine

[rivernews]

AWS provides this great built in async worker mechanism, and there's a lot of potentials to save cost. Without this, we have to maintain running-all-day servers or at least managed EKS, and not to mention managing that distributed workload, and even build a management portal yourself.

Ideally, we can run scrapers on state machine. But there's some research we need to do before we confirm it's worth of carrying it out.

Cost Research

AWS on-demand billing - charge per execution

Is state machine similar to lambda and server less cost charging model, and can save us tons of money, comparing to self hosting a EKS while building a workload management system on our own?

Pro

• AWS Step function + AWS Batch seems recommended, and does seem to save cost because of no need to self host server.

Cons

• Based on this post, state machine can invoke lambda, but lambda only executes up to 5 minutes of workload. How long will our scraper run? If it's short, then state machine could be an ideal solution.

Reusing existing k8s cluster

We have an existing k8s cluster, so we already are paying stuff. If the current k8s can fully cover this project need, then probably it makes more sense to just build our own servers running in our k8s cluster. But of course if we're even planning to migrate everything to lambda, and eventually let go of that k8s cluster, then this is not an issue.

Need to Consider CI/CD platform as well

But if the workload is small... probably Github Action or other CI/CD is a better platform, totally free of charge.

Cons

• The limitation is these platform might have time limit on the workload, such as TravisCI is said to have an hour limit.

If it's really cheaper... wait, is deeper coupling with AWS a good move?

However, another perspective is if we want to really shift our gears, including future skillset, to all cloud computation and more coupled with one of the cloud providers, then we can say we might even want to migrate iriversland and appl-tracky backend to server less! (thank god the frontend probably don't need much change)

Of course it's debatable whether if we really want to put our entire self into these cloud provider's server less architecture, because any single of these providers are limited and could rarely compete with the open source world in terms of community (stack overflow contributions), documentation (some popular open source tool maintained by really passionate people), and user (developer) friendliness. These cloud providers usually prioritize their business needs when it comes to their tooling, whereas open source tools receives feedback from the entire community can could achieve developer-centered tools better.

That said, why not a hybrid solution. Serverless now there're people de-coupling it into an architecture, where you can swap in and out different cloud provider behind - called "vendor agnostic" server less architecture. If you're using SAM, then yes it's tied to AWS and thus its limited documentation, online community, and limited motivation to help developers outside of its business. But with the server less architecture, or even like Terraform, yet another open source tool (yet maintained by a private company), you gain the benefits of great tooling, better documentation and online community brought by Terraform, etc.

Open discussion: workload orchestration, async processing

• AWS SWF simple workflow service (>5 minutes)
  • AWS SFN state machine step function (lambda for workflow <=5 minutes, AWS Batch for longer process?)
  • This mentions AWS Glue can do longer jobs too.
  • A list of AWS services that can corporate with AWS step function.
  • After all, it's recommended to use step function as a central orchestrator to run computation on AWS services.
• Celery
• Cadence (by Uber, in Golang)
• Apache Airflow (in Python)
• Luigi (Spotify, in python, includes UI)
• Azkaban (used by Linkedin, etc, in Java though)
• (on K8s) Argo workflow: so-so documentation

Additional Keywords: workflow engine, workflow execution engine, long-running workflows

[rivernews]

POC

Run a scraper in step function, scrape html, and store it on S3. Knowing the basics to create a step function and required resources using Terraform.

• State machine resource
• IAM role for state machine
• Policy for the IAM role of state machine We could also add a lambda function to run by the state machine. Here in this article, we use manually archiving zip to create the lambda function.
• Lambda function resource
• IAM role for lambda
• (static) IAM policy document - lambda assume role
• Zip archiving & packaging local source code

Also see Terraform official doc.

[rivernews]

POC

Rethink about it - we want to be event-driven:

• 💡 Slack calls the API Gateway - use slack to trigger (quick & sleek) instead of a web UI!
  • [x] Create a web hook in Slack, outgoing.
  • [x] Create API Gateway - use HttpApi instead of RestApi: https://slacker.ro/2020/06/02/how-to-use-the-aws_apigatewayv2_api-to-add-an-http-api-to-a-lambda-function/
  • [x] Prepare layer for lambda
  • [x] Prepare lambda
  • [x] Connect lambda and API together
  • [x] Create domain, certificate, etc
  • [x] Set up custom domain name for API Gateway
  • [x] Create Swagger that defines routes
• How should we authenticate it? See 💡 Slack's signing secret.
  • Maybe we can have a 👀 custom API Gateway authorizer? No, it's not supported, let's just setup a lambda to do the auth.
  • The request parameter-based lambda authorizer, but need to 👀 return allow policy documents, how to generate that?
    • principleId - normally this could be username, but if authorizer lambda just checking a valid Slack request, this can just be slack, etc
    • Allowed resource - use from event provided when authorizer func invoked. - https://aws.amazon.com/blogs/compute/introducing-custom-authorizers-in-amazon-api-gateway/
    • How to access request from authorizer func?
    • [x] Any formal way to validate slack request? Do we really need to do this ourselves?
• 💡 Finally get into lambda function - call the step function
  • [x] Create lambda layer
  • [x] Create lambda function
  • [x] Allow API Gateway to call lambda
• 💡Step function: run the scraper logic
  • Harvest the results
  • Some kind of notification? Either email or slack

Reference

• ✅ Step function, Lambda, and their IAM role required: https://nahidsaikat.com/blog/invoke-aws-lambda-from-aws-step-functions-with-terraform/
• ✅ API Gateway, Lambda, w/ Terraform:
  • Swagger + API Gateway + Lambda: https://stackoverflow.com/questions/54047171/deploy-api-gateway-with-terraform-based-on-a-swagger-file (but no IAM role set up, no Lambda set up)
  • ✅ API Gateway module with many features built-in: https://github.com/corpit-consulting-public/terraform-aws-api-gateway (but only partial Swagger, no IAM role set up, no Lambda set up)
  • ✅ API Gateway integrates with Lambda (Terraform official): https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_integration
  • ✅ Serverless = API Gateway + Lambda (Terraform official): https://learn.hashicorp.com/tutorials/terraform/lambda-api-gateway
  • ✅ Needs more way or detail on lambda func source code deployment - most common way: zip + upload
  • ✅ How to manage dependencies? Pip install / npm install where?
    • Take a look at lambda Layer: https://medium.com/craftsmenltd/automate-python-libraries-deployment-at-aws-lambda-layer-from-pipfile-with-terraform-d28de0eb765f
  • ✅ Think about ways of how you want to run and test code - on local, on dev, and on prod
  • ✅ Testing: mocking AWS services: moto python package.
  • Lambda function - should be able to just BE function - then the rest is unit tests.
• Monolithic approach: Serverless.tf: https://serverless.tf/#aws-serverless
• Deploying Swagger UI doc on S3 and serve static website: https://advancedweb.hu/interactive-api-documentation-using-swagger-ui-deployed-with-terraform/

[rivernews]

• API Gateway
• Step func