Evaluation - Stage A

[algrebe]

• [ ] General stats - num urls, amount of data and time taken for a particular site crawl, urls/sec
• [ ] Amount of concurrency (at what point does its benefits stop).
  • [ ] Are our lambda handlers stressing the source website's servers?
  • [ ] Can our master handle that amount of concurrency?
  • [ ] Does varying number of handlers give us a significant speedup?
  • [ ] Is fan-in from multiple lambdas a problem?
• [ ] Cost
• [ ] Is our bottleneck masters network bandwidth or disk (crawlqueue?), or is it in invocation?
  • [ ] Would SSD's make a difference?

[gauthamsunjay]

• Timers:

  • Time to complete each lambda - lambdas.invocation.success, lambdas.invocation.failures

• Counters:

  • How many URLs processed per second? - lambdas.urls.in_progress
  • New URLs inserted per second - urls.new
  • Invocations per second - lambdas.invoked
  • Bytes/second(data) - store.writes
  • Crawlq, get and put - crawlq.get, crawlq.put

• Guages:

  • How many lambdas are currently active? - lambdas.active
  • How many sources are being run?
  • How many URLs are waiting/processing/completed in the database? - urls.waiting.on_disk, urls.waiting.in_mem, urls.in_progress.in_lambda, urls.completed

[gauthamsunjay]

Issues found:

1.) Boto3 client internally maintains only 10 open connections. To avoid this limit we first created one boto client object per thread. Potential solutions: Fanout lambdas, use event invocation method, write to DynamoDB to trigger lambdas from there.

[algrebe]

our initial implementation was wrong - we need one resource session per thread https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing