Checkoff 2

[sheikhshack]

Hey guys, can comment whatever questions here to be asked to prof. Thanks!

[sheikhshack]

Production

1. Do we need to use elastic IPs? Can we use placement groups for increasing throughput ?
2. Are we graded based on speed of deployment? Or will just a deployment script that works be sufficient
3. For 'take in credentials as input', are we assuming user feeds via .\aws\creds or must we let user specify each argument via CLI?
4. For teardown, is terminating instances sufficient

HDFS

1. Can we use libraries like this flintrock? Or others like in this article

Spark

1. Can we just install mongo on the namenode and do an import direct via mongo cli?
2. Or can we import mongo db data via spark?

General

1. Any performance metric/benchmark we should compare with to evaluate how good our script is?
2. What is meant by code quality?

[sheikhshack]

Prof Reply

Checkpoint 2 DB Creds

• Can be either Elastic IP
• Just continue as we were Speed
• Minimum requirements is functional
• If you can do it fast, more points Flintrock
• You are allowed to use it! Logs
• Yep this is what we want Hadoop and Spark
• Store output of TF-IDF on a file or on the CLI
• TF-IDF might be very big, might want to store it somewhere Timing of performance
• Set up of infra, and running processes
• Prof will specify how many nodes he wants Startup when
• Separate scripts for production and analytics
• Give prof the option to startup up the analytics when production is done