venkata91
commented
5 years ago @habemusne Thanks for trying Spark on Lambda out. I understand in its current form its not easy to set it up and try out. Some time back @faromero also had issues running it but with the AWS setup fixed it worked fine.
As far as AWS Lambda, EC2 and VPC setup is concerned, this is what I did which worked.
VPC stuff - This is what I did.
1. Created a VPC - venkat-vpc
2. Created a public subnet - venkat-public-subnet with in the above VPC. This automatically comes up with an Internet gateway. Check routes you'll see something like igw-*
3. Created a private subnet - venkat-private-subnet with in the above VPC as well.
4. Then created a NAT gateway for the above private subnet. In the route table, created a route for the NAT gateway (nat-*) to the Internet gateway available in the public subnet (igw-*).
Bring up the lambda function inside the VPC (venkat-vpc) with in the private subnet (venkat-private-subnet) and an EC2 instance where you'll run spark-submit or spark-shell in the same VPC (venkat-vpc) but in the public subnet (venkat-public-subnet). Also create a security group with inbound rules (access to TCP, UDP from the same security group) and outbound rules (All traffic (0.0.0.0/0). Associate both the EC2 instance (Spark Driver node) as well as the lambda function (Spark executor).
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html
Above doc was useful, the picture in the documentation helped in setting up VPC and subnets.
This works for me. Hope it works for you as well.
Regarding passing the input and output files, use S3 locations I don't remember using local files. But for application jars and other dependent jars, those can be passed from the local machine (file:///).There are couple of questions I have,
- Are you able to run map only functions using Spark on Lambda, something like
sc.parallelize(1 to 10, 5).map(x => x * 2).collect()If this works then the lambda setup is good and we are good on that part. - If first step works, then running a spark job involving map and reduce fails most likely some config is not setup or possibly a bug.
Let me try running this by this week and share you the set of configs I used to make it work.
habemusne
faromero
achinmishra
I have been struggling with setting up this framework on my EC2 server. I tried the best to follow the instruction of both this repo and also faromero's forked repo, but I have been getting this error message each time I run
sudo ./../driver/bin/spark-submit ml_kmeans.py --master lambda://test:Here is the full log
My config file:
~/.aws/configis us-east-1. VPC subnets are configured following the forked repo's instruction.My Lambda function is tested to be able to write and read to S3. My spark-submit command ran on EC2 is able to write to S3 (it generates a
tmp/folder on the bucket), but does not let the Lambda run at all. CloudWatch for my Lambda has no logs. However, I am able to run my Lambda from EC2 using something likeaws lambda invoke --function-name spark-lambda ~/test.txt. I guess I configured Spark-on-Lambda wrong but I've been following the instructions.I am now trying to dive into the source code. Is there any clue for this message?