Use Maven to build; add README

[philipmw]

This removes our own copies of needed JARs!

[barbeau]

ICLA manually verified :+1:

[barbeau]

Thanks @philipmw, Readme is sorely needed, and a Maven build would be great.

From your other PR, it looks like you are building in IntelliJ? I've been using Eclipse with the Amazon Lambda plugin to build/deploy to Lambda. I'll need to test with that setup to see if it works - let me know if you already have.

[philipmw]

I just force-updated the pull request's branch. Now I tried it.

START RequestId: 28c7ac5c-ace9-11e5-a3d9-3357e0306e65 Version: $LATEST
Dec 27, 2015 10:28:40 PM com.google.maps.GeoApiContext getWithPath
INFO: Request: https://maps.googleapis.com/maps/api/geocode/json?key=AIzaSyA6PiJXUmxic5fkIC4WEPIBldi6omdVZnQ&address=Seattle
END RequestId: 28c7ac5c-ace9-11e5-a3d9-3357e0306e65
REPORT RequestId: 28c7ac5c-ace9-11e5-a3d9-3357e0306e65  Duration: 15002.58 ms   Billed Duration: 15000 ms   Memory Size: 128 MB Max Memory Used: 45 MB  
Task timed out after 15.00 seconds

So something's wrong, but it's not the build system. :)

[barbeau]

Thanks @philipmw, a few comments in-line.

[philipmw]

Ok, I implemented all your suggestions.

GREAT NEWS: Maven allows us to exclude 3rd party dependencies that we consider invalid, so I went ahead and excluded alexa-skills-kit:alexa-skills-kit. Everything still builds and runs.

[barbeau]

@philipmw Thanks for this, it's looking awesome.

My only reservations at this point are related to the automated upload/execute/output workflow - currently, in the master branch we can use the Eclipse AWS plugin to right click on the project and do "Amazon Web Services -> Run function on Amazon Lambda...", and it automatically uploads/executes, and you can write feedback to the console in Eclipse via the output stream. This is great for making quick changes and immediately (well, with some delay) seeing the output.

With this PR, we gain the Maven build (which has great benefits like removing the bundled JAR dependencies from the Git repo, and running build/tests on Travis CI), but we currently lose this automated upload/execute/output functionality. I've played with Eclipse a little hoping to fix this, but at this point it seems that AWS plugin and Maven are mutually exclusive in Eclipse - the project can be either an AWS Lambda project, or a Maven project, but not both. I tried copying the .settings/com.amazonaws.eclipse.lambda.project file from the AWS project to the Maven project, and it doesn't seem to have any effect.

@philipmw do you know if there is a way to automate the upload/execute/output functionality to Lambda while keeping the Maven build?

Or, as mentioned offline, we can look to switching to EC2 instead of Lambda. I'm still on the fence with this, though.

[philipmw]

I didn't know you have an established workflow involving the Eclipse AWS plugin -- I wouldn't have moved to Maven so quickly if I had known. I didn't know this plugin exists; I've just been uploading the JAR to Lambda manually using the AWS Console.

We don't have to give up Lambda for anything but (potentially) production. Here are some options I see:

1. Use a Maven plugin to automate submitting to Lambda. I found this, though haven't tried it: https://github.com/SeanRoy/lambduh-maven-plugin. If it works, I think this is the best option because it's easy to integrate with IDEs and continuous deployment systems, and it allows us to correctly set dependencies (build and package before uploading).
2. If the Maven plugin falls through, could you keep your existing Eclipse+AWS+local JARs development environment? The JARs would live in your local lib/.
3. Write a small shell script that uses the AWS CLI or AWS SDK to offer similar functionality to the Eclipse AWS plugin.

As far as being able to interact with your Lambda function in your development environment, that's more challenging. You might be able to do it with that custom shell script, but long-term, I think we should move to integration tests rather than ad-hoc interaction with Lambda.

[barbeau]

Agreed, the 1. Lambda maven plugin is what I had in mind. Agreed that integration tests are best for long term. On Dec 30, 2015 3:38 PM, "Philip M. White" notifications@github.com wrote:

I didn't know you have an established workflow involving the Eclipse AWS plugin -- I wouldn't have moved to Maven so quickly if I had known. I didn't know this plugin exists; I've just been uploading the JAR to Lambda manually using the AWS Console.

We don't have to give up Lambda for anything but (potentially) production. Here are some options I see:

1. Use a Maven plugin to automate submitting to Lambda. I found this, though haven't tried it: https://github.com/SeanRoy/lambduh-maven-plugin. If it works, I think this is the best option because it's easy to integrate with IDEs and continuous deployment systems, and it allows us to correctly set dependencies (build and package before uploading).
2. If the Maven plugin falls through, could you keep your existing Eclipse+AWS+local JARs development environment? The JARs would live in your local lib/.
3. Write a small shell script that uses the AWS CLI or AWS SDK to offer similar functionality to the Eclipse AWS plugin.

As far as being able to interact with your Lambda function in your development environment, that's more challenging. You might be able to do it with that custom shell script, but long-term, I think we should move to integration tests rather than ad-hoc interaction with Lambda.

— Reply to this email directly or view it on GitHub https://github.com/OneBusAway/onebusaway-alexa/pull/5#issuecomment-168069936 .

[philipmw]

Ok, I integrated with lambduh-maven-plugin and updated the readme.

[philipmw]

Here's how I run it using a local script (to keep from committing my AWS credentials):

mvn package &&
mvn lambduh:deploy-lambda \
    -DaccessKey={key} \
    -DsecretKey={key} \
    -Ds3Bucket=pmw-oba-jar \
    -Dregion=us-east-1 \
    -DlambdaRoleArn=arn:aws:iam::101804781795:role/lambda_basic_execution

[barbeau]

Thanks @philipmw! Hopefully I'll have a chance to look at this in the next few days. So new workflow using this script to upload to Lambda, and then trigger it via the web console and view the log there?

[philipmw]

Yep, that's the stopgap solution. My next goal is to get some functionality working, with supporting tests. It should be as simple as maven test or maybe maven integration-test.

[barbeau]

@philipmw So it turns out the Task timed out after 15.00 seconds message in the logs is just a function of the "memory" and timeout configuration values (under "Configuration->Advanced" for the Lambda instance in the AWS Lambda console). On Lambda the instance is allocated CPU proportional to memory, so with the default 128MB setup it can't complete the execution in less than the default 15 second timeout. If you bump the memory to 256MB or 512MB it's substantially faster and doesn't the timeout (well, 256MB might - last execution time was 17 sec). It seems like everything works here - I'm going to finish a few last tests and then merge if all looks ok.

[philipmw]

It's surprising that it takes >15 secs to start. Good to know.

[philipmw]

Tangential: I submitted a PR to the lambduh plugin so it doesn't re-upload the JAR every time you sync, leaving it alone if unchanged. It hasn't been merged yet, but the plugin author has been receptive.

[barbeau]

Cool!

So one issue (maybe two?), and likely something I'm just missing in the new workflow/config - I can't seem to get code changes to show up in the console.

So, I edit the code locally (e.g., add another println), and then do:

mvn clean
mvn package &&
mvn lambduh:deploy-lambda \
    -DaccessKey={key} \
    -DsecretKey={key} \
    -Ds3Bucket=onebusaway \
    -Dregion=us-east-1 \
    -DlambdaRoleArn=arn:aws:iam::533962225716:role/lambda_basic_execution

..with my info. I see that it uploads to my onebusaway S3 bucket.

However, two things are suspicious:

• It shows up as a new "onebusaway-alexa" Lambda function (and not my original "OneBusAway" method) // <-- I think this is ok, just a function of the lambduh plugin?
• When I run a test with "onebusaway-alexa" Lambda function via the Lambda console, my local code changes aren't reflected.

Any ideas what I'm missing?

[barbeau]

Nevermind, I'm an idiot. I had this cloned in more than one place on my machine and was editing the wrong project in Eclipse :/.

[barbeau]

Ok, this looks good to me! Thanks @philipmw!

[barbeau]

So one thing that's annoying, and I'm not sure if its a side-effect or "feature" of the lambduh plugin - it seems that the Lambda memory and timeout for the "onebusaway-alexa" function is reset to 128MB and 10 sec, respectively, each time you run mvn lambduh:deploy-lambda. Would be nice to keep this consistent, otherwise you have to continuously go in and reset it.

[barbeau]

Looks like these are both configurable - I'm going to tweak.

[barbeau]

Timeout and memorySize tweaked via https://github.com/OneBusAway/onebusaway-alexa/commit/da550b6a92541d395bdd7e5a39795ff6b38475b2.