Zappa deploy

[fergyfresh]

I'll have to resurrect the README from #167, but I got this to work with API Gateway + AWS Lambda + S3. If you look at geemusic/controllers.py you'll see I introduced two new env's. They are USE_S3_BUCKET and S3_BUCKET_NAME. The steps are pretty similar, but use my newer zappa_settings.json file as a guide for yours. You can also mess with all those same settings in the AWS console.

There was also a minor bug that I ran into where flask-ask used BytesIO and API Gateway+Lambda only use string body, not binary so I had to make that minor modification.

Also not sure about how necessary the changes to __init__.py were, but it was something when I was spitballing trying to make this stuff work.

[fergyfresh]

@digiltd BOOOYAAAAAAAAAAAAAAAAAA and you thought it couldn't be done.

[stevenleeg]

@fergyfresh Wow this is awesome! I'm assuming this allows people to deploy GeeMusic to Lambda without needing to go through all of the ridiculous process of setting up a server themselves (I'm not familiar with Zappa)?

I'd love to merge this in, but would you mind adding a section to the README documenting how this works?

[fergyfresh]

So zappa integrates API Gateway + AWS Lambda OOTB for you so that you don't have to worry about that stuff. It unfortunately has its own env-var setup as well so you need to use a zappa_settings.json file. I will update the readme before you pull this in there.

There are two reasons why I took this approach. The first reason is because the webhook for the playback functionality cannot be done without a URL and Lambda functions only give an ARN, which doesn't allow routing, /alexa for example.

The second reason being the need for integrating S3. Lambda functions have an upper limit of 6MB and our non-binary streams for playing the song are ~8MB on average. I had to use a redirect proxy to get that through an S3 signed URL to the end user. Many hours spent on this, but I really wanted to get a handle on using AWS for something like this.

[fergyfresh]

Another thing to note is my egg for flask-ask. I'm having to maintain a fork for all of my AWS Lambda deployments (not just the zappa ones) in order to fix a bug introduced by https://github.com/johnwheeler/flask-ask/commit/23ff87e89c6633b60ea4d61e94b5f34396cb627d. To use StringIO instead of BytesIO for the WSGI req body. json.dumps returns a string so I don't know how that PR even works in the first place. It def breaks for everyone unless there is something major I'm missing.

[fergyfresh]

The other good part about it is that it only costs ~30 cents a month and I use it almost every day.

[digiltd]

is the s3 traffic chargeable? granted it looks like it wouldn't be much if it was, but interested to see how it adds up

[fergyfresh]

@digiltd the S3 is chargeable and there is also a data transfer charge and an API Gateway charge. Lambda charges once you've used it for a year (you get a year for free). I'm almost 3 weeks in at this point using this and this is the results in billing so far:

S3 = $0.03 (dynamic by usage) ApiGateway = $0.02 (static per month) DataTransfer = $0.02 (dynamic by usage) Lambda = $0.00 (will be $0.20 as of 5/1/2019) Total = $0.07

Next year it will be $0.27 because of the pricing model of Lambda functions right now. I used it pretty much all day last Saturday and it cost me like 1 cent. So its usage dependent. for sure but shouldn't be more than 50 cents a month.

Edit: Only cost me 2 cents and it ran for the whole billing period of May.

[fergyfresh]

@stevenleeg got the readme stuff in there now. Should be allset to rip this one out there.

[fergyfresh]

Also, here's an update. The billing cycle of May only charged me $0.02, but it didn't hit my credit card so I paid 0.00 so far. It said 7 cents the other day, but checking the billing for month of May I only got charged 2 cents, but there is no charge on my credit card for that amount. Hmmmmmm. It's even cheaper than I thought!

[fergyfresh]

@stevenleeg @digiltd any of you guys get a chance to try this? Want someone else to verify that my README gets you to a spot where you can deploy it. I deleted my project and followed it and it worked for me. I just wanna make sure there isn't some default setting on my computer that is causing me to whiff on anything.

[stevenleeg]

This is awesome. Merging and we'll see if there are any issues that arise but thanks a ton for the contribution. Hopefully this helps people get Geemusic working faster!

[digiltd]

Finally gave it a try. As much as i want it to, i have been unable to get it to work :(

A few bits that confused me but I worked around them, usually zappa init is run to generate the zappa_settings.json file, but there is already one in the repo so it fails. I can see why you included it, but maybe call it something else and then the user can copy and paste the env var bits.

...

I started this reply then went off to have another try, and I have managed to get it to work.

Yay, well done.

Some issues though. I think this maybe Mac related but some of the default packages that pip downloads will not run on lambda servers. I know there is a lambda-packages repo with precompiled packages/bundle/things for lambda, and Zappa is supposed to use it in these instances. But experience has taught me this has to be done manually.

To do so you go into venv/lib/python3.6/site-packages/lambda_packages/ and find cryptography/python3.6-cryptography-1.9.tar.gz unarchive it to get a folder with two folders inside. Drag the two folders and drop them into the venv/lib/python3.6/site-packages folder, you might want to get rid of any cryptography folders already in there first.

Then do the same with venv/lib/python3.6/site-packages/lambda_packages/pycrypto expand the 3.6 folder and drag the Crypto folder into site-packages.

Then (I think) run the pip freeze > requirements.txt command to make sure Zappa uploads them, make sure cryptography is set to cryptography==1.9 not cryptography==2.0.3, I also replaced flask-ask with your git+https://github.com/fergyfresh/flask-ask.git@master#egg=Flask-Ask, you should see (when running zappa update dev) in the terminal something like:

Downloading and installing dependencies..
 - protobuf==3.3.0: Using locally cached manylinux wheel
 - cryptography==1.9: Using precompiled lambda package
 - cffi==1.10.0: Using locally cached manylinux wheel
 - sqlite==python36: Using precompiled lambda package

Again this might be Mac only as you don't appear to have the same problem. I think it is when pip downloads and builds some dependancies it does it for the platform it is on, not the target platform

[fergyfresh]

There's hope and there's sorrow. So I guess after using this for a month I got banned by Google. I'm still trying to sort it out as my failure is in a very similar fashion as https://github.com/simon-weber/gmusicapi/issues/590. I'm going to change my app password tommorrow when I get home from work and hope that fixes it. I'll incorporate your fix to the readme when I also figure out the fix for not getting banned for using S3. I'm not sure if thats the culprit, but it might be. There might have to be a tweak in gmusicapi to make this work.

[fergyfresh]

Changing the app password didn't fix me. Hmmmm, this is gonna be a fun dig.

[digiltd]

yeah my "it works" was a bit premature, it appeared to work on the skill test page (I wasn't expecting audio, I know audio device stuff doesn't work on the test page), but when i tried it on the device nothing played, logs telling me "forbidden"

not sure how the S3 bucket fits in, i created a bucket and added it to the env vars, but not sure if i am supposed to grant permissions or anything

though it sounds like you are also having trouble again, did you have the aws method working for a month or did you get "banned" after a couple of weeks (also can you elaborate what you mean by that?, Google close your whole account?)

[digiltd]

ok another update

i have got it to work again through the device

i added (hardcoded) an android device id (from an old tablet) into music.py and that seems to get past the "forbidden" page

well, it has done for 2 songs

[fergyfresh]

Yeah. I'll have to add that to the README as well. I didn't hardcode it, I used Mobileclient.get_registered_devices() from the shell manually to find my currently valid device ids and then set that as ANDROID_ID env-var in my zappa_settings.json.

That was the next thing I was going to try so I really appreciate you working it out for me! The only reason why I didn't try it first was because I skimmed through the Issues on the https://github.com/simon-weber/gmusicapi project and there was an issue about device ids, but the 403 error explicitly expressed an issue with the device id.

[drkmullins]

Regarding the fixes from @digiltd , I dragged Crypto to site-packages, and also copied and replaced the existing Cryptodome folder with that same Crypto folder. Seems like there's probably a better way, but that worked for me. Thanks for the info.

Now to find a valid android ID 😄

[fergyfresh]

Well that seems to only be an issue for Mac users. I don't have that problem.