Careful with Mac deployment on Lambda

[soapergem]

I'm trying to get this deployed to Lambda/API Gateway using Zappa as outlined in the readme. However, everytime I call zappa deploy dev (or zappa update dev) it gives me the following error about API Gateway:

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 502 response code.

Also, if I try going to the API Gateway URL directly I just see this:

{
  "message": "An uncaught exception happened while servicing this request. You can investigate this with the `zappa tail` command.",
  "traceback": [
    "Traceback (most recent call last):\n",
    "  File '/var/task/handler.py', line 505, in handler\n    response = Response.from_app(self.wsgi_app, environ)\n",
    "  File '/var/task/werkzeug/wrappers.py', line 939, in from_app\n    return cls(*_run_wsgi_app(app, environ, buffered))\n",
    "  File '/var/task/werkzeug/wrappers.py', line 59, in _run_wsgi_app\n    return _run_wsgi_app(*args)\n",
    "  File '/var/task/werkzeug/test.py', line 923, in run_wsgi_app\n    app_rv = app(environ, start_response)\n",
    "TypeError: 'NoneType' object is not callable\n"
  ]
}

Tagging @fergyfresh since I believe he was the one who did the Zappa integration.

[fergyfresh]

what does the zappa tail tell you?

do something like zappa tail dev --since 1m # 1 minute i could help you more after seeing that.

[soapergem]

Thank you so much for the tip on using zappa tail - that has been immensely helpful in debugging problems this weekend. Unfortunately I still am having problems after over 10 hours of different trials with different versions of Zappa, Python, and virtualenv tools.

I wonder if perhaps you could go into Lambda and export your function's deployment ZIP for me, and host that somewhere where I could download it? That might be the fastest path to fruition at this point, as I feel like I've thrown everything and the kitchen sink at this problem, trying to get it to work. Plus I'd like to compare your presumably-working deployment package with my own.

So here's a not-so-quick history of what I've tried...

First I wanted to make things easier and use a deterministic build tool like pipenv to handle the installation of packages and virtual environment management. However that uses PIP 10 internally, and it seems both Flask and Flask-Ask (the current versions in PyPI anyway) won't work with PIP 10 because of their own buggy code. I checked the Flask-Ask repo and saw that they already have bugfixes committed for PIP 10 issues, but who knows how long it will be before those changes make their way to a release and then the new release versions percolate up to projects like Zappa.

So instead I went with virtualenvwrapper to create my venv, and then downgraded to pip 9.0.1 once inside. This seemed to work but I had still had problems getting Flask and Flask-Ask to install correctly. When I ran my zappa tail commands here I kept getting errors like this:

ModuleNotFoundError: No module named 'flask'

I saw an issue on the Zappa Github that others were getting this same error as well. So I tried all of their suggestions - which involved using different requirements.txt files or even different versions of Python. For instance I tried doing it with Python 2.7 but that failed too - I don't remember the specific problem but I know it failed.

And that was on my my Windows machine, running Ubuntu WSL. The WSL version also had troubles because it doesn't have python3-dev so it can't install Python-Levenshtein. So eventually I switched over to my Mac. On the Mac I encountered a variety of other headaches... first I couldn't get Homebrew to link Python correctly (you have to add sbin to the PATH), then I had to downgrade from Python 3.7 to 3.6 (since 3.7 is the current version which is not yet supported by zappa). Finally, miraculously, I was able to get everything deployed through Zappa without the 502 complaint. It even said "Your updated zappa deployment is live!"

Unfortunately this was a lie. This is where I encountered another error. When I go to the deployed URL I get the same TypeError: 'NoneType' object is not callable. But for a different reason than before. Before I believe it had something to do with zappa not bundling Flask and Flask-Ask in the deployment package. But now when I do zappa tail I see this error:

[1533509465006] /var/task/fuzzywuzzy/fuzz.py:35: UserWarning: Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning
[1533509465006] warnings.warn('Using slow pure-python SequenceMatcher. Install python-Levenshtein to remove this warning')
[1533509467046] Cannot load native module 'Cryptodome.Cipher._raw_ecb': OSError
Traceback (most recent call last):
  File "/var/task/handler.py", line 509, in lambda_handler
  return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 237, in lambda_handler
  handler = cls()
  File "/var/task/handler.py", line 129, in __init__
  self.app_module = importlib.import_module(self.settings.APP_MODULE)
  File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/var/task/geemusic/__init__.py", line 7, in <module>
  from .utils.music import GMusicWrapper
  File "/var/task/geemusic/utils/music.py", line 7, in <module>
  from gmusicapi import CallFailure, Mobileclient
  File "/var/task/gmusicapi/__init__.py", line 4, in <module>
  from gmusicapi.clients import Webclient, Musicmanager, Mobileclient
  File "/var/task/gmusicapi/clients/__init__.py", line 4, in <module>
  from gmusicapi.clients.webclient import Webclient
  File "/var/task/gmusicapi/clients/webclient.py", line 20, in <module>
  import gmusicapi.session
  File "/var/task/gmusicapi/session.py", line 11, in <module>
  import gpsoauth
  File "/var/task/gpsoauth/__init__.py", line 4, in <module>
  from . import google
  File "/var/task/gpsoauth/google.py", line 4, in <module>
  from Cryptodome.PublicKey import RSA
  File "/var/task/Cryptodome/PublicKey/RSA.py", line 79, in <module>
  from Cryptodome.IO import PKCS8, PEM
  File "/var/task/Cryptodome/IO/PKCS8.py", line 73, in <module>
  from Cryptodome.IO._PBES import PBES1, PBES2, PbesError
  File "/var/task/Cryptodome/IO/_PBES.py", line 44, in <module>
  from Cryptodome.Cipher import DES, ARC2, DES3, AES
  File "/var/task/Cryptodome/Cipher/__init__.py", line 78, in <module>
  from Cryptodome.Cipher._mode_ecb import _create_ecb_cipher
  File "/var/task/Cryptodome/Cipher/_mode_ecb.py", line 46, in <module>
  """
  File "/var/task/Cryptodome/Util/_raw_api.py", line 171, in load_pycryptodome_raw_lib
  raise OSError("Cannot load native module '%s'" % name)
OSError: Cannot load native module 'Cryptodome.Cipher._raw_ecb'

I Googled for that error and an issue came up which implies I'm missing an optional dependency. But their solution (installing python-levenshtein) doesn't work because pip just says it's already installed.

I am pretty much at my wit's end right now trying and repeatedly failing to get this to work. Which is why I'm wondering if comparing the ZIP deployment packages might shed some light. Also any info about your system (were you on a Mac? Windows? Linux?) when you deployed your copy would be helpful too.

[soapergem]

Oh and one last thing - what on Earth should I be setting the ANDROID_ID environment variable to? I don't have any Android devices.

[fergyfresh]

I was on WSL. why dont you use virtualenv with python3-dev on WSL. Mine was 3.6.4 if I recall, but that shouldn't matter. Source the env and install the requirements.txt dependencies from the root of the project. That worked. There is also a comment from @digitald on the original PR that implemented these changes.

[fergyfresh]

The ANDROID_ID is the only current workaround I have for using this on a PAAS such as AWS Lambda

[fergyfresh]

If you login using the example code for the Mobileclient class in gmusicapi and use the function get_registered_devices, you can be a Guinea pig for the rest of the non android users. If you dont use the android I'd, though, eventually your shit is gonna be blocked by google. They need a valid device id in order to validate the context of the request, or at least that is how I understand it

[soapergem]

First of all, afaik there isn't a way to install python3-dev on WSL - not the 3.6 version anyway. WSL comes with Python 3.5 which isn't sufficient for zappa; you have to use the third party ppa:jonathonf/python-3.6 apt repo to install it and even then it's just python -- not python3-dev. If there is some other way to install the 3.6 version of python3-dev on WSL please let me know! Because I haven't found any.

Secondly, I confess I don't know what you mean when you say "source the env" - forgive my ignore but I just don't know that nomenclature.

Third, I tried running pip install -r requirements.txt (after downgrading to pip==9.0.1 inside the venv again - doesn't work without that step) to install the exact requirements.txt file you provide in the root of the project. It's the requirement.txt file which references your fork of Flask-Ask. I tried it again on my Mac (where I've had the most success so far), running Python 3.6. That wasn't enough I still got the Cryptodome error.

I also tried @digiltd 's suggestion (from that other PR you mentioned) of moving around the cryptography/pycrypto files from the lambda_packages folder, but in the end this didn't make a difference. I still get the same Cryptodome error when I run zappa tail.

So I'd like to ask once again, can you go into your Lambda function, hit Actions > Export Function, click Download Deployment Package, and share that ZIP file somewhere? You'd probably want to then remove the zappa_settings.json from the file and re-zip if you're worried about env vars. Thank you in advance.

[fergyfresh]

I actually lied to you. Its python3.6-dev

[fergyfresh]

My python version is 3.6.3

[soapergem]

Holy crap I think I got it working. And side note: wow - given how many headaches I encountered along the way, this Zappa solution really isn't ready for primetime yet.

It occurred to me that if my remaining problems were Mac-specific, I could just try doing this whole Zappa deploy process from inside a Docker container, thereby effectively from within Linux, and just copy all my files there and run all the Python commands there too, as if the container were my build machine. Once I did that it seems like I got past all the errors I was having. Still had to downgrade the pip version once inside the venv of course.

At first I got a invalid device ID in Zappa tail, but saw it suggested I use some generated device ID (that starts with the string "ios")... and I am very relieved to report that has actually been working! (so far)

[soapergem]

This is immaterial at this point but I can't find python3.6-dev in any of the WSL repos. Did you have to add a custom one?

[fergyfresh]

https://stackoverflow.com/questions/41712326/how-to-install-python3-6-in-window-ubuntu-bash

[fergyfresh]

Can you send me what you're zappa settings json looks like? Maybe redact the private stuff, but use redacted1, redacted2, etc... for things that are the same, but redacted.

[fergyfresh]

Yeah @soapergem the problems with your deployment i think is more Lambda with Mac related than Zappa related. There are some conflicts between Lambda python packages and Mac python packages. For a few of them atleast if i remember correctly.

[fergyfresh]

I found a link that is a tutorial for setting up a docker container with the same environment as the Lambda function: https://blog.zappa.io/posts/simplified-aws-lambda-deployments-with-docker-and-zappa

@soapergem do you think you'd be able to try this after I get it working in my environment? I think this would be the ultimate breakthrough to making the Lambda deployment as streamlined as possible.

[soapergem]

@fergyfresh in effect I already have. His mcrowson/zappa-builder container is an extension of the lambci/lambda Docker container which has every different runtime under the sun. When I used Docker on my Mac to do the Zappa deploy I made a simple image based on python:3.6-stretch instead. But in just about every way it was functionally the same to exactly what he's outlined in the guide you linked. Even down to passing in the AWS access environment variables. So without even looking I can tell you that yes, it would work, as I've already done it - just with a slightly different base container.

As far as making this easy for others to run, we of course can't add a new Dockerfile definition to the existing repo as it would conflict with the one there now, but I could modify his docker run line to be the appropriate one to run here. In fact I'll make a short PR to the Readme file sometime this week to set that up.

In other news, remember how you said I could be the guinea pig for the non-Android users? Well long story short it isn't working - I'm seeing some forbidden errors. With my iOS device ID in place I can run commands like "ask gee music to play songs by Taylor swift" and it even shows the album art in the Alexa Test Console, but it then fails to actually play anything. Apparently getting the media URLs are returning Forbidden errors.

So I think I'm going to have to get my hands on a friend's Android device, although I was hoping we could build out more documentation about what needs to be done there to get an Android ID. Having not used Android at all I don't really know where to start.

[fergyfresh]

As for the docker stuff, thanks so much for clarifying. I didn't know if you went the docker route or the manually swapping out the python packages like @digiltd did.

As for the iOS device ID stuff, I'm sorry to hear that. The issue resides in the fact that about 6 months ago Google Play Services changed their OAuth scheme to require a device ID that is generated as a result of your MAC address on your device with some special sauce (for Android phones). Without knowing the differences between the special sauce in Android vs. iOS and having none of the maintainers of gmusicapi (the base repo that makes all of this possible) it's hard to tell how to 'hack' around the integration for iOS users to use the PAAS solutions provided in the README here.

Is there any context around the 'forbidden' errors. Also I believe a user from Eastern Europe was able to get this working using iOS I believe. Basically the device ID for android is 0x123456ABDCEF and the ID I use is everything after the 0x. I believe the iOS device ID is something like ios:0123456789 and he used 0123456789. The length of the IDs in my example post here probably aren't correct, but basically just remove the prefix and it should just work. Try to scour the issues with me and see if you cant find it before I do. If I find that issue I will tag it in this issue as well, since it is related to something you're seeing.

[fergyfresh]

The post I was talking about originally was #232, which moved us over to #220 which @Bronskiy is the person I said was from Eastern Europe, he's actually from Russia.

[fergyfresh]

Is anyone still following this issue? I think I have a fix for this that I wanna try tommorrow. It revolves around using a docker container to zappa deploy instead of patching in separate modules on a mac.