arn:aws:sns:::topic_name failing

cscetbon commented 7 years ago

Trying to follow the documentation, I've met an issue when trying to use an SNS event.

A workaround is to add region and account id, but we shouldn't have to do it and that's why I consider it as a Bug.

## Expected Behavior

I should not get any error and the app should be deployed.

Actual Behavior

see https://pastebin.com/raw/K9AwXHbf

Possible Fix

Steps to Reproduce

use the following sample code test.py :

from flask import Flask
from zappa.async import task_sns
app = Flask(__name__)

@task_sns
def make_pie():
    """ This takes a long time! """
    print "make_pie()"

@app.route('/api/order/pie')
def order_pie():
    """ This returns immediately! """
    make_pie()
    return "Your pie is being made!"

with zappa_settings.json similar to the one provided below.

Your Environment

Zappa version used: 0.42.0
Operating System and Python version: MacOs 10.12, Python 2.7.13

The output of pip freeze:

appdirs==1.4.3
appnope==0.1.0
argcomplete==1.8.2
backports.shutil-get-terminal-size==1.0.0
base58==0.2.4
boto==2.46.1
boto3==1.4.4
botocore==1.5.40
click==6.7
decorator==4.0.11
docutils==0.13.1
durationpy==0.4
enum34==1.1.6
Flask==0.12.2
future==0.16.0
futures==3.1.1
hjson==2.0.2
ipython==5.3.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.9.6
jmespath==0.9.2
kappa==0.6.0
lambda-packages==0.15.0
MarkupSafe==1.0
packaging==16.8
pathlib2==2.2.1
pexpect==4.2.1
pickleshare==0.7.4
placebo==0.8.1
prompt-toolkit==1.0.14
ptyprocess==0.5.1
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.0
python-slugify==1.2.4
PyYAML==3.12
requests==2.14.2
s3transfer==0.1.10
scandir==1.5
simplegeneric==0.8.1
six==1.10.0
toml==0.9.2
tqdm==4.11.2
traitlets==4.3.2
troposphere==1.9.3
Unidecode==0.4.20
wcwidth==0.1.7
Werkzeug==0.12
wsgi-request-logger==0.4.6
zappa==0.42.0

Link to your project (optional):

Your zappa_settings.py:

{
"pipo": {
    "app_function": "test.app",
    "aws_region": "us-east-1",
    "profile_name": "default",
    "s3_bucket": "zappa-xxxx",
    "keep_warm": false,
"events": [
            {
                "function": "test.make_pie",
                "event_source": {
                    "arn":  "arn:aws:sns:::topic_name",
                    "events": [
                        "sns:Publish"
                    ]
                }
            }
        ]
    }
}

Miserlou commented 7 years ago

Thanks for reporting. Does the same behavior happen if you remove the events key-value from your settings? That decorator is supposed to handle that seamlessly.

cscetbon commented 7 years ago

@Miserlou, I had to add the event cause I was getting the following error :

[1495411919625] Traceback (most recent call last):
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-7CyJnE/flask/flask/app.py", line 1982, in wsgi_app
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-7CyJnE/flask/flask/app.py", line 1614, in full_dispatch_request
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-7CyJnE/flask/flask/app.py", line 1517, in handle_user_exception
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-7CyJnE/flask/flask/app.py", line 1612, in full_dispatch_request
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-7CyJnE/flask/flask/app.py", line 1598, in dispatch_request
[1495411919625] File "/var/task/test.py", line 16, in order_pie
[1495411919625] make_pie()
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-9iUMx6/zappa/zappa/async.py", line 306, in _run_async
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-9iUMx6/zappa/zappa/async.py", line 141, in send
[1495411919625] File "/private/var/folders/zw/v7zl8lnd3kx6wt6xvpnpsq6nzlczyd/T/pip-build-9iUMx6/zappa/zappa/async.py", line 195, in _send
[1495411919625] File "/var/runtime/botocore/client.py", line 253, in _api_call
[1495411919625] return self._make_api_call(operation_name, kwargs)
[1495411919625] File "/var/runtime/botocore/client.py", line 557, in _make_api_call
[1495411919625] raise error_class(parsed_response, operation_name)
[1495411919625] NotFoundException: An error occurred (NotFound) when calling the Publish operation: Topic does not exist

Are you saying that when we use task_sns, Zappa is supposed to create one SNS topic per task ? or one SNS topic for the whole app ? Is it possible to have the same function triggered by SNS events and by another function using task_sns ? or should I implement it using 2 different functions calling the same code ?

cscetbon commented 7 years ago

Hey @Miserlou, were you able to find out what's going on ?

cscetbon commented 7 years ago

@Miserlou so digging into the issue, I found that I needed to add "async_source": "sns" to my settings to have the SNS topic created. But then I get a new issue that is :

Cannot find a function to process the triggered event

This error happens when the lambda receives the SNS event. It does https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L298-L299 So you see that it expects the event to be defined in settings. It should instead use the command in u'Message': u'{"args": [], "command": "zappa.async.route_sns_task", "task_path": "test.make_pie", "kwargs": {}}' or return task_path which is the fastest.

something similar to https://github.com/Miserlou/Zappa/blob/master/zappa/handler.py#L331-L339 should be done in case Message contains a command or task_path in records[0].Message. But to do it, we need to load the Json string into a python object as in case of an SNS event, the command value is not directly accessible. That's why I'm thinking that we should directly use task_path cause otherwise we would load the Json string twice

Miserlou / Zappa