search

Miserlou / Zappa

Serverless Python
https://blog.zappa.io/
MIT License
11.89k stars 1.2k forks source link

Parameter validation failed when using @task_sns. #1323

Open Debetux opened 6 years ago

Debetux commented 6 years ago

Context

I want to invoke a task which use @task_sns.

E.g. code:

@task_sns
def my_async_task(data):
    a = data.get('a')
    b = data.get('b')
    return a + b

def my_endpoint():
    my_async_task({
        'a': 4,
        'b': 10
    })

    return jsonify({
        'success': True
    })

This code produces this error:

Parameter validation failed:
Invalid type for parameter Message, value: b'{"task_path": "my_async_task", "capture_response": false, "response_id": null, "args": [{"a": 4, "b": 10}], "kwargs": {}, "command": "zappa.async.route_sns_task"}', type: <class 'bytes'>, valid types: <class 'str'>: ParamValidationError
Traceback (most recent call last):
  File "/var/task/handler.py", line 509, in lambda_handler
    return LambdaHandler.lambda_handler(event, context)
  File "/var/task/handler.py", line 240, in lambda_handler
    return handler.handler(event, context)
  File "/var/task/handler.py", line 345, in handler
    result = self.run_function(app_function, event, context)
  File "/var/task/handler.py", line 270, in run_function
    result = app_function(event, context) if varargs else app_function()
  File "/var/task/main.py", line 81, in update_results
    my_async_task(search_serialized)
  File "/var/task/zappa/async.py", line 421, in _run_async
    capture_response=capture_response).send(task_path, args, kwargs)
  File "/var/task/zappa/async.py", line 170, in send
    self._send(message)
  File "/var/task/zappa/async.py", line 249, in _send
    Message=payload
  File "/var/runtime/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 586, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/var/runtime/botocore/client.py", line 641, in _convert_to_request_dict
    api_params, operation_model)
  File "/var/runtime/botocore/validate.py", line 291, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())

Expected Behavior

Invoking the task should not fail. Message argument sent to botocore must be a string, not bytes.

Actual Behavior

We can't invoke the task using a SNS channel, because botocore expect a string, not bytes.

Possible Fix

Steps to Reproduce

  1. use the code pasted above
  2. zappa invoke dev main.my_endpoint
  3. See the result.

Your Environment

lu911 commented 6 years ago

+1

guhakashyap commented 6 years ago

It's been over six months, and this issue is still open. As a user who would dearly love to see this get fixed - can anyone please look into this? Any idea when we can expect to see a fix be pushed in an update?

Caian commented 6 years ago

+1

relwell commented 6 years ago

What's the status on this? I'm surprised to see this doesn't work.

Debetux commented 6 years ago

@Miserlou What is missing for merging #1331 ? :) Thanks a lot!

relwell commented 5 years ago

It's been eight months since I dropped my last comment into this issue, and I've had to essentially monkey-patch the async file for all this time.

This seems to me like an issue related to supporting both Python2 and Python3's understanding of strings simultaneously. Python2 is reaching the end of its life this year, and so it's unreasonable for there to be broken functionality in Python3 in the interest of supporting old code. Python2 users should expect to be pinning versions left and right.

When can we expect this problem to be solved?

rorybyrne commented 5 years ago

I'm having the same issue, any advice for how to get this working @relwell ?

relwell commented 5 years ago

@RoryOfByrne here's a gist of my workaround: https://gist.github.com/relwell/3eb9d68526ba8eadb41b3b1399e747a9

you would import that file instead of the normal zappa_async module.

rorybyrne commented 5 years ago

Brilliant, thank you @relwell . Am I right in thinking that the only change is removing .encode('utf-8') - here? I'm working on my own patch right now and that's the change I'm making.

relwell commented 5 years ago

I think so, @RoryOfByrne. It's been a while since I inspected the solution.

gerosalesc commented 5 years ago

+1

gustavopergola commented 4 years ago

Also bumped into the same problem here. @relwell approach wasn't a fit for me, as it is not so easy to apply his monkey-patch on the env I'm working with, although it worked just fine. I declared the boto3 session manually, used a fixed ARN for the publish request and hard-coded the dict that zappa builds before sending to SNS. 

import boto3
message = {
    "task_path": "module.my_async_method",
    "capture_response": False,
    "response_id": None,
    "args": ["myarg1", "myarg2"],
    "kwargs": {},
    "command": "zappa.asynchronous.route_sns_task"
}
message = json.dumps(message)
aws_session = boto3.Session()
client = aws_session.client('sns')
client.publish(
    TargetArn="arn:aws:sns:zone:123123123:sns-topic-name",
    Message=message
)

It is basically the same as _send function from zappa but without the payload size exception and SNS message Id response. I hope the problem is solved soon so I can properly make SNS calls 😄

kornpow commented 2 years ago

Im also hitting this issue... hopefully it gets fixed soon?