search

aws / aws-sam-cli

CLI tool to build, test, debug, and deploy Serverless applications using AWS SAM
https://aws.amazon.com/serverless/sam/
Apache License 2.0
6.5k stars 1.17k forks source link

500 Server Error: Internal Server Error ("trying to send message larger than max (21062276 vs. 16777216): unknown") #1228

Closed ccornici closed 5 years ago

ccornici commented 5 years ago

Description

Can't debug lambda function locally

Steps to reproduce

1) Have any lambda function sample at hand. 2) run sam local invoke -d 5858 -e insanely_large_20_mb_file.json groupdata

Observed result

2019-06-15 08:31:14 Using SAM Template at C:\Projects\ev_data_grouping\template.yaml
2019-06-15 08:31:14 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-06-15 08:31:14 Changing event name from before-call.apigateway to before-call.api-gateway
2019-06-15 08:31:14 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-06-15 08:31:14 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-06-15 08:31:14 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-06-15 08:31:14 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-06-15 08:31:14 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-06-15 08:31:14 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-06-15 08:31:14 Changing event name from before-call.apigateway to before-call.api-gateway
2019-06-15 08:31:14 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-06-15 08:31:14 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-06-15 08:31:14 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-06-15 08:31:14 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-06-15 08:31:14 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-06-15 08:31:14 local invoke command is called
2019-06-15 08:31:14 No Parameters detected in the template
2019-06-15 08:31:14 1 resources found in the template
2019-06-15 08:31:14 Found Serverless function with name='groupdata' and CodeUri='.'
2019-06-15 08:31:14 Trying paths: ['C:\\Users\\claud\\.docker\\config.json', 'C:\\Users\\claud\\.dockercfg']
2019-06-15 08:31:14 Found file at path: C:\Users\claud\.docker\config.json
2019-06-15 08:31:14 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-06-15 08:31:14 Config entry for key stackOrchestrator is not auth config
2019-06-15 08:31:14 http://localhost:None "GET /v1.35/_ping HTTP/1.1" 200 2
2019-06-15 08:31:14 Trying paths: ['C:\\Users\\claud\\.docker\\config.json', 'C:\\Users\\claud\\.dockercfg']
2019-06-15 08:31:14 Found file at path: C:\Users\claud\.docker\config.json
2019-06-15 08:31:14 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-06-15 08:31:14 Config entry for key stackOrchestrator is not auth config
2019-06-15 08:31:14 Found one Lambda function with name 'groupdata'
2019-06-15 08:31:14 Invoking index.handler (nodejs10.x)
2019-06-15 08:31:14 No environment variables found for function 'groupdata'
2019-06-15 08:31:14 Environment variables overrides data is standard format
2019-06-15 08:31:14 Loading AWS credentials from session with profile 'None'
2019-06-15 08:31:14 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-06-15 08:31:14 Changing event name from before-call.apigateway to before-call.api-gateway
2019-06-15 08:31:14 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-06-15 08:31:14 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-06-15 08:31:14 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-06-15 08:31:14 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2019-06-15 08:31:14 Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2019-06-15 08:31:14 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-06-15 08:31:14 Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-06-15 08:31:14 Looking for credentials via: env
2019-06-15 08:31:14 Looking for credentials via: assume-role
2019-06-15 08:31:14 Looking for credentials via: shared-credentials-file
2019-06-15 08:31:14 Looking for credentials via: custom-process
2019-06-15 08:31:14 Looking for credentials via: config-file
2019-06-15 08:31:14 Looking for credentials via: ec2-credentials-file
2019-06-15 08:31:14 Looking for credentials via: boto-config
2019-06-15 08:31:14 Looking for credentials via: container-role
2019-06-15 08:31:14 Looking for credentials via: iam-role
2019-06-15 08:31:14 Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
2019-06-15 08:31:14 Starting new HTTP connection (1): 169.254.169.254:80
2019-06-15 08:31:14 Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\util\connection.py", line 80, in create_connection
    raise err
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\util\connection.py", line 70, in create_connection
    sock.connect(sa)
OSError: [WinError 10051] A socket operation was attempted to an unreachable network

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\httpsession.py", line 262, in send
    chunked=self._chunked(request.headers),
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\util\retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\packages\six.py", line 686, in reraise
    raise value
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "c:\program files\python37\lib\http\client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\awsrequest.py", line 125, in _send_request
    method, url, body, headers, *args, **kwargs)
  File "c:\program files\python37\lib\http\client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "c:\program files\python37\lib\http\client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\awsrequest.py", line 152, in _send_output
    self.send(msg)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\awsrequest.py", line 236, in send
    return super(AWSConnection, self).send(str)
  File "c:\program files\python37\lib\http\client.py", line 956, in send
    self.connect()
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 183, in connect
    conn = self._new_conn()
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\urllib3\connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPConnection object at 0x00000238035D3898>: Failed to establish a new connection: [WinError 10051] A socket operation was attempted to an unreachable network

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\utils.py", line 303, in _get_request
    response = self._session.send(request.prepare())
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\botocore\httpsession.py", line 282, in send
    raise EndpointConnectionError(endpoint_url=request.url, error=e)
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
2019-06-15 08:31:14 Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.
2019-06-15 08:31:14 Resolving code path. Cwd=C:\Projects\ev_data_grouping, CodeUri=.
2019-06-15 08:31:14 Resolved absolute path to code is C:\Projects\ev_data_grouping
2019-06-15 08:31:14 Code C:\Projects\ev_data_grouping is not a zip/jar file
2019-06-15 08:31:14 Skipping building an image since no layers were defined
2019-06-15 08:31:14 Trying paths: ['C:\\Users\\claud\\.docker\\config.json', 'C:\\Users\\claud\\.dockercfg']
2019-06-15 08:31:14 Found file at path: C:\Users\claud\.docker\config.json
2019-06-15 08:31:14 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-06-15 08:31:14 Config entry for key stackOrchestrator is not auth config
2019-06-15 08:31:14 http://localhost:None "GET /v1.35/images/lambci/lambda:nodejs10.x/json HTTP/1.1" 200 None
2019-06-15 08:31:14 Looking for auth config
2019-06-15 08:31:14 No auth config in memory - loading from filesystem
2019-06-15 08:31:14 Trying paths: ['C:\\Users\\claud\\.docker\\config.json', 'C:\\Users\\claud\\.dockercfg']
2019-06-15 08:31:14 Found file at path: C:\Users\claud\.docker\config.json
2019-06-15 08:31:14 Couldn't find auth-related section ; attempting to interpret as auth-only file
2019-06-15 08:31:14 Config entry for key stackOrchestrator is not auth config
2019-06-15 08:31:14 Looking for auth entry for 'docker.io'
2019-06-15 08:31:14 No entry found
2019-06-15 08:31:14 No auth config found
2019-06-15 08:31:16 http://localhost:None "POST /v1.35/images/create?tag=nodejs10.x&fromImage=lambci%2Flambda HTTP/1.1" 200 None

Fetching lambci/lambda:nodejs10.x Docker container image......
2019-06-15 08:31:16 Mounting C:\Projects\ev_data_grouping as /var/task:ro,delegated inside runtime container
2019-06-15 08:31:18 http://localhost:None "POST /v1.35/containers/create HTTP/1.1" 201 90
2019-06-15 08:31:18 http://localhost:None "GET /v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208/json HTTP/1.1" 200 None
2019-06-15 08:31:18 http://localhost:None "GET /v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208/json HTTP/1.1" 200 None
2019-06-15 08:31:19 http://localhost:None "POST /v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208/start HTTP/1.1" 500 86
2019-06-15 08:31:19 http://localhost:None "GET /v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208/json HTTP/1.1" 200 None
2019-06-15 08:31:20 http://localhost:None "DELETE /v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208?v=False&link=False&force=True HTTP/1.1" 204 0
Traceback (most recent call last):
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\requests\models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localnpipe/v1.35/containers/cef317b354308a91f78f25ab45d4ad4fbd59523accf47223254456082efeb208/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\program files\python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\program files\python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\Scripts\sam.exe\__main__.py", line 9, in <module>
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\invoke\cli.py", line 56, in cli
    parameter_overrides)  # pragma: no cover
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\invoke\cli.py", line 100, in do_cli
    stderr=context.stderr)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\commands\local\lib\local_lambda.py", line 93, in invoke
    self.local_runtime.invoke(config, event, debug_context=self.debug_context, stdout=stdout, stderr=stderr)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\local\lambdafn\runtime.py", line 86, in invoke
    self._container_manager.run(container)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\manager.py", line 98, in run
    container.start(input_data=input_data)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\samcli\local\docker\container.py", line 187, in start
    real_container.start()
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\models\containers.py", line 400, in start
    return self.client.api.start(self.id, **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\utils\decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\api\container.py", line 1093, in start
    self._raise_for_status(res)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\api\client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "C:\Users\claud\AppData\Roaming\Python\Python37\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("trying to send message larger than max (21062276 vs. 16777216): unknown")

Expected result

Being able to get the result advertised in the AWS docs when running the above command

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Windows 10
  2. sam --version: SAM CLI, version 0.17.0
htmljosh commented 5 years ago

In case you haven't sorted this out yet, your problem is most likely that you haven't shared your drive with docker. See this post: https://github.com/awslabs/aws-sam-cli/issues/909#issuecomment-451492844

awood45 commented 5 years ago

What result exactly are you looking for? This is definitely an input payload larger than the 6 MB limit assuming the filename is accurate. Looking for a different error?

ccornici commented 5 years ago

It was my mistake here, I didn't know that there's a 6 MB limit. So there is no issue, The message is as expected.