serverless deploy fails with 404 'Function not found...'

[wkhatch]

I'm attempting to create a containerized local dev environment using serverless and localstack. I can run the services using docker-compose, and access both containers, run sls commands, etc. When I try to deploy, it eventually fails with a 404 as mentioned, with these container logs:

ack    | 2020-04-14T19:00:17:INFO:localstack.services.cloudformation.cloudformation_starter: Resource BdpipeLambdaFunction cannot be deployed, found unsatisfied dependencies. {'Type': 'AWS::Lambda::Function', 'Properties': {'Code': {'S3Bucket': '__local__', 'S3Key': '/usr/src/app'}, 'FunctionName': 'api-local-bdpipe', 'Handler': 'src/handlers/bdpipe/handler.bdpipe', 'MemorySize': 1024, 'Role': 'arn:aws:iam::123456789012:role/api-local-us-east-1-lambdaRole', 'Runtime': 'nodejs12.x', 'Timeout': 6}, 'DependsOn': ['BdpipeLogGroup', 'IamRoleLambdaExecution']}
localstack    | 2020-04-14T19:00:17:WARNING:localstack.services.awslambda.lambda_api: Function not found: arn:aws:lambda:us-east-1:000000000000:function:api-local-bdpipe
localstack    | 2020-04-14T19:00:17:DEBUG:localstack.services.cloudformation.cloudformation_starter: Deploying CloudFormation resource (update=False, exists=False, updateable=False): {'Type': 'AWS::Lambda::Version', 'DeletionPolicy': 'Retain', 'Properties': {'FunctionName': 'api-local-bdpipe', 'CodeSha256': 'v7Fnk3nzhFrM3Cu8+79Z9s/bxeV0jJD90uBDvW+/+08='}}
localstack    | 2020-04-14T19:00:17:DEBUG:localstack.utils.cloudformation.template_deployer: Running action "create" for resource type "Lambda::Version" id "BdpipeLambdaVersionUqVEqKWOcJPDf9qm4UPGlmcvUxD3pvE95yVtV3biwpg"
localstack    | 2020-04-14T19:00:17:DEBUG:localstack.utils.cloudformation.template_deployer: Request for resource type "Lambda::Version" in region us-east-1: publish_version {'FunctionName': 'api-local-bdpipe', 'CodeSha256': 'v7Fnk3nzhFrM3Cu8+79Z9s/bxeV0jJD90uBDvW+/+08='}
localstack    | 2020-04-14T19:00:17:WARNING:localstack.services.awslambda.lambda_api: Function not found: arn:aws:lambda:us-east-1:000000000000:function:api-local-bdpipe
localstack    | 2020-04-14T19:00:17:WARNING:localstack.utils.cloudformation.template_deployer: Error calling <bound method ClientCreator._create_api_method.<locals>._api_call of <botocore.client.Lambda object at 0x7fc929806340>> with params: {'FunctionName': 'api-local-bdpipe', 'CodeSha256': 'v7Fnk3nzhFrM3Cu8+79Z9s/bxeV0jJD90uBDvW+/+08='} for resource: {'Type': 'AWS::Lambda::Version', 'DeletionPolicy': 'Retain', 'Properties': {'FunctionName': 'api-local-bdpipe', 'CodeSha256': 'v7Fnk3nzhFrM3Cu8+79Z9s/bxeV0jJD90uBDvW+/+08='}}
localstack    | 2020-04-14T19:00:17:ERROR:localstack.services.cloudformation.cloudformation_starter: Unable to parse and create resource "BdpipeLambdaVersionUqVEqKWOcJPDf9qm4UPGlmcvUxD3pvE95yVtV3biwpg": An error occurred (ResourceNotFoundException) when calling the PublishVersion operation: Function not found: arn:aws:lambda:us-east-1:000000000000:function:api-local-bdpipe Traceback (most recent call last):
localstack    |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 183, in parse_and_create_resource
localstack    |     return _parse_and_create_resource(
localstack    |   File "/opt/code/localstack/localstack/services/cloudformation/cloudformation_starter.py", line 292, in _parse_and_create_resource
localstack    |     result = deploy_func(logical_id, resource_map_new, stack_name=stack_name)
localstack    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 961, in deploy_resource
localstack    |     return execute_resource_action(resource_id, resources, stack_name, ACTION_CREATE)
localstack    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 987, in execute_resource_action
localstack    |     result = configure_resource_via_sdk(resource_id, resources, resource_type, func, stack_name)
localstack    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1064, in configure_resource_via_sdk
localstack    |     raise e
localstack    |   File "/opt/code/localstack/localstack/utils/cloudformation/template_deployer.py", line 1061, in configure_resource_via_sdk
localstack    |     result = function(**params)
localstack    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 316, in _api_call
localstack    |     return self._make_api_call(operation_name, kwargs)
localstack    |   File "/opt/code/localstack/.venv/lib/python3.8/site-packages/botocore/client.py", line 626, in _make_api_call
localstack    |     raise error_class(parsed_response, operation_name)
localstack    | botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the PublishVersion operation: Function not found: arn:aws:lambda:us-east-1:000000000000:function:api-local-bdpipe
localstack    | 
localstack    | 2020-04-14T19:00:17:DEBUG:localstack.services.cloudformation.cloudformation_listener: Error response for CloudFormation action "UpdateStack" (400) POST /: b'Function not found: arn:aws:lambda:us-east-1:000000000000:function:api-local-bdpipe'

Here's my serverless.yml:

service: api

provider:
  name: aws
  runtime: nodejs12.x
  stage: local
  region: us-east-1

plugins:
  - serverless-offline
  - serverless-localstack

functions:
  - '${file(src/handlers/bdpipe/bdpipe.serverless.yml)}'

deploymentBucket:
  name: iotr-functions

custom:
  localstack:
    debug: true
    stages:
      - local
      - dev
    host: http://localstack # optional - LocalStack host to connect to
    endpoints:
      S3: http://localstack:4572
      # DynamoDB: http://localocalstacklhost:4570
      CloudFormation: http://localstack:4581
      IAM: http://localstack:4593
      CloudWatchLogs: http://localstack:4586
      # Elasticsearch: http://localstack:4571
      # ES: http://localstack:4578
      # SNS: http://localstack:4575
      # SQS: http://localstack:4576
      Lambda: http://localstack:4574
      APIGateway: http://localstack:4567
      # Kinesis: http://localstack:4568
    lambda:
      mountCode: True
    docker:
      sudo: false
  serverless-offline:
    host: 0.0.0.0 # for docker container hosting

And my docker-compose:

`version: '3'

services:
  api:
    build: .
    image: local/api
    depends_on:
      - localstack
    ports:
      - '3000:3000'
    container_name: local_api
    volumes:
      - .:/usr/src/app

    # these are the environment variables that are used in the api
    environment:
      AWS_ENDPOINT: 'http://localstack:8000' # localstack container host for dynamodb
      AWS_REGION: 'us-east-1'
      AWS_ACCESS_KEY_ID: 'fake-access-key-id'
      AWS_SECRET_ACCESS_KEY: 'fake-secret-key'
      LOCALSTACK_HOST: 'localstack' # for awslocal cli tool

  localstack:
    image: localstack/localstack:latest
    container_name: localstack
    ports:
      - '8080:8080'
      - '4540-4599:4540-4599'
    environment:
      - SERVICES=s3,lamda,cloudformation,apigateway,serverless
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
      - LAMBDA_EXECUTOR=docker
      - DEFAULT_REGION=us-east-1
      - LAMBDA_REMOTE_DOCKER=true
      - LAMBDA_DOCKER_NETWORK=localstack_default
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - './.localstack:/tmp/localstack'
      - '/var/run/docker.sock:/var/run/docker.sock'
`Any pointers on how to better debug this greatly appreciated. Thanks.

┆Issue is synchronized with this Jira Task by Unito

[hecjhs]

I got the same problem

[hecjhs]

@wkhatch do you have any luck with this?

[wkhatch]

@hecjhs not yet. I'm going to attempt a few more configuration changes, then maybe dig into source.

[bharadwajturlapati]

Same problem any leads on this ?

[rogueturnip]

I'm seeing the same issue here. Any one found a solution?

[wkhatch]

I have been able to get mine working. I modified some of the serverless config with respect to remote docker and a few other things, but I'm able to initiate a successful sls deploy, from my serverless client container, against the localstack container running in the same compose stack. Also, I took out serverless-local from the mix entirely. Sorry it took me so long to get back to this. After my first attempt, I had to move onto some other stuff before coming back to it, and I just got all this squared a few days ago. Below are some relevant configuration snippets:

First, the service as configured in docker-compose, for localstack:

localstack:
    image: localstack/localstack-full
    container_name: localstack
    ports:
      - '8080:8080'
      - '4540-4599:4540-4599'
    env_file:
      - .env.local
    volumes:
      - '/var/run/docker.sock:/var/run/docker.sock'
      - './.localstack:/tmp/localstack' #this can create problems as it will persist stack data between runs

The .env.local:

AWS_ENDPOINT='http://localstack:8000'
AWS_REGION='us-east-1'
AWS_ACCESS_KEY_ID='fake-access-key-id'
AWS_SECRET_ACCESS_KEY='fake-secret-key'

LOCALSTACK_HOST='localstack' # for awslocal cli tool
SERVICES=edge,s3,lambda,cloudformation,apigateway,serverless, kinesis, sqs, firehose, iot 
DEBUG=1
DATA_DIR=/tmp/localstack/data
LAMBDA_EXECUTOR=docker
DEFAULT_REGION=us-east-1
# LAMBDA_REMOTE_DOCKER=true
# LAMBDA_DOCKER_NETWORK=localstack_default
SLS_DEBUG=*
DOCKER_HOST=unix:///var/run/docker.sock

custom block in serverless.yml:

custom:
  localstack:
    debug: true
    stages:
      - local
      - dev
    host: http://localstack # optional - LocalStack host to connect to
    edgePort: 4566
    lambda:
      mountCode: true
    docker:
      sudo: false

Important differences are the following two lines being commented out:

LAMBDA_REMOTE_DOCKER=true

LAMBDA_DOCKER_NETWORK=localstack_default

and, I upgraded to the latest localstack version, which completely removes the single service/port configuration in favor of the unified host:... and edgePort: ... Their docker image from 2.0 on has some changes, so I also modified the images I'm pulling to be ..._full so I still get all the services pulled down as opposed to dynamically at run time. Not sure if that's relevant to our shared problem, but mentioning it for completeness. Hope this helps, and feel free to tag me in any questions; happy to do what I can to help. It was certainly a super frustrating issue for me lol

[whummer]

Thanks for reporting @wkhatch @bharadwajturlapati @rogueturnip @hecjhs . In the meantime we've made several enhancements in the LocalStack CloudFormation and Lambda implementations - can you please pull the latest Docker image and let us know if the problem persists? Thanks for your help!

[steffyP]

Closing the issue due to inactivity. The issue should be fixed in the latest localstack image.