localstack / serverless-localstack

⚡ Serverless plugin for running against LocalStack
519 stars 83 forks source link

Getting TemplateURL must be a supported URL while doing "sls deploy --stage dev" in mac #237

Closed prasaddls closed 8 months ago

prasaddls commented 9 months ago

Here is my serverless.yml

service: sample-service

configValidationMode: error

custom:
  localstack:
    stages:
      - ${sls:stage}
    host: http://localhost
    edgePort: 4566 
    autostart: true 
    debug: true
    lambda:
      mountCode: True
  CRUD_TOPIC: '${self:service}-${sls:stage}-CRUD'
  MEDICAL_LICENSE_SERVICE_TOPIC_ARN:
    Fn::ImportValue: 'medical-license-service-${sls:stage}-TopicArn'
  DYNAMO_ERROR_TOPIC: '${self:service}-${sls:stage}-DYNAMO_ERROR'
  SNS_DLQ:
    Fn::ImportValue: 'shared-infrastructure-${sls:stage}-SNSDLQArn'
  SQS_DLQ:
    Fn::ImportValue: 'shared-infrastructure-${sls:stage}-SQSDLQArn'
  MQ_MASTER_KEY:
    Fn::ImportValue: 'shared-infrastructure-${sls:stage}-MQMasterKeyAlias'
  MQ_MASTER_KEY_ARN:
    Fn::ImportValue: 'shared-infrastructure-${sls:stage}-MQMasterKeyArn'
  deployBot: ${file(../../sharedConfig.yml):deployBot}

provider:
  name: aws
  runtime: go1.x
  region: us-east-1  # or your preferred region
  profile: sample_${sls:stage}
  versionFunctions: true
  memorySize: 1792
  tracing:
    lambda: true
  environment:
    AWS_NODEJS_CONNECTION_REUSE_ENABLED: 1
    CRUD_TOPIC_ARN:
      Ref: CrudSNSTopic
    DYNAMO_ERROR_TOPIC_ARN:
      Ref: DynamoErrorSNSTopic
    DYNAMODB_TABLE: ${self:service}-${sls:stage}
    FORCE_DYNAMO_INGESTION_ERROR: 'false' # force dynamo ingestion errors to test DLQ
    NODE_OPTIONS: --enable-source-maps
    SERVICE: ${self:service}
    STAGE: ${sls:stage}
    CUSTOM_HOST: http://localhost:4566

package:
  individually: true
  exclude:
    - ./**

functions:
  createEmployee:
    handler: bin/createEmployee
    timeout: 900
    package:
      include:
        - ./bin/createEmployee
  enrollEmployeeInJET:
    handler: bin/enrollEmployeeinJET
    timeout: 900
    package:
      include:
        - ./bin/enrollEmployeenJET
  getOneEmployee:
    handler: bin/getOneEmployee
    timeout: 900
    package:
      include:
        - ./bin/getOneEmployee
  getEmployeesByAddress:
    handler: bin/getEmployeeByAddress
    timeout: 900
    package:
      include:
        - ./bin/getEmployeeByAddress
  getEmployeesByGroup:
    handler: bin/getEmployeesByGroup
    timeout: 900
    package:
      include:
        - ./bin/getEmployeesByGroup

resources:
  Resources:
    DynamoDbTable:
      Type: 'AWS::DynamoDB::Table'
      DeletionPolicy: ${file(../../sharedConfig.yml):dbDeletionPolicy.${sls:stage}}
      Properties:
        AttributeDefinitions:
          - AttributeName: PartitionKey
            AttributeType: S
          - AttributeName: SortKey
            AttributeType: S
          - AttributeName: PSI2_Partition
            AttributeType: S
          - AttributeName: PSI2_Sort
            AttributeType: S
          - AttributeName: PSI3_Partition
            AttributeType: S
          - AttributeName: PSI3_Sort
            AttributeType: S
          - AttributeName: PSI4_Partition
            AttributeType: S
          - AttributeName: PSI4_Sort
            AttributeType: S
        BillingMode: PAY_PER_REQUEST
        KeySchema:
          - AttributeName: PartitionKey
            KeyType: HASH
          - AttributeName: SortKey
            KeyType: RANGE
        PointInTimeRecoverySpecification:
          PointInTimeRecoveryEnabled: ${file(../../sharedConfig.yml):dbPointInTimeRecoveryEnabled.${sls:stage}}
        SSESpecification:
          SSEEnabled: true
        StreamSpecification:
          StreamViewType: NEW_AND_OLD_IMAGES
        TableName: ${self:provider.environment.DYNAMODB_TABLE}
        TimeToLiveSpecification:
          AttributeName: ttl
          Enabled: true
        GlobalSecondaryIndexes:
          - IndexName: PSi1
            KeySchema:
              - AttributeName: SortKey
                KeyType: HASH
              - AttributeName: PartitionKey
                KeyType: RANGE
            Projection:
              ProjectionType: ALL
          - IndexName: PSi2
            KeySchema:
              - AttributeName: PSI2_Partition
                KeyType: HASH
              - AttributeName: PSI2_Sort
                KeyType: RANGE
            Projection:
              ProjectionType: ALL
          - IndexName: PSi3
            KeySchema:
              - AttributeName: PSI3_Partition
                KeyType: HASH
              - AttributeName: PSI3_Sort
                KeyType: RANGE
            Projection:
              ProjectionType: ALL
          - IndexName: PSi4
            KeySchema:
              - AttributeName: PSI4_Partition
                KeyType: HASH
              - AttributeName: PSI4_Sort
                KeyType: RANGE
            Projection:
              ProjectionType: ALL
    CrudSNSTopic:
      Type: 'AWS::SNS::Topic'
      Properties:
        DisplayName: ${self:custom.CRUD_TOPIC}
        KmsMasterKeyId: '${self:custom.MQ_MASTER_KEY}'
        TopicName: ${self:custom.CRUD_TOPIC}
    DynamoErrorSNSTopic:
      Type: 'AWS::SNS::Topic'
      Properties:
        DisplayName: ${self:custom.DYNAMO_ERROR_TOPIC}
        KmsMasterKeyId: '${self:custom.MQ_MASTER_KEY}'
        TopicName: ${self:custom.DYNAMO_ERROR_TOPIC}

  Outputs:
    EmployeeServiceCRUDTopic:
      Description: 'CRUD SNS Topic Arn'
      Export:
        Name: '${self:service}-${sls:stage}-CRUDTopicArn'
      Value:
        Ref: CrudSNSTopic

plugins:
  - serverless-localstack

I am getting the following error while trying to deploy: [ I am using docker compose and localstack as well]

Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for Lambda: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for S3: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for S3: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for S3: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Skipping template validation: Unsupported in Localstack
Using custom endpoint for CloudFormation: http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)
Using custom endpoint for CloudFormation: http://localhost:4566/
Overriding S3 templateUrl to http://localhost:4566/
Skipping reconfiguring of endpoints (already reconfigured)

✖ Stack Employee-service-dev failed to deploy (10s)
Environment: darwin, node 16.20.2, framework 3.27.0 (local) 3.38.0v (global), plugin 6.2.3, SDK 4.3.2
Credentials: Local, environment variables
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
TemplateURL must be a supported URL.

1 deprecation found: run 'serverless doctor' for more details 
metojava commented 9 months ago

I got same on windows 10 with localstack-pro latest tried to downgrade localstack-pro to 2.2.0, 2.3.0 and 3.0.0 versions and deployment worked but they are not useful for lambda at all with serverless projects /installed via npm/ serverless examples modified according to serverless-localstack plugin are not working

localstack tried to spin up docker containers for nodejs but failing , seems they create local container but got error : no such container ... for python it kind of working but throwing error that event['body'] was null ... or response not in json format . all this is working from awslocal , but to practice serverless and cdk are not

I increased memorySize property for function executions , gave lambda timeout config 300 but still no result was using different configs for Lambda Executor but still no result localstack couldn't do simplest thing localstack-demo which is on website is not working at all it can't even deploy on latest localstack ... I was not able to use it for any ways and also configs and how to start... them please I used docker-compose.yaml file that is standard given by localstack itself serverless-examples repo projects not working they are repeating existing serverless examples repo and there is no configuration or way how to run projects given

can someone from localstack team give us working examples of serverless projects for node and python languages ? and which version of what was used ?

thanks

steffyP commented 9 months ago

Hi @prasaddls,

thanks for reaching out!

can someone from localstack team give us working examples of serverless projects

Please check our Serverless Transcription App, where we use the serverless framework to deploy node-js functions. It contains a very detailed README, I hope you find it helpful!

serverless-examples repo projects not working

Sorry about your experience here! It's a forked repo that is not maintained. In general we are currently trying to setup more samples in the localstack-samples organization.

Regarding the original issue: Unfortunately, I wasn't able yet to reproduce the error you have reported with TemplateURL must be a supported URL. You are saying it's not working with localstack/localstack-pro:latest but it does work with 3.0.0 and lower versions, right?

I am using docker compose and localstack as well

I assume you mean, you tried starting localstack using the cli (localstack start) and using a docker-compose file, and you get the same error? Could you try to enable the DEBUG=1 flag, and check if there any errors logged in LocalStack (e.g. the container logs)?

jorgenfb commented 8 months ago

I say the same issue today (on linux).

The logs says:

 Overriding S3 templateUrl to http://localhost:4566/

Looking at the source: https://github.com/localstack/serverless-localstack/blob/master/src/index.js#L672-L673 it looks like its replacing the aws url without the trailing slash. The result on my machine is that the template url becomes:

http://localhost:4566//floorplan-converter-local-serverlessdeploymentbuck-cba42efa/serverless/floorplan-converter/local/1704380511014-2024-01-04T15:01:51.014Z/compiled-cloudformation-template.json

Note: the double slash after the port number.

This is the interesting parts of my log when running sls deploy --stage local:

lifecycle:command:invoke:hook: [79]   < aws:deploy:deploy:updateStack
Using custom endpoint for CloudFormation: http://localstack:4566/
Overriding S3 templateUrl to http://localstack:4566/
Skipping reconfiguring of endpoints (already reconfigured)
aws: [16] { name: 'CloudFormation',
  params:
   { credentials:
      EnvironmentCredentials { expired: false, expireTime: null, refreshCallbacks: [], accessKeyId: 'test', sessionToken: undefined, envPrefix: 'AWS' },
     endpoint: 'http://localstack:4566/',
     region: 'eu-west-1',
     isS3TransferAccelerationEnabled: false } } updateStack [ { StackName: 'floorplan-converter-local',
    Capabilities: [ 'CAPABILITY_IAM', 'CAPABILITY_NAMED_IAM' ],
    Parameters: [],
    Tags: [ { Key: 'STAGE', Value: 'local' } ],
    TemplateURL:
     'http://localstack:4566//floorplan-converter-local-serverlessdeploymentbuck-cba42efa/serverless/floorplan-converter/local/1704380511014-2024-01-04T15:01:51.014Z/compiled-cloudformation-template.json',
    NotificationARNs: [] } ]
aws: [16] { Error [ValidationError]: TemplateURL must be a supported URL.
jorgenfb commented 8 months ago

Digging further into this it looks like the issue might be related to the function:

  /**
   * If the given `endpointURL` points to `localhost`, then inject the given `hostname` into the URL
   * and return it. This helps fix IPv6 issues with node v18+ (see also getConnectHostname() above)
   */
  injectHostnameIntoLocalhostURL(endpointURL, hostname) {
    const url = new URL(endpointURL);
    if (hostname && url.hostname === 'localhost') {
      url.hostname = hostname;
    }
    return url.href;
  }

as the href property includes a trailing slash:

> new URL("http://localstack:4566")
URL {
  href: 'http://localstack:4566/',
  origin: 'http://localstack:4566',
  protocol: 'http:',
  username: '',
  password: '',
  host: 'localstack:4566',
  hostname: 'localstack',
  port: '4566',
  pathname: '/',
  search: '',
  searchParams: URLSearchParams {},
  hash: ''
}

Changing the return value to be return url.origin; seems to fix my issue.

steffyP commented 8 months ago

thanks so much for your input @jorgenfb! Indeed it seems like this behavior is caused by setting the env AWS_ENDPOINT_URL. Please let me know if you would like to issue a PR to fix this?

Not setting the AWS_ENDPOINT_URL will use the default http://localhost:4566, which might be a suitable workaround for you @prasaddls?

jorgenfb commented 8 months ago

I've added a PR to fix the issue :)

steffyP commented 8 months ago

Thanks so much @jorgenfb!

We merged the fix, but we still need to prepare the release, which will probably go out next week! /cc @prasaddls

steffyP commented 8 months ago

We just published serverless-localstack 1.1.3