search

compose-x / ecs_composex

Manage, Configure and Deploy your services and AWS services and applications from your docker-compose definitions
https://docs.compose-x.io
Mozilla Public License 2.0
165 stars 17 forks source link

UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality when running ecs-compose-x up command #645

Closed xender69 closed 1 year ago

xender69 commented 1 year ago

hi Guys,

JohnPreston suggested I use ecs-compose-x command to use with my docker compose command to deploy to aws ecs. So, I installed python3 and ecs-compoose-x.

I tried a simple command ecw-compose-x up docker-compose.yml and received the following warnings:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
usage: ecs-compose-x up [-h] -n NAME [--format {json,yaml,text}] [--region REGIONNAME] [--azs ZONES] [-b BUCKETNAME] [--role-arn ROLEARN]
                        [--disable-rollback] -f DOCKERCOMPOSEXFILE [-d OUTPUTDIRECTORY] [--ignore-ecr-findings]
ecs-compose-x up: error: the following arguments are required: -n/-p/--name, -f/--docker-compose-file

Please help resolve this issue as it looks like ecs-compose-x is the only way that I can get the aws deployment that I am trying to deploy.

version: '3.8'

# Define networks and map these to x-vpc Subnets, to use with the services

networks:
  application:
    x-vpc: AppSubnets
  storage:
    x-vpc: StorageSubnets

services:
  osticket:
    container_name: osticket-web
    image: osticket/osticket
    environment:
      MYSQL_HOST: db
      MYSQL_PASSWORD: secret
    depends_on:
      - db
    ports:
      - 80:80
    networks:
      - application
  db:
    container_name: osticket-db
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: osticket
      MYSQL_USER: osticket
      MYSQL_PASSWORD: secret
    networks:
      - storage

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: vpc-0603884f9eaf1ebb3
#    PublicSubnets: ** not used **
#      Tags:
#        - Name: my-public-subnets
#            - subnet-093223fe760e52016 #public subnet-1
#            - subnet-08120f88feb55e3f1 #public subnet-2
    AppSubnets:
      Tags:
        - Name: my-application-subnets
            - subnet-093223fe760e52016 #public subnet-1
            - subnet-08120f88feb55e3f1 #public subnet-2
    StorageSubnets:
      Tags:
        - Name: my-db-storage-subnets
            - subnet-0c68a298227d9c2e8 #private subnet-1
            - subnet-042cae15125ba9b1b #private subnet-2

Thank you everyone!!!!

JohnPreston commented 1 year ago

Hey man. Thanks for the follow up here. So here is a touch up of the compose file. As said, you must have PublicSubnets defined for Lookup. Does not matter whether they are public or not, you can use the same ones as the AppSubnets, it doesn't matter.

version: '3.8'

# Define networks and map these to x-vpc Subnets, to use with the services

networks:
  application:
    x-vpc: AppSubnets
  storage:
    x-vpc: StorageSubnets

services:
  osticket:
    container_name: osticket-web
    image: osticket/osticket
    environment:
      MYSQL_HOST: db
      MYSQL_PASSWORD: secret
    depends_on:
      - db
    ports:
      - 80:80
    networks:
      - application

  db:
    container_name: osticket-db
    image: mariadb
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: osticket
      MYSQL_USER: osticket
      MYSQL_PASSWORD: secret
    networks:
      - storage

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: vpc-0603884f9eaf1ebb3
    PublicSubnets: # ** not used **
      Tags:
        - Name: my-public-subnets
          # - subnet-093223fe760e52016 #public subnet-1
          # - subnet-08120f88feb55e3f1 #public subnet-2
    AppSubnets:
      Tags:
        - Name: my-application-subnets
        # - subnet-093223fe760e52016 #public subnet-1
        # - subnet-08120f88feb55e3f1 #public subnet-2
    StorageSubnets:
      Tags:
        - Name: my-db-storage-subnets
        # - subnet-0c68a298227d9c2e8 #private subnet-1
        # - subnet-042cae15125ba9b1b #private subnet-2

With the above compose file, you would run

ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yaml

The lookup with the above tags doesn't work for me given I don't have a VPC & Subnets with these tags, but if you do have these set, you should be good to go.

All you seemed to be missing in your command was the -p/-n that is the name of the "root" stack. Let me know how you get on

xender69 commented 1 year ago

Thank you for quick reply.

So I have uncommented the AppSubnets: and StorageSubnets: as these 2 subnets are valid in my VPC. So the docker-compose.yml file looks the same expect for the below changes:

        - Name: my-application-subnets
        - subnet-093223fe760e52016 #public subnet-1
        - subnet-08120f88feb55e3f1 #public subnet-2
    StorageSubnets:
      Tags:
        - Name: my-db-storage-subnets
        - subnet-0c68a298227d9c2e8 #private subnet-1
        - subnet-042cae15125ba9b1b #private subnet-2

I run the command that you gave and get the following result:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-01 12:18:58 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-01 12:18:58 [    INFO] Service families to process ['osticket', 'db']
2022-11-01 12:18:58 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-01 12:18:58 [    INFO] Processing x-vpc
2022-11-01 12:18:58 [   ERROR] vpc.vpc - Definition is not conform to schema.
Traceback (most recent call last):
  File "/System/Volumes/Data/Users/johnchang/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 183, in main
    root_stack = generate_full_template(settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 286, in generate_full_template
    add_x_resources(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 199, in add_x_resources
    x_stack = module.stack_class(
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 391, in __init__
    self.vpc_resource = Vpc(
                        ^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 115, in __init__
    super().__init__(name, definition, module, settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/environment_x_resources.py", line 30, in __init__
    super().__init__(name, definition, module, settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/__init__.py", line 75, in __init__
    self.validate_schema(name, definition, module.mod_key)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/__init__.py", line 194, in validate_schema
    jsonschema.validate(
  File "/Users/johnchang/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 1110, in validate
    raise error
jsonschema.exceptions.ValidationError: [{'Name': 'my-application-subnets'}, 'subnet-093223fe760e52016', 'subnet-08120f88feb55e3f1'] is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['Lookup']['properties']['AppSubnets']['properties']['Tags']:
    {'oneOf': [{'patternProperties': {'[\\x20-\\x7E]+': {'maxLength': 256,
                                                         'minLength': 0,
                                                         'type': 'string'}},
                'type': 'object'},
               {'items': {'additionalProperties': False,
                          'patternProperties': {'[\\S]+$': {'maxLength': 256,
                                                            'minLength': 0,
                                                            'type': 'string'}},
                          'type': 'object'},
                'type': 'array'}]}

On instance['Lookup']['AppSubnets']['Tags']:
    [{'Name': 'my-application-subnets'},
     'subnet-093223fe760e52016',
     'subnet-08120f88feb55e3f1']

The very first line complains with needing to install ecs-coposex[ercscan]:

/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(

Thank you for all your help, at least we are getting somewhere now :-)

JohnPreston commented 1 year ago

Yes, so you can't just give the subnet IDs :/ It was possible in the past, but too limiting, so only via Lookup you can find these.

Is there a common tag set on these topics? If not, maybe create a tag on these two such as Key=Discovery,Value=compose-x-storage

and then to "find" them, use

StorageSubnets:
  Tags:
    - Discovery: compose-x-storage

Does that make sense?

The error you are getting now is JSON schema telling you that the input compose file does not comply with the expected input, that's all.

xender69 commented 1 year ago

I believe you are asking me to change the AppSubnets and StorageSubnets from specifying the subnets explicitly to a tag based "search" method so, the new config would/could look something like this:

    AppSubnets:
      Tags:
        - Discovery1: compose-x-public
#        - Name: my-application-subnets
#        - subnet-093223fe760e52016 #public subnet-1
#        - subnet-08120f88feb55e3f1 #public subnet-2
    StorageSubnets:
      Tags:
        - Discovery2: compose-x-private
#        - Name: my-db-storage-subnets
#        - subnet-0c68a298227d9c2e8 #private subnet-1
#        - subnet-042cae15125ba9b1b #private subnet-2

You are saying that now, ecs-compose-x should be able to find Tags labeled Discovery1 and Discovery2 and use the associated values. Is this correct?

JohnPreston commented 1 year ago

Correct. You can use any key value that you want for these tags and perform the discovery. It uses the resource tagging API to discover the subnets.

I for example use on all my VPCs tags such as vpc::usage as the Key and application / storage / public.

Replied from phone and mis clicked on close :/

xender69 commented 1 year ago

Thank you. So I changed to look like below:

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: vpc-0603884f9eaf1ebb3
    PublicSubnets: # ** not used **
      Tags:
        - Name: my-public-subnets
          # - subnet-093223fe760e52016 #public subnet-1
          # - subnet-08120f88feb55e3f1 #public subnet-2
    AppSubnets:
      Tags:
        - PubDir1: compose-x-public1
        - PubDir2: compose-x-public2
    StorageSubnets:
      Tags:
        - PrivDir1: compose-x-private1
        - PrivDir2: compose-x-private2

The new output looks different:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-01 15:39:59 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-01 15:40:00 [    INFO] Service families to process ['osticket', 'db']
2022-11-01 15:40:00 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-01 15:40:00 [    INFO] Processing x-vpc
Traceback (most recent call last):
  File "/System/Volumes/Data/Users/johnchang/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 183, in main
    root_stack = generate_full_template(settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 286, in generate_full_template
    add_x_resources(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 199, in add_x_resources
    x_stack = module.stack_class(
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 395, in __init__
    self.vpc_resource.lookup_vpc(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 190, in lookup_vpc
    vpc_settings = lookup_x_vpc_settings(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_aws.py", line 104, in lookup_x_vpc_settings
    vpc_arn = find_aws_resource_arn_from_tags_api(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/aws.py", line 234, in find_aws_resource_arn_from_tags_api
    return handle_search_results(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/aws.py", line 173, in handle_search_results
    raise LookupError(
LookupError: ('No resources were found with the provided tags and information', None, 'ec2:vpc')

The Last line, seems to indicate that I could not find any tags related to what I had configured above.

xender69 commented 1 year ago

so I have the below change just to see what happens:

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: vpc-0603884f9eaf1ebb3
#    PublicSubnets: # ** not used **
#      Tags:
#        - Name: my-public-subnets
          # - subnet-093223fe760e52016 #public subnet-1
          # - subnet-08120f88feb55e3f1 #public subnet-2
    AppSubnets:
      Tags:
        -Name: docker-ecs-test-subnet-public1-us-east-1a
#        - PubDir1: compose-x-public1
#        - PubDir2: compose-x-public2
    StorageSubnets:
      Tags:
        -Name: docker-ecs-project-subnet-private1-us-east-1a
#        - PrivDir1: compose-x-private1
#        - PrivDir2: compose-x-private2

Here is the outcome:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-01 17:06:11 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-01 17:06:12 [    INFO] Service families to process ['osticket', 'db']
2022-11-01 17:06:12 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-01 17:06:12 [    INFO] Processing x-vpc
2022-11-01 17:06:12 [   ERROR] vpc.vpc - Definition is not conform to schema.
Traceback (most recent call last):
  File "/System/Volumes/Data/Users/johnchang/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 183, in main
    root_stack = generate_full_template(settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 286, in generate_full_template
    add_x_resources(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 199, in add_x_resources
    x_stack = module.stack_class(
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 391, in __init__
    self.vpc_resource = Vpc(
                        ^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 115, in __init__
    super().__init__(name, definition, module, settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/environment_x_resources.py", line 30, in __init__
    super().__init__(name, definition, module, settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/__init__.py", line 75, in __init__
    self.validate_schema(name, definition, module.mod_key)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/x_resources/__init__.py", line 194, in validate_schema
    jsonschema.validate(
  File "/Users/johnchang/venv/lib/python3.11/site-packages/jsonschema/validators.py", line 1110, in validate
    raise error
jsonschema.exceptions.ValidationError: 'PublicSubnets' is a required property

Failed validating 'required' in schema['properties']['Lookup']:
    {'patternProperties': {'\\b(?!RoleArn\\b)\\w+': {'$ref': 'x-resources.common.spec.json#/definitions/Lookup'}},
     'properties': {'AppSubnets': {'$ref': 'x-resources.common.spec.json#/definitions/Lookup'},
                    'PublicSubnets': {'$ref': 'x-resources.common.spec.json#/definitions/Lookup'},
                    'RoleArn': {'type': 'string'},
                    'StorageSubnets': {'$ref': 'x-resources.common.spec.json#/definitions/Lookup'},
                    'VpcId': {'$ref': 'x-resources.common.spec.json#/definitions/Lookup'}},
     'required': ['VpcId', 'AppSubnets', 'PublicSubnets', 'StorageSubnets'],
     'type': 'object'}

On instance['Lookup']:
    {'AppSubnets': {'Tags': {'-Name': 'docker-ecs-test-subnet-public1-us-east-1a'}},
     'StorageSubnets': {'Tags': {'-Name': 'docker-ecs-project-subnet-private1-us-east-1a'}},
     'VpcId': {'Tags': [{'Name': 'vpc-0603884f9eaf1ebb3'}]}}

Seems using the -Name Tag it is finding the values, am I correct? Also it looks like I DO need to define the PublicSubnet based on this error:

jsonschema.exceptions.ValidationError: 'PublicSubnets' is a required property

JohnPreston commented 1 year ago

Yes. It didn't find a VPC with a tag Name of value vpc-0603884f9eaf1ebb3 So either set that or change the value for Name in Lookup

You are almost there!

xender69 commented 1 year ago

Latest update:

I changed the x-vpc to following as the command wanted the PublicSubnets Defined and as you suggested, I went with the Name Tag in the Vpcidz.

jsonschema.exceptions.ValidationError: 'PublicSubnets' is a required property

So I added it in the x-vpc config:

x-vpc:
  Lookup:
    VpcId:
      Tags:
        - Name: docker-ecs-test-vpc
    PublicSubnets: # ** not used **
      Tags:
        -Name: docker-ecs-test-subnet-public1-us-east-1a
    AppSubnets:
      Tags:
        -Name: docker-ecs-test-subnet-public1-us-east-1a
    StorageSubnets:
      Tags:
        -Name: docker-ecs-project-subnet-private1-us-east-1a

Seems I am still missing something:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-02 09:31:15 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-02 09:32:10 [    INFO] Service families to process ['osticket', 'db']
2022-11-02 09:32:10 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-02 09:32:10 [    INFO] Processing x-vpc
2022-11-02 09:32:11 [   ERROR] An error occurred (ValidationException) when calling the GetResources operation: 1 validation error detected: Value '[d, o, c, k, e, r, -, e, c, s, -, t, e, s, t, -, s, u, b, n, e, t, -, p, u, b, l, i, c, 1, -, u, s, -, e, a, s, t, -, 1, a]' at 'tagFilters.1.member.values' failed to satisfy constraint: Member must have length less than or equal to 20
2022-11-02 09:32:11 [   ERROR] Not processing this resource. Skipping
Traceback (most recent call last):
  File "/System/Volumes/Data/Users/johnchang/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 183, in main
    root_stack = generate_full_template(settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 286, in generate_full_template
    add_x_resources(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/ecs_composex.py", line 199, in add_x_resources
    x_stack = module.stack_class(
              ^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 395, in __init__
    self.vpc_resource.lookup_vpc(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_stack.py", line 190, in lookup_vpc
    vpc_settings = lookup_x_vpc_settings(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc/vpc_aws.py", line 119, in lookup_x_vpc_settings
    subnet_arns = find_aws_resource_arn_from_tags_api(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/aws.py", line 234, in find_aws_resource_arn_from_tags_api
    return handle_search_results(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/aws.py", line 173, in handle_search_results
    raise LookupError(
LookupError: ('No resources were found with the provided tags and information', None, 'ec2:subnet')
xender69 commented 1 year ago

update:

Don't know if you saw, but I had to add a space between - and Name, which I did, and here is the outcome of the command:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-02 10:23:47 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-02 10:23:47 [    INFO] Service families to process ['osticket', 'db']
2022-11-02 10:23:47 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-02 10:23:47 [    INFO] Processing x-vpc
2022-11-02 10:23:49 [    INFO] x-vpc - Found VPC - vpc-0603884f9eaf1ebb3
2022-11-02 10:23:49 [    INFO] Adding dependency between db and osticket
2022-11-02 10:23:51 [    INFO] x-vpc - Found VPC - vpc-0603884f9eaf1ebb3
2022-11-02 10:23:51 [    INFO] networks.application - mapped x-vpc.AppSubnets to osticket
2022-11-02 10:23:51 [    INFO] networks.storage - mapped x-vpc.StorageSubnets to db
2022-11-02 10:23:51 [    INFO] osticket - Hostname set to osticket
2022-11-02 10:23:51 [    INFO] osticket - Ephemeral storage: 0
2022-11-02 10:23:51 [    INFO] osticket - LaunchType set to FARGATE_PROVIDERS
2022-11-02 10:23:51 [    INFO] osticket - TaskDefinition containers: ['osticket']
2022-11-02 10:23:51 [    INFO] db - Hostname set to db
2022-11-02 10:23:51 [    INFO] db - Ephemeral storage: 0
2022-11-02 10:23:51 [    INFO] db - LaunchType set to FARGATE_PROVIDERS
2022-11-02 10:23:51 [    INFO] db - TaskDefinition containers: ['db']
Traceback (most recent call last):
  File "/System/Volumes/Data/Users/johnchang/venv/bin/ecs-compose-x", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/cli.py", line 184, in main
    process_stacks(root_stack, settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/stacks/__init__.py", line 334, in process_stacks
    process_stacks(resource, settings, is_root=False)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/stacks/__init__.py", line 342, in process_stacks
    root_stack.render(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/stacks/__init__.py", line 242, in render
    template_file.upload(settings)
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/files.py", line 166, in upload
    self.url = upload_file(
               ^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/common/files.py", line 58, in upload_file
    client.put_object(
  File "/Users/johnchang/venv/lib/python3.11/site-packages/botocore/client.py", line 507, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/johnchang/venv/lib/python3.11/site-packages/botocore/client.py", line 943, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

Thank you
JohnPreston commented 1 year ago

Ah! That's the problem when you run the command 1000 times a day, you don't even think of these things at all anymore (talking of myself ;)) As described in the requirements you should run the command ecs-compose-x init to ensure you have everything in ECS ready to go from a settings point of view.

It also creates a bucket that is used as the default bucket to upload the templates to. But, you can override it with -b / --bucket-name and use an existing template of yours.

Generally speaking, you have the main help

ecs-compose-x -h and each subcommand (i.e. up, render, create etc.) has their own submenu

for example

ecs-compose-x up -h
/home/john/.cache/pypoetry/virtualenvs/ecs-composex-6XqxN9Zj-py3.8/lib/python3.8/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
usage: ecs-compose-x up [-h] [--loglevel LOGLEVEL] -n NAME [--format {json,yaml,text}] [--region REGIONNAME] [--azs ZONES] [-b BUCKETNAME] [--role-arn ROLEARN] [--disable-rollback] -f DOCKERCOMPOSEXFILE
                        [-d OUTPUTDIRECTORY] [--ignore-ecr-findings]

optional arguments:
  -h, --help            show this help message and exit
  -n NAME, -p NAME, --name NAME
                        Name of your stack / docker project
  --format {json,yaml,text}
                        Defines the format you want to use.
  --region REGIONNAME   Specify the region you want to build fordefault use default region from config or environment vars
  --azs ZONES           List AZs you want to deploy to specifically within the region
  -b BUCKETNAME, --bucket-name BUCKETNAME
                        Bucket name to upload the templates to
  --role-arn ROLEARN    Allow you to run API calls using a specific IAM role, within same or for cross-account
  --disable-rollback    On create/plan, disable stack automatic rollback.
  -f DOCKERCOMPOSEXFILE, --docker-compose-file DOCKERCOMPOSEXFILE
                        Path to the Docker compose file
  -d OUTPUTDIRECTORY, --output-dir OUTPUTDIRECTORY
                        Output directory to write all the templates to.
  --ignore-ecr-findings
                        For services with x-ecr defined, ignores errors if any found

Apologies for the very non-smooth experience so far. I'd really like to see what could have been documented better (and where) to get you up and running.

JohnPreston commented 1 year ago

PS: looking at your logs, it is also going to create you a new ECS Cluster with some additional security features (default). If you already have an ECS Cluster you want to use (even it is default), you might want to do

x-cluster:
  Lookup:
    ClusterName: my-existing-ecs-cluster

It might not look like much, but it does a lot of discovery to make sure that you have Fargate enabled, if you have EC2 instances, and so on. Again one thing that without API calls discovery couldn't be done.

xender69 commented 1 year ago

Thank you John,

Running the init command definitely makes a BIG difference :-)... So, running the init command creates the new S3 bucket, which I can via the aws console. Then running the up commands, shows the following output:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ ecs-compose-x up -d templates -p my-stack-name -f docker-compose.yml
/Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/compose/compose_services/service_image/docker_opts.py:26: UserWarning: You must install ecs-composex[ecrscan] extra to use this functionality
  warnings.warn(
2022-11-02 10:49:15 [    INFO] Validating against input schema /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/specs/compose-spec.json
2022-11-02 10:49:15 [    INFO] Service families to process ['osticket', 'db']
2022-11-02 10:49:15 [    INFO] No cluster information provided. Creating a new one
Loaded x-vpc vpc vpc /Users/johnchang/venv/lib/python3.11/site-packages/ecs_composex/vpc
2022-11-02 10:49:15 [    INFO] Processing x-vpc
2022-11-02 10:49:17 [    INFO] x-vpc - Found VPC - vpc-0603884f9eaf1ebb3
2022-11-02 10:49:17 [    INFO] Adding dependency between db and osticket
2022-11-02 10:49:19 [    INFO] x-vpc - Found VPC - vpc-0603884f9eaf1ebb3
2022-11-02 10:49:19 [    INFO] networks.application - mapped x-vpc.AppSubnets to osticket
2022-11-02 10:49:19 [    INFO] networks.storage - mapped x-vpc.StorageSubnets to db
2022-11-02 10:49:19 [    INFO] osticket - Hostname set to osticket
2022-11-02 10:49:19 [    INFO] osticket - Ephemeral storage: 0
2022-11-02 10:49:19 [    INFO] osticket - LaunchType set to FARGATE_PROVIDERS
2022-11-02 10:49:19 [    INFO] osticket - TaskDefinition containers: ['osticket']
2022-11-02 10:49:19 [    INFO] db - Hostname set to db
2022-11-02 10:49:19 [    INFO] db - Ephemeral storage: 0
2022-11-02 10:49:19 [    INFO] db - LaunchType set to FARGATE_PROVIDERS
2022-11-02 10:49:19 [    INFO] db - TaskDefinition containers: ['db']
2022-11-02 10:49:20 [    INFO] iam.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/iam.json
2022-11-02 10:49:20 [    INFO] osticket.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/osticket.json
2022-11-02 10:49:21 [    INFO] osticket.params.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/osticket.params.json
2022-11-02 10:49:21 [    INFO] osticket.config.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/osticket.config.json
2022-11-02 10:49:22 [    INFO] db.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/db.json
2022-11-02 10:49:22 [    INFO] db.params.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/db.params.json
2022-11-02 10:49:23 [    INFO] db.config.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/db.config.json
2022-11-02 10:49:23 [    INFO] my-stack-name.json uploaded successfully to https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1449/2e0911/my-stack-name.json
2022-11-02 10:49:24 [    INFO] Stack my-stack-name successfully deployed.
2022-11-02 10:49:24 [    INFO] arn:aws:cloudformation:us-east-1:638606483737:stack/my-stack-name/8ad1d6c0-5abd-11ed-8342-0efef2cb81fd

Looking great but when I run docker compose ps, it shows the following output:

 ~/Desktop/docker-projects/osTicket/ecs-osticket/ docker compose ps
ValidationError: Stack with id ecs-osticket does not exist
        status code: 400, request id: 089dd18f-2471-463c-9b72-7b7382560532

When I look at the aws console, I do see my-stack-name ecs cluster, I'm not exactly sure why the docker compose ps command is referencing the old ecs cluster. Also, as you can see, the ps command is not showing any of the containers running and in aws console, I only see db task and not osticket as well.

Great progress, thank you...

xender69 commented 1 year ago

Here is the my-stack-name converted to yml for easier reading:

---
AWSTemplateFormatVersion: '2010-09-09'
Description: Root template generated via ECS ComposeX
Mappings:
  ComposeXDefaults:
    ECS:
      PlatformVersion: 1.4.0
  Network:
    AppSubnets:
      Azs:
      - us-east-1a
      Ids:
      - subnet-093223fe760e52016
    PublicSubnets:
      Azs:
      - us-east-1a
      Ids:
      - subnet-093223fe760e52016
    StorageSubnets:
      Azs:
      - us-east-1a
      Ids:
      - subnet-0c68a298227d9c2e8
    VpcId:
      VpcId: vpc-0603884f9eaf1ebb3
Metadata:
  Properties:
    GeneratedOn: '2022-11-02T15:11:20.313179'
    Version: 0.22.6
  Type: ComposeX
Resources:
  EcsCluster:
    Metadata:
      Properties:
        Version: 0.22.6
        ecs_composex::module: ecs_composex.ecs
      Type: ComposeX
    Properties:
      CapacityProviders:
      - FARGATE
      - FARGATE_SPOT
      ClusterName:
        Ref: AWS::StackName
      Configuration:
        ExecuteCommandConfiguration:
          LogConfiguration:
            CloudWatchLogGroupName:
              Ref: EcsExecLogGroup
          Logging: OVERRIDE
      DefaultCapacityProviderStrategy:
      - Base: 1
        CapacityProvider: FARGATE_SPOT
        Weight: 2
      - CapacityProvider: FARGATE
        Weight: 1
    Type: AWS::ECS::Cluster
  EcsExecLogGroup:
    Properties:
      LogGroupName:
        Fn::Sub: ecs/execute-logs/${AWS::StackName}
      RetentionInDays: 120
    Type: AWS::Logs::LogGroup
  db:
    DependsOn:
    - iam
    Properties:
      Parameters:
        AppSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - AppSubnets
            - Ids
        EcsCluster:
          Ref: EcsCluster
        EcsLaunchType: FARGATE_PROVIDERS
        FargateCpuRamConfiguration: 256!512
        FargatePlatformVersion:
          Fn::FindInMap:
          - ComposeXDefaults
          - ECS
          - PlatformVersion
        MicroServiceName: db
        MicroserviceHostname: db
        MicroservicesCount: 1
        PublicSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - PublicSubnets
            - Ids
        RootStackName:
          Ref: AWS::StackName
        ServiceLogGroupRetentionPeriod: 30
        StorageSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - StorageSubnets
            - Ids
        VpcId:
          Fn::FindInMap:
          - Network
          - VpcId
          - VpcId
        dbEcsExecutionRoleIamRoleArn:
          Fn::GetAtt:
          - iam
          - Outputs.dbEcsExecutionRoleIamRoleArn
        dbEcsExecutionRoleIamRoleName:
          Fn::GetAtt:
          - iam
          - Outputs.dbEcsExecutionRoleIamRoleName
        dbEcsTaskRoleIamRoleArn:
          Fn::GetAtt:
          - iam
          - Outputs.dbEcsTaskRoleIamRoleArn
        dbEcsTaskRoleIamRoleName:
          Fn::GetAtt:
          - iam
          - Outputs.dbEcsTaskRoleIamRoleName
        dbImageUrl: mariadb
      TemplateURL: https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1511/be7cc6/db.json
    Type: AWS::CloudFormation::Stack
  iam:
    DependsOn: []
    Properties:
      Parameters:
        EcsCluster:
          Ref: EcsCluster
        RootStackName:
          Ref: AWS::StackName
      TemplateURL: https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1511/be7cc6/iam.json
    Type: AWS::CloudFormation::Stack
  osticket:
    DependsOn:
    - iam
    - db
    Properties:
      Parameters:
        AppSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - AppSubnets
            - Ids
        EcsCluster:
          Ref: EcsCluster
        EcsLaunchType: FARGATE_PROVIDERS
        FargateCpuRamConfiguration: 256!512
        FargatePlatformVersion:
          Fn::FindInMap:
          - ComposeXDefaults
          - ECS
          - PlatformVersion
        MicroServiceName: osticket
        MicroserviceHostname: osticket
        MicroservicesCount: 1
        PublicSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - PublicSubnets
            - Ids
        RootStackName:
          Ref: AWS::StackName
        ServiceLogGroupRetentionPeriod: 30
        StorageSubnets:
          Fn::Join:
          - ","
          - Fn::FindInMap:
            - Network
            - StorageSubnets
            - Ids
        VpcId:
          Fn::FindInMap:
          - Network
          - VpcId
          - VpcId
        osticketEcsExecutionRoleIamRoleArn:
          Fn::GetAtt:
          - iam
          - Outputs.osticketEcsExecutionRoleIamRoleArn
        osticketEcsExecutionRoleIamRoleName:
          Fn::GetAtt:
          - iam
          - Outputs.osticketEcsExecutionRoleIamRoleName
        osticketEcsTaskRoleIamRoleArn:
          Fn::GetAtt:
          - iam
          - Outputs.osticketEcsTaskRoleIamRoleArn
        osticketEcsTaskRoleIamRoleName:
          Fn::GetAtt:
          - iam
          - Outputs.osticketEcsTaskRoleIamRoleName
        osticketImageUrl: osticket/osticket
      TemplateURL: https://s3.amazonaws.com/ecs-composex-638606483737-us-east-1/2022/11/02/1511/be7cc6/osticket.json
    Type: AWS::CloudFormation::Stack

Like I said earlier, I only see db task and no osticket task in the cluster. I don't see a LoadBalancer created. I'm not sure what else is missing. I guess those are the 2 biggest to get this working.

Thank you for all the follow up support.

JohnPreston commented 1 year ago

If that's easier, feel free to join Slack to go over it "live".

So, you should have seen both nested stacks, one for your osticket and one for db service. As for a load-balancer in front of osticket, we need to use the x-elbv2 module.

Depending on how you want to handle SSL certificates, if at all, you will also need to look at either create a new one or using existing ones via x-acm

But, with just plain http, the following should do


x-elbv2:
  public-alb:
    Properties:
      Type: application
    MacroParameters:
      Ingress:
        ExtSources:
          - IPv4: "0.0.0.0/0"
            Description: ANY
    Listeners:
      - Port: 80 # ALB HTTP Listener, sends all traffic to osticket
        Protocol: HTTP
        Targets:
          - name: osticket:osticket
            access: /
    Services: # Target Group, osticket container, on port 80, and checking on path `/`
      - name: osticket:osticket
        port: 80
        protocol: HTTP
        healthcheck: HTTP:80:/

There are multiple examples in the labs, for example, here.

The source code / docker compose files etc. for all these is available on GitHub too

JohnPreston commented 1 year ago

Hi @xender69 Haven't seen you on Slack in a while. Have you been able to deploy your app successfully since? Let me know on slack or in a comment below. Feel free to close the issue if your problem has since resolved.

xender69 commented 1 year ago

hi John,

I needed to put this project on hiatus as we are working with AWS SA and their approach. So, once we are done with this, I will probably come back and finish this. For now, I will go ahead and close the issue.

Once again, I want to thank you from bottom of my heart. Your patience and willingness to help have been greatly appreciated.

When I come back to this, I hope it will be ok if I can contact you again either here or through slack.