search

ObrienlabsDev / eventstream

eventstream app
Apache License 2.0
0 stars 0 forks source link

eventstream Requirements/Architecture/Design/Implementation/Pipelines #1

Open obriensystems opened 1 year ago

obriensystems commented 1 year ago

Business Requirements

Architecture

Functionality

F1: Public REST API

F2:

REST API

Work Items

WI1: Onboarding

WI2: Landing Zone

WI3: IaC

WI4: API

WI5: Persistence

WI6: Compute

WI7: Reporting

WI8: Security

Older Implementations

obriensystems commented 7 months ago

Design Issues

DI 1: Data Storage

DI 2: Application Hosting

DI 3: API Endpoints

obriensystems commented 7 months ago

Artifacts

Identity

Security

Logging

obriensystems commented 7 months ago

Research

Serverless

API Gateway

Screenshot 2023-11-18 at 14 40 32 Screenshot 2023-11-18 at 14 42 12 Screenshot 2023-11-18 at 14 42 34

https://ilon4v81k3.execute-api.ca-central-1.amazonaws.com/dev

Add token

https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html

Screenshot 2023-11-18 at 14 46 55 Screenshot 2023-11-18 at 14 48 29 Screenshot 2023-11-18 at 14 50 42

test: allow

Screenshot 2023-11-18 at 14 52 04 Screenshot 2023-11-18 at 14 52 24

Tokens

michaelobrien@mbp7 pbmm-on-gcp-onboarding % curl -X GET -H "Authorization: Bearer ${TOKEN}" https://ilon4v81k3.execute-api.ca-central-1.amazonaws.com/dev
{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Bearer"}%                                                                                        michaelobrien@mbp7 pbmm-on-gcp-onboarding % export TOKEN=

https://docs.aws.amazon.com/apigateway/latest/developerguide/permissions.html

Checking https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-as-simple-proxy-for-lambda.html

Lambda

Aurora

Fargate

obriensystems commented 7 months ago

Fargate

CF template

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The template used to create an ECS Cluster from the ECS Console.",
  "Parameters": {
    "ECSClusterName": {
      "Type": "String",
      "Description": "Specifies the ECS Cluster Name with which the resources would be associated",
      "Default": "eventstream"
    },
    "SecurityGroupIds": {
      "Type": "CommaDelimitedList",
      "Description": "Optional - Specifies the Comma separated list of the Security Group Id of an existing Security Group.",
      "Default": ""
    },
    "VpcId": {
      "Type": "String",
      "Description": "Optional - Specifies the ID of an existing VPC in which to launch your container instances. If you specify a VPC ID, you must specify a list of existing subnets in that VPC. If you do not specify a VPC ID, a new VPC is created with at least 1 subnet.",
      "Default": "",
      "AllowedPattern": "^(?:vpc-[0-9a-f]{8,17}|)$",
      "ConstraintDescription": "VPC Id must begin with 'vpc-' and have a valid uuid"
    },
    "SubnetIds": {
      "Type": "CommaDelimitedList",
      "Description": "Optional - Specifies the Comma separated list of existing VPC Subnet Ids where ECS instances will run",
      "Default": ""
    }
  },
  "Resources": {
    "ECSCluster": {
      "Type": "AWS::ECS::Cluster",
      "Properties": {
        "ClusterName": {
          "Ref": "ECSClusterName"
        },
        "CapacityProviders": [
          "FARGATE",
          "FARGATE_SPOT"
        ],
        "ClusterSettings": [
          {
            "Name": "containerInsights",
            "Value": "disabled"
          }
        ],
        "Configuration": {
          "ExecuteCommandConfiguration": {
            "Logging": "DEFAULT"
          }
        },
        "ServiceConnectDefaults": {
          "Namespace": "eventstream"
        },
        "Tags": [
          {
            "Key": "environment",
            "Value": "dev"
          }
        ]
      }
    }
  },
  "Outputs": {
    "ECSCluster": {
      "Description": "The created cluster.",
      "Value": {
        "Ref": "ECSCluster"
      }
    }

  }
}

2023-11-18 15:27:52 UTC-0500 | Infra-ECS-Cluster-eventstream-48395779 | CREATE_COMPLETE | -
-- | -- | -- | --
2023-11-18 15:27:51 UTC-0500 | ECSCluster | CREATE_COMPLETE | -
Screenshot 2023-11-18 at 15 28 09

Deploy a task from obrienlabs/magellan-nbi:0.0.3-ia64

{
    "taskDefinitionArn": "arn:aws:ecs:us-east-1:453279094200:task-definition/magellan:1",
    "containerDefinitions": [
        {
            "name": "magellan",
            "image": "obrienlabs/magellan-nbi:0.0.3-ia64",
            "cpu": 0,
            "portMappings": [
                {
                    "name": "magellan-8080-tcp",
                    "containerPort": 8080,
                    "hostPort": 8080,
                    "protocol": "tcp",
                    "appProtocol": "http"
                }
            ],
            "essential": true,
            "environment": [],
            "environmentFiles": [],
            "mountPoints": [],
            "volumesFrom": [],
            "ulimits": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/magellan",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "ecs"
                },
                "secretOptions": []
            }
        }
    ],
    "family": "magellan",
    "taskRoleArn": "arn:aws:iam::453279094200:role/ecsTaskExecutionRole",
    "executionRoleArn": "arn:aws:iam::453279094200:role/ecsTaskExecutionRole",
    "networkMode": "awsvpc",
    "revision": 1,
    "volumes": [],
    "status": "ACTIVE",
    "requiresAttributes": [
        {
            "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
        },
        {
            "name": "ecs.capability.execution-role-awslogs"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
        },
        {
            "name": "com.amazonaws.ecs.capability.task-iam-role"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
        },
        {
            "name": "ecs.capability.task-eni"
        },
        {
            "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
        }
    ],
    "placementConstraints": [],
    "compatibilities": [
        "EC2",
        "FARGATE"
    ],
    "requiresCompatibilities": [
        "FARGATE"
    ],
    "cpu": "1024",
    "memory": "3072",
    "runtimePlatform": {
        "cpuArchitecture": "X86_64",
        "operatingSystemFamily": "LINUX"
    },
    "registeredAt": "2023-11-18T21:29:11.304Z",
    "registeredBy": "arn:aws:iam::453279094200:root",
    "tags": []
}
Screenshot 2023-11-18 at 16 30 09

create a service

Screenshot 2023-11-18 at 16 31 44 
{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The template used to create an ECS Service from the ECS Console.",
  "Parameters": {
    "ECSClusterName": {
      "Type": "String",
      "Default": "eventstream"
    },
    "ECSServiceName": {
      "Type": "String",
      "Default": "magellan"
    },
    "SecurityGroupIDs": {
      "Type": "CommaDelimitedList",
      "Default": "sg-20273e45"
    },
    "SubnetIDs": {
      "Type": "CommaDelimitedList",
      "Default": "subnet-ece37889,subnet-2fcc7b58,subnet-5142b37a,subnet-957ae6af,subnet-2564e129,subnet-af9f4ef6"
    },
    "VpcID": {
      "Type": "String",
      "Default": "vpc-97c256f2"
    },
    "LoadBalancerName": {
      "Type": "String",
      "Default": ""
    }
  },
  "Resources": {
    "ECSService": {
      "Type": "AWS::ECS::Service",
      "Properties": {
        "Cluster": "eventstream",
        "CapacityProviderStrategy": [
          {
            "CapacityProvider": "FARGATE",
            "Base": 0,
            "Weight": 1
          }
        ],
        "TaskDefinition": "arn:aws:ecs:us-east-1:453279094200:task-definition/magellan:1",
        "ServiceName": "magellan",
        "SchedulingStrategy": "REPLICA",
        "DesiredCount": 1,
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "AssignPublicIp": "ENABLED",
            "SecurityGroups": {
              "Ref": "SecurityGroupIDs"
            },
            "Subnets": {
              "Ref": "SubnetIDs"
            }
          }
        },
        "PlatformVersion": "LATEST",
        "DeploymentConfiguration": {
          "MaximumPercent": 200,
          "MinimumHealthyPercent": 100,
          "DeploymentCircuitBreaker": {
            "Enable": true,
            "Rollback": true
          }
        },
        "DeploymentController": {
          "Type": "ECS"
        },
        "ServiceConnectConfiguration": {
          "Enabled": false
        },
        "Tags": [],
        "EnableECSManagedTags": true
      }
    }
  },
  "Outputs": {
    "ClusterName": {
      "Description": "The cluster used to create the service.",
      "Value": {
        "Ref": "ECSClusterName"
      }
    },
    "ECSService": {
      "Description": "The created service.",
      "Value": {
        "Ref": "ECSService"
      }
    }
  }
}
Screenshot 2023-11-18 at 16 34 21 Screenshot 2023-11-18 at 16 34 47 Screenshot 2023-11-18 at 16 36 14 

November 18, 2023 at 16:32 (UTC-5:00) | 2023-11-18 21:32:15.561 INFO 1 --- [ main] global.packet.magellan.NbiApplication : Started NbiApplication in 13.9 seconds (JVM running for 15.596) | 77c01e039df54e71b9241d1b0c097f66 | magellan
-- | -- | -- | --

November 18, 2023 at 16:32 (UTC-5:00)   2023-11-18 21:32:15.561 INFO 1 --- [ main] global.packet.magellan.NbiApplication : Started NbiApplication in 13.9 seconds (JVM running for 15.596)  [77c01e039df54e71b9241d1b0c097f66](https://us-east-1.console.aws.amazon.com/ecs/v2/clusters/eventstream/services/magellan/tasks/77c01e039df54e71b9241d1b0c097f66?region=us-east-1)  magellan

ECS service is up - now expose externally via ALB or API Gateway https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-private-integration.html