search

pagarme / superbowleto

:football: A microservice to issue, register and manage boletos
MIT License
5 stars 0 forks source link

Novo mecanismo para rodar migrations antes do deploy #353

Closed lucianopf closed 1 year ago

lucianopf commented 3 years ago

Descrição

Esse PR tem como objetivo segregar o momento em que rodamos as migrations do projeto para evitar que haja falha no meio de uma migration longa.

Antes as migrations eram rodadas quando o container do server subia e isso podia causar 2 problemas:

1 - mais de um container do server tentar rodar a msm migration no mesmo momento (se causar lock pode atrasar e até dar problema) 2 - uma migration longa ser cancelada pois o servidor não ficou saudável a tempo e interromper a migration no meio.

Agora mudamos um pouco a ordem de execução para rodar as migrations antes de atualizar as tasks do serviço.

Nova ordem de deploy:

  1. Testes + Sonar
  2. Build imagem
  3. Approve manual
  4. Upload da imagem pro ECR
  5. Criação da task definition dessa nova imagem
  6. Rodar uma task do serviço com o parâmetro que força rodar migrations (e esperar concluir a execução)
  7. Atualização dos services com a nova task definition

Obs: Tivemos que mudar a forma como a gnt fazia deploy pra poder segregar as etapas de criação de task definition e update do service.

Screen Shot 2021-01-06 at 15 10 16 Screen Shot 2021-01-06 at 15 09 56

Logs de exemplo da execução do run_migrations

added 1187 packages in 32.363s
Started task
Checking if task is stoped -> PROVISIONING
Checking if task is stoped -> PROVISIONING
Checking if task is stoped -> PROVISIONING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> PENDING
Checking if task is stoped -> RUNNING
Checking if task is stoped -> RUNNING
Checking if task is stoped -> RUNNING
Checking if task is stoped -> DEPROVISIONING
Checking if task is stoped -> DEPROVISIONING
Checking if task is stoped -> DEPROVISIONING
Checking if task is stoped -> STOPPED
{ taskArn: 'arn:aws:ecs:*********:************:task/***********/54fc25cf20b145c58182225b58882329',
  clusterArn: 'arn:aws:ecs:*********:************:cluster/***********',
  taskDefinitionArn: 'arn:aws:ecs:*********:************:task-definition/************-w-stg:56',
  overrides: { containerOverrides: [ [Object], [Object] ] },
  lastStatus: 'STOPPED',
  desiredStatus: 'STOPPED',
  cpu: '512',
  memory: '1024',
  containers: 
   [ { containerArn: 'arn:aws:ecs:*********:************:container/14f5adcf-f41a-4b84-a515-05f9c475afee',
       taskArn: 'arn:aws:ecs:*********:************:task/***********/54fc25cf20b145c58182225b58882329',
       name: 'datadog-agent',
       lastStatus: 'STOPPED',
       exitCode: 0,
       networkBindings: [],
       networkInterfaces: [Object],
       healthStatus: 'UNKNOWN' },
     { containerArn: 'arn:aws:ecs:*********:************:container/6e66dd91-45ed-4381-89eb-a6fd3ae3eace',
       taskArn: 'arn:aws:ecs:*********:************:task/***********/54fc25cf20b145c58182225b58882329',
       name: '************-w-stg',
       lastStatus: 'STOPPED',
       exitCode: 0,
       networkBindings: [],
       networkInterfaces: [Object],
       healthStatus: 'UNKNOWN' } ],
  version: 7,
  stoppedReason: 'Essential container in task exited',
  stopCode: 'EssentialContainerExited',
  connectivity: 'CONNECTED',
  connectivityAt: 2021-01-06T18:08:00.299Z,
  pullStartedAt: 2021-01-06T18:08:08.095Z,
  pullStoppedAt: 2021-01-06T18:09:04.095Z,
  executionStoppedAt: 2021-01-06T18:09:09.000Z,
  createdAt: 2021-01-06T18:07:51.633Z,
  startedAt: 2021-01-06T18:09:08.095Z,
  stoppingAt: 2021-01-06T18:09:23.711Z,
  stoppedAt: 2021-01-06T18:09:37.237Z,
  group: 'migration',
  launchType: 'FARGATE',
  platformVersion: '1.3.0',
  attachments: 
   [ { id: '6f2acc66-702e-4fd5-af16-b5c5fe34af7c',
       type: 'ElasticNetworkInterface',
       status: 'DELETED',
       details: [Object] } ],
  healthStatus: 'UNKNOWN',
  tags: [] }
CircleCI received exit code 0

Obs: é necessário liberar a permissão pro deployer pra rodar ecs:runTask =D