[RestAPI] Out of memory error

[rafaelsantanaep]

Mandatory informations:

Are there any customers directly impacted by this bug? Which ones?

---

• ativycom

Bug Category

• [ ] Connection Manager
• [ ] Connection Test
• [ ] Creating Pipeline
• [x] Executing Pipeline
• [ ] Cataloging Data Asset
• [ ] Other

Describe the bug

The ECS task is terminated because the allocated amount is exceeded. This error occurred with the tap-rest-api-msdk, which currently have 24g allocated. To solve it temporally, we should increase the amount of memory allocated for the singer connector and, later on, decrease the amount of memory to keep costs controlled.

To increase the amount of memory, we have to change this file:

• pipeline_id: f2f0343b_4a7c_4fe6_bd82_f3805309e0d6

Other informations:

What environment of software are you using?

• [x] PRD
• [ ] Other

When the bug happened: 2023-03-16

[rafaelsantanaep]

@cicerojmm @oliveira-devops Refatorei o CICD do singer-connector para conseguirmos customizar a quantidade alocada de memória do ECS por plugin em vez de por connector como estava hoje. Joguei a quantidade alocada pro rest-api-msdk lá pra cima e vou pedir pro @vitorddf testar em conjunto com a ativy para ver se resolveu.

Posteriormente, é bom descermos a quantidade de memória alocada pra diminuirmos nossos custos.

Houveram outras PRs, mas a que houve mais mudanças foi essa aqui: https://github.com/dadosfera/singer-connector/pull/272/files

Também aproveitei para refatorar o CI com o intuito de ficar mais clean e utilizar a feature de environments do github.