search

mage-ai / mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.
https://www.mage.ai
Apache License 2.0
8k stars 776 forks source link

ECS Memory Utilization keeps rising #4921

Open dumim opened 7 months ago

dumim commented 7 months ago

Mage version

v0.9.68

Describe the bug

We have deployed the Mage image into ECS using terraform as described in the docs: https://github.com/mage-ai/mage-ai-terraform-templates

There was an earlier issue with memory rising quickly that was reported by us: https://github.com/mage-ai/mage-ai/issues/3244 which was fixed by an update.

This increase is much slower than the above issue, happening slowly over days and weeks: image image

The drop in memory is when the ECS was restarted due to us pushing a new image. We have not let it run for long enough without any restarts to see it crash at 100%, this is a preemptive report to the team before it happens.

To reproduce

No response

Expected behavior

No response

Screenshots

No response

Operating system

No response

Additional context

No response

dumim commented 6 months ago

@wangxiaoyou1993 any update on this? This is still rising: image

jfcieslak commented 2 months ago

We are experiencing a similar issue on azure container instance (also from tf template).

Screenshot 2024-09-06 at 10 40 09