Open damianesteban opened 6 months ago
scottslowe
commented
6 months ago Thanks for this issue, @damianesteban! It is a known issue that it sometimes takes 2-3 minutes before the ECS service will start responding, and until it does start responding you'll get a 503. Do you mind if I ask how long you waited after the service was created?
matthewelwell
commented
3 months ago I am also seeing a similar issue. It looks as though the ECS service is just flapping, but I can't see any useful logs being outputted from the tasks that are being killed. See screenshot below for event logs from the service.
You can also see that the tasks never seem to register as healthy in the target group (which is likely what is causing the 503/504s). I just see this in the target group.
Inspecting a failed task, this is unfortunately all I can see.
Confirmation of empty logs.
I have tried updating the health check endpoint in the target group to /api/health as per the article here, however, that only seemed to make the issue worse and caused the health checks to actively fail.
I'm continuing to investigate but any assistance here would be greatly appreciated. Note that, in direct response to the question asked by @scottslowe to the author of this issue, I waited ~10 minutes for the service to come online and I was still seeing 503s (well technically a mixture of 503s, 502s and 504s as the ECS service flapped about).
matthewelwell
commented
3 months ago For a limited period of time, it looks like the service does at least try to load, but I just see the following (it never loads past this loading screen):
matthewelwell
commented
3 months ago For anyone getting here, the issue is that the RDS configuration is outdated. This package is configured to create a serverless (v1) RDS instance using mysql 5.7 which (it seems) metabase no longer supports. Because of this, the migrations fail and hence, the service never becomes healthy.
To resolve this, I manually added logging configuration which seems to have been commented out here. This was possibly because of incorrect IAM policies. I think perhaps the task execution role needs access to createLogGroup although I have not yet verified this - I just created the log group manually myself.
Once I had the logs, I was able to determine the above DB issue and tried to just update the engine version using the (undocumented) database: DatabaseArgs init parameter to the Metabase constructor (see here). I tried to set engine_version=8.0.mysql_aurora.3.05.2 but unfortunately with the combination of other parameters that are hardcoded, this doesn't work (since 8.0 is not available via serverless v1).
Finally, I ended up just creating a new RDS serverless v2 instance from scratch and manually updated the task definition to point to the new database.
Based on this, I think there is a significant chunk of work to do against this repository to update the RDS configuration. A few options here would be:
Probably a combination of all 3 of the above is the best approach.
tday
commented
1 month ago I was able to resolve this by changing the metabase version to one that works with the supported Aurora version
const metabaseService = new metabase.Metabase("metabaseService", {
metabaseVersion: "v0.47.0"
});
What happened?
I followed the directions as per the documentation, and though Pulumi is successful, I get a 503 when I try to access Metabase.
Unfortunately, I cannot see any logs in the ECS console.
Example
Here is my Pulumi Code:
Output of
pulumi aboutAdditional context
I would love to use this Provider and even contribute to it.
Contributing
Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).