[HTTP 429 - too many requests] - error on terraform apply/refresh

[alpersid]

Hi there,

During terraform refresh/apply I receive the error HTTP 429 - too many requests.

From what I've found, the issue is on the API side and it can be solved by using parallelism=n.

I've tried with all types of parallelism, from 4 to 1 and still got the same issue.

From what I see from the output, the alarm rules are refreshed successfully, but when refreshing the as_policy, which is linked with an alarm, the error appears

module.autoscaling_karma.flexibleengine_as_configuration_v1.as_config_["OPL_autoscaling_pub"]: Refreshing state... [id=311455b4-b1ae-44d7-966a-42d17a7f8dd9]
module.autoscaling_karma.flexibleengine_as_configuration_v1.as_config_["OPL_autoscaling_priv_as"]: Refreshing state... [id=a40ec11c-a1fb-42b4-9fb0-b91bddefff6a]
module.autoscaling_papyrus.flexibleengine_as_group_v1.as_group_["OPL_autoscaling_priv_cc"]: Refreshing state... [id=81c21f4a-481a-473f-b6ac-08cda85dc425]
module.autoscaling_karma.flexibleengine_as_group_v1.as_group_["OPL_autoscaling_priv_as"]: Refreshing state... [id=09f8725a-278a-44a0-a3df-4dabd431a11a]
module.autoscaling_karma.flexibleengine_as_group_v1.as_group_["OPL_autoscaling_pub"]: Refreshing state... [id=02a7fc25-5cd1-4468-b88e-1cc64f9f8cdd]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_ADD_AS_CON_400"]: Refreshing state... [id=77e330d9-2099-4df5-ade3-d2aded9046e4]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_REM_AS_CON_200"]: Refreshing state... [id=9050a3c9-6cd9-45e2-a98a-770cc32d4eaf]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_REM_AS_CON_800"]: Refreshing state... [id=7abb16e1-84f7-4fbb-a6a7-23a2c9c86a9c]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_ADD_AS_ABNORMAL"]: Refreshing state... [id=59fd5fc5-c970-472c-a3e6-d685597d36a6]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_REM_RP_CON_200"]: Refreshing state... [id=848f2d73-fa98-4926-af47-74be53bb9634]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_REM_RP_CON_800"]: Refreshing state... [id=ae192282-71c9-4dec-af42-a3279d40ba01]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_REM_RP_CON_400"]: Refreshing state... [id=5207c459-c999-486c-acb2-01796d106d61]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_ADD_AS_CON_200"]: Refreshing state... [id=992a0a53-3362-484e-b01a-a91d14350457]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_ADD_RP_CON_800"]: Refreshing state... [id=6eed05d0-9bce-417e-a021-155e02991d90]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_REM_AS_CON_400"]: Refreshing state... [id=09cb37b2-8b7c-4a1b-b3ba-9871808ea51c]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_ADD_RP_CON_400"]: Refreshing state... [id=84d38e7e-5960-46bb-94d7-ff68e1cb9586]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_priv_as_ELB_ADD_AS_CON_800"]: Refreshing state... [id=e1578ee0-7d0c-4669-9f29-e3235a3694d2]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_ADD_RP_CON_200"]: Refreshing state... [id=a283bf0e-070c-4b16-90d2-c274168ce8f1]
module.autoscaling_karma.flexibleengine_as_policy_v1.as_policy_elb["OPL_autoscaling_pub_ELB_ADD_RP_ABNORMAL"]: Refreshing state... [id=10662d80-fb6e-4f49-8647-a4ddab36adca]
╷
│ Error: alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.
│ 
│ 
╵
╷
│ Error: alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.
│ 
│ 
╵

Terraform Version

15

FYI On openstack I see that there's a parameter for the openstack provider "max_retries", which can help with the error, but it's not currently implementedfor provider flexibleengine

max_retries - (Optional) If set to a value greater than 0, the OpenStack client will retry failed HTTP connections and Too Many Requests (429 code) HTTP responses with a Retry-After header within the specified value.

Test:

╷
│ Error: Unsupported argument
│ 
│   on provider.tf line 9, in provider "flexibleengine":
│    9:   max_retries = 2
│ 
│ An argument named "max_retries" is not expected here.
╵

Thank you in advance, Alex

[ShiChangkuo]

@alpersid the max_retries is not available in FlexibleEngine provider.

How many resources were managed in your module?

[alpersid]

Hello,

In the mentioned module I have 32 resources. Seems that using parallelism=1 makes it work after ~2 retries, but it's taking a lot of time.

Best regards, Alex

[ShiChangkuo]

@alpersid can you show the debug log? we can get the specified API which has exceeded the limitation on the cloud side.

[alpersid]

Hello,

Here you can find the full debug log:

2021-05-04T13:36:56.592Z [TRACE] vertex "module.autoscaling_karma.flexibleengine_ces_alarmrule.elb_alarm_rule_": dynamic subgraph encountered errors: alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try
 again.,alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.,alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.,alarmrule
: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.
2021-05-04T13:36:56.592Z [TRACE] vertex "module.autoscaling_karma.flexibleengine_ces_alarmrule.elb_alarm_rule_": visit complete
2021-05-04T13:36:56.592Z [TRACE] vertex "module.autoscaling_karma.flexibleengine_ces_alarmrule.elb_alarm_rule_ (expand)": dynamic subgraph encountered errors: alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute
, and try again.,alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.,alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.,
alarmrule: Too many requests have been sent in a given amount of time. Pause requests, wait up to one minute, and try again.

Thank you in advance, Alex

[ShiChangkuo]

Hi, Alex

After analyzing the debug log, it is confirmed that the issue is caused by the API flow control, and the corresponding API is Querying an Alarm Rule.

@antonin-a @qukuijin Maybe we need your help to expand the throttling threshold of specified API.

[antonin-a]

Hello @alpersid it seems to be caused by an API rate limit on platform side. The best solution for such cases is to open a ticket on Cloud Provider side (Flexible Engine Service Desk). Can you please send them the URL of this issue with a short description ? (@ShiChangkuo @qukuijin1989 )

[DafuSHI]

Meanwhile I will check with Ops side to see if they observed api limitation

[qukuijin1989]

we will raise the limitation

[alpersid]

Thanks guys for your quick reaction! I really appreciate all your hard work :)

[qukuijin1989]

HI @alpersid please have a try