Closed finkjordanj closed 2 months ago
obrigg
commented
2 months ago This doesn't look right. How often does this happen? Kindly open a case with Meraki support so they can look into this issue. You should not receive random 404 responses for a valid request. That said, this is not an issue with the Terraform Provider but the responses you're getting.
finkjordanj
commented
2 months ago I have opened a support case# 12146799
I also randomly got a number of 500 internal server errors. I applied a baseline config to a number of sites and each seemed to fail with an internal server error but on different calls. this is one example of the failure.
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: 2024/09/11 00:26:58
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ==============================================================================
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ~ REQUEST(curl) ~
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: CURL:
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: curl -X PUT -H 'Accept: application/json' -H 'Authorization:
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ~ REQUEST ~
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: PUT /api/v1/networks/L_783626335162466505/switch/mtu HTTP/1.1
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: HOST : api.meraki.com
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: HEADERS:
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Accept: application/json
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Authorization:
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Content-Type: application/json
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: User-Agent: go-meraki/1.44.1 MerakiTerraform/1.47.0 Cisco
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: BODY :
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: {
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: "defaultMtuSize": 3000
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: }
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ------------------------------------------------------------------------------
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ~ RESPONSE ~
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: STATUS : 500 Internal Server Error
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: PROTO : HTTP/2.0
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: RECEIVED AT : 2024-09-11T00:26:58.727694805Z
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: TIME DURATION: 1.157107657s
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: HEADERS :
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Content-Length: 37
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Content-Type: application/json
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Date: Wed, 11 Sep 2024 00:26:58 GMT
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Etag: "66df77f0-25"
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: Strict-Transport-Security: max-age=31536000; includeSubDomains
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: BODY :
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: {
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: "errors": [
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: "Internal server error"
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ]
2024-09-11T00:26:58.727Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: }
2024-09-11T00:26:58.728Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha
2024-09-11T00:26:58.728Z [DEBUG] provider.terraform-provider-meraki_v0.2.11-alpha: ==============================================================================
Error: -11T00:26:58.728Z [ERROR] provider.terraform-provider-meraki_v0.2.11-alpha: Response contains error diagnostic: diagnostic_summary="Failure when executing UpdateNetworkSwitchMtu" tf_provider_addr=registry.terraform.io/CiscoDevNet/meraki tf_req_id=f9046f3c-4966-f7f5-c354-3e532f7dece5 @caller=github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov6/internal/diag/diagnostics.go:58 diagnostic_detail="error with operation UpdateNetworkSwitchMtu" diagnostic_severity=ERROR tf_proto_version=6.6 tf_resource_type=meraki_networks_switch_mtu tf_rpc=ApplyResourceChange @module=sdk.proto timestamp=2024-09-11T00:26:58.727Z
2024-09-11T00:26:58.728Z [DEBUG] State storage *remote.State declined to persist a state snapshot
Error: -11T00:26:58.728Z [ERROR] vertex "module.pfg-managed-meraki-baseline.meraki_networks_switch_mtu.this_site_switch_mtu[0]" error: Failure when executing UpdateNetworkSwitchMtu
obrigg
commented
2 months ago Hi Jordan, I'm following the support case. Do you mind running the plan again, and share the debug (assuming the issue is still happening)? I'm looking for a "fresh" occurrence.
finkjordanj
commented
2 months ago Its a very strange behavior. If it just runs a state check/refresh in my nightly drift, i let it run at that 1 request per second it runs fine, but during a new apply which i can trigger by just wiping my stored state file it will sometimes give that error.
I will try and run this in my test environment after removing the state files and see if i can trigger this 500 internal error.
finkjordanj
commented
2 months ago @obrigg i have attached a copy of a fresh full debug output from the init to the apply/error to the Meraki case if you are able get the debug from there. I didn't want to try and review the debug before posting on a public github.
obrigg
commented
2 months ago Thanks. This time it shows a 500 error, which is clearly an internal issue. Support should be able to handle it from there.
obrigg
commented
1 month ago Hi @finkjordanj, I'm still keeping track of the support case. I'll be clarifying a few things internally. Meanwhile, when running an initial built, where many POST/PUTs are expected, a workaround would be leveraging the meraki_requests_per_second attribute to avoid the specific backend congestion you've hit.
finkjordanj
commented
1 month ago Understood @obrigg . My main pipeline does have that set to running only 1 request per second and i limit my runners running an apply to one site per Org at a time. If i did leave it at the default I would hit the API rate limit errors a lot more often due to other systems also leveraging API calls at random times. This is why i don't think its exactly a rate limit issue at this point but instead more what they described in the ticket where multiple calls may occur and isn't responding fast enough to stay out of its own way. It is strange as its not every time so I'm either hitting odd que timers which may result in this error or some node in a lb farm or something that isn't handling this happily.
Once i get my main apply completed and its at most smaller corrections in drift checks its usually fine, but ideally I'm looking to build out entire networks from code and risking these overruns just leave me in a state where I'm not sure if I trust the pipeline enough to commit that far.
obrigg
commented
1 month ago I expect you can be fine with a higher rate than 1 call per second. You know better than me what other integrations consume your budget.
In that note, if you happen to look into identifying causes for hitting your rate limit, would you be open to have a conversation with one of my colleague so we can get your feedback on a few ideas?
finkjordanj
commented
1 month ago Sure that would be fine with me. They can reach out to me with my company email fink.jordan@principal.com. I have been slowly trying to organizing some of the API access by key as we have a few system using the same API key so if there are ways to better track usage would def be glad to look into.
obrigg
commented
1 month ago Thank you. If you join our API early access group, you might find a few interesting and relevant things for organizing API keys and multiple systems.
finkjordanj
commented
1 month ago @obrigg related to the oauth discussion from the API group. I haven't had a chance to dig into it a ton, but based on the auth token just being passed back will the terraform provider/registry be able to use this in place of the API key or will there need to be adjustments for being able to try and leverage this. Based on your python example i think it would match up. The other question i can pose over there but you may have the answer, does the oauth have the same API call rate limiting imposed onto it?
obrigg
commented
1 month ago Hi Jordan, An OAuth access token can be used just like an API key (for 60 minutes). OAuth Tokens and API keys share the same API rate-limit. OAuth will make it easier to identifying which app is consuming how many operations (as opposed to all integrations using the same API key/identity).
Prerequisites
Describe the bug I've been encountering random 404 responses on some resource calls, both during builds/changes and state refreshes. We have a nightly drift detection job that verifies our site configurations match the Terraform state.
Currently, I've set the API rate limit to 3 requests per second, but I plan to reduce it to 1 to see if this improves the situation. However, I'm unsure if the issue is related to rate limits or something else.
I've noticed a mix of behaviors—most runs either complete successfully or hit an API rate limit, while others return 404 errors according to the debug output. These 404 errors are particularly puzzling since I haven't seen them when testing the API directly, and support has been unable to reproduce the issue outside of my GitHub environment.
When I hit the API rate limit, several resource calls are usually affected, but in this instance, only one call received a 404, while two other resource calls completed successfully in the same run.
Expected behavior Would get correct response or error out.
Screenshots
Debug
These are run off a github runner so getting an actual pcap of the packets to validate the response is actually from the Meraki portal
The calls earlier in this same run that responded fine
Failure 404 response
Environment (please complete the following information):
Additional context