freeze at apply and plan stages if we have many resources

[eill]

Hello, seems the provider (because of Mikrotik rate limiting or something like that) is freezing after checking or creating many records (greater than 30 dns records, for example).

Sure, we can create the resources by groups of 10 resources, for example, just by uncommenting them 10 by 10, but at some moment, when the total records are about 60 or 70 the plan command stops working too.

2020/10/28 14:49:08 [TRACE] EvalReadState: reading state for module.mikrotik_dns_pve.mikrotik_dns_record.static_records["pve200"] 2020/10/28 14:49:08 [TRACE] UpgradeResourceState: schema version of module.mikrotik_dns_pve.mikrotik_dns_record.static_records["pve200"] is still 0; calling provider "mikrotik" for any other minor fixups module.mikrotik_dns_pve.mikrotik_dns_record.static_records["pve200"]: Refreshing state... [id=pve200.company.local] 2020/10/28 14:49:12 [TRACE] dag/walk: vertex "module.mikrotik_dns_pve (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:12 [TRACE] dag/walk: vertex "root" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records[\"pve200\"]" 2020/10/28 14:49:13 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:13 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" 2020/10/28 14:49:17 [TRACE] dag/walk: vertex "module.mikrotik_dns_pve (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:17 [TRACE] dag/walk: vertex "root" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records[\"pve200\"]" 2020/10/28 14:49:18 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:18 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" 2020/10/28 14:49:22 [TRACE] dag/walk: vertex "module.mikrotik_dns_pve (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:22 [TRACE] dag/walk: vertex "root" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records[\"pve200\"]" 2020/10/28 14:49:23 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:23 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" 2020/10/28 14:49:27 [TRACE] dag/walk: vertex "module.mikrotik_dns_pve (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:27 [TRACE] dag/walk: vertex "root" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records[\"pve200\"]" 2020/10/28 14:49:28 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:28 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" 2020/10/28 14:49:32 [TRACE] dag/walk: vertex "module.mikrotik_dns_pve (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:32 [TRACE] dag/walk: vertex "root" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records[\"pve200\"]" 2020/10/28 14:49:33 [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)" is waiting for "module.mikrotik_dns_pve.mikrotik_dns_record.static_records (expand)" 2020/10/28 14:49:33 [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/ddelnano/mikrotik\"] (close)"

I've tried to google some deep details about API rate limits in Mikrotik, but didn't found any useful information.

Maybe we should somehow ratelimit requests to API on our side?

Thanks in advance for you answer.

[ddelnano]

@eill thanks for the bug report. I found these docs of rate limiting (https://wiki.mikrotik.com/wiki/User_Manager/Limiting). I'm not sure what the "user manager" is but it seems like it might be related to Mikrotik's RADIUS AAA. Are you using RADIUS and/or do you have anything configured under /ip hotspot user print or ip hotspot profile print? What user are you providing to the terraform provider?

The docs on this aren't very clear so maybe the isn't the right thing to look at but I'll have to see if I can reproduce this on my deployment.

[eill]

We've found that terraform parameter -parallelism=1 resolve this issue completely. I think we can close the issue or rework the provider to handle all calls to API via single connection object (but I don't know is it possible)

[ddelnano]

This is definitely a bug imo but glad that you are able to work around it by preventing parallelism. Since there is an easy workaround I don't know when I'll prioritize this but it should be fixed.