Azure: Hitting API limit - maybe add throttling?

[cheekoo]

Running Komiser on mac, against Azure cloud. Hitting 429, RESPONSE 429: 429 Too Many Requests. Can we add either throttling or re-starting where previous run failed?

Logs below -

_``` komiser start INIT!!! INFO[2024-04-13T19:55:15-05:00] Debug logging is enabled INFO[2024-04-13T19:55:15-05:00] Data will be stored in SQLite WARN[2024-04-13T19:55:15-05:00] Azure account cannot be inserted to database constraint failed: UNIQUE constraint failed: accounts.credentials (2067) INFO[2024-04-13T19:55:15-05:00] there are no new migrations to run (database is up to date) INFO[2024-04-13T19:55:15-05:00] Komiser version: 3.1.15, commit: 783b8a4b096ef0b37e0fb4557e779885ae762337, buildt: 1712400646 INFO[2024-04-13T19:55:15-05:00] Fetching resources workflow has started INFO[2024-04-13T19:55:15-05:00] Workflow triggered for client: Sample Subscription INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service="Application Gateway" INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Image INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Firewall INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Snapshot INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=0 service=Databox INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=3 service="PostgreSQL Database Servers" INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=63 service=Disk INFO[2024-04-13T19:55:16-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=6 service="Load Balancer" INFO[2024-04-13T19:55:18-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=370 service=Table INFO[2024-04-13T19:55:18-05:00] Fetched resources account="Sample Subscription" provider=Azure resources=1 service=Queue WARN[2024-04-13T19:55:34-05:00] failed to query usage: POST https://management.azure.com/subscriptions/xyz/providers/Microsoft.CostManagement/query

RESPONSE 429: 429 Too Many Requests ERROR CODE: 429

{ "error": { "code": "429", "message": "Too many requests. Please retry." } }

panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x2 addr=0x28 pc=0x102135b28]

goroutine 50 [running]: github.com/tailwarden/komiser/providers/azure/compute.VirtualMachines({0x103b11d40, 0x105d4d6e0}, {0x0, 0x0, {0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}) /home/runner/work/komiser/komiser/providers/azure/compute/virtual_machines.go:58 +0x768 github.com/tailwarden/komiser/providers/azure.FetchResources.func1() /home/runner/work/komiser/komiser/providers/azure/azure.go:37 +0x7c github.com/tailwarden/komiser/providers.(WorkerPool).worker(0x14001100040) /home/runner/work/komiser/komiser/providers/providers.go:85 +0x3c created by github.com/tailwarden/komiser/providers.(WorkerPool).Start in goroutine 7 /home/runner/work/komiser/komiser/providers/providers.go:69 +0x2c

[github-actions[bot]]

Hey, thank you for opening your first Issue! 🙂 While a Tailwarden team member takes a look at your issue we would like to invite you to join our official Discord server, where you can interact directly with other contributors and Tailwarden team members. Link here: https://discord.tailwarden.com

[thiagosestini]

I'm having the same issue, any updates? This currently causes the application to fail.

[azgcloudev]

@mlabouardy is this been reviewed? If not I can take a look