user_table not returning all results

[rmaalouf]

Describe the bug

select * from jira_user doesn't display all records and returns an inconsistent number of records every time it is run (even without clearing the cache)

Additional information

• There is no error so you'd think everything worked properly
• select count(*) from jira.jira_user does display the correct number of users
• Adding order by causes a 429 error every time. Example: select * from jira.jira_user order by display_name Error: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.
• Reducing MaxConcurrency seems to fix the issue. I reduced it from 50 to 5 and consistently get all records.
• Running against a Jira cloud instance with a few hundred users

Steampipe version (steampipe -v) Example: v0.16.4

Plugin version (steampipe plugin list) Example: v0.6.1

[misraved]

Thanks for raising the bug @rmaalouf. Sorry to see that you are bumping into issues.

Could you please share the plugin-level logs when you run the query select * from jira_user?

[e-gineer]

Hey @rmaalouf ... Steampipe does sub-API calls for some columns to get their data. In this case, the problem is the group_names column calling a different API for each row (too quickly).

Notice the Hydrate function in this code:

https://github.com/turbot/steampipe-plugin-jira/blob/main/jira/table_jira_user.go#L69-L75

As a workaround, does this work? (Avoiding the group_names column):

select account_id, display_name from jira_user

For a more permanent solution, I think we need to add retry / backoff to the getUserGroups function:

https://github.com/turbot/steampipe-plugin-jira/blob/main/jira/table_jira_user.go#L148-L164

[rmaalouf]

Thanks for the assistance. On previous days I was able to replicate this on every query. This morning select * from jira_user worked several times in a row. However after a few clear/query cycles I was able to replicate the issue (results are incomplete but no error is returned). Here are the logs:

2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429" 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429" 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429" 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: jira_user.getUserGroups: api_error="429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429" 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111) 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111) 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: streamRows for jira-1667223580111 - execution has failed (table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: executeForConnection jira returned error table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: callHydrate getUserGroups finished with error: 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: error channel received table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111) 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: hydrate error chan select error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: getRow failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429 (jira-1667223580111) 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [ERROR] steampipe-plugin-jira.plugin: [ERROR] 1667223580111: table 'jira_user' failed to get column data, connectionCallId jira-1667223580111: table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429. 2022-10-31 13:39:42.466 UTC [WARN] steampipe-plugin-jira.plugin: [WARN] 1667223580111: QueryData StreamError table 'jira_user' column 'group_names' requires hydrate data from getUserGroups, which failed with error 429 Too Many Requests: local_rate_limited: request failed. Please analyze the request body for more details. Status code: 429.

[rmaalouf]

@e-gineer that appears to do the trick. After hitting the 429 issue several times in a row I avoided selecting the group_names column as you suggested and got the expected full number of records.

[rmaalouf]

Does this approach make sense:

• If retry/backoff limit is reached, user should be made aware that 429 was encountered
• Query should still succeed as is currently the case, but group_names should be left joined instead of inner joined so that the query still returns the full list of users.

[misraved]

Hello @rmaalouf, thanks a lot for extensively testing out the table and providing valuable insights on the error logs 👍 . Apologies for the delay, but I was finally able to get a hold of this issue.

Initially, I was inclined towards reducing the MaxConcurrency parameter to 5, since it worked for you in the first instance, but I finally settled on adding a RetryConfig to the getUserGroups function.

I have created a PR - https://github.com/turbot/steampipe-plugin-jira/pull/72 which fixes the issue of API Rate limiting in the jira_user table. Could you please rebuild the plugin using this PR and let us know if it fixes your issue?

[rmaalouf]

Hi @misraved, thank you for taking the time to look into this. Good news, it appears this PR fixes the issue.

I tested it by first reproducing the issue and then I rebuilt the plugin using this PR and ran the following 10 times in a row:

select * from jira_user
.cache clear

The query returned all users (~700) every time.