ParthaI
commented
2 months ago Hello @laurensknoll,
Thank you for bringing this issue to our attention.
I've reviewed the current code and noted that when querying the googledirectory_group_member table, the group_id parameter is essential for calling the List Members API.
I've implemented some updates in the issue-88 branch to enhance how members are listed concurrently:
Updated Table Behavior in issue-88:
- The
googledirectory_grouptable now acts as the parent forgoogledirectory_group_member. - The
group_idcolumn has been made optional. - The Group ID can be derived from the parent table.
- It is now optional to specify
group_idwhen queryinggoogledirectory_group_member, allowing for simpler queries likeselect * from googledirectory_group_member.
I have modified the query plan to make it faster. The behavior of the query(mentioned by you in the issue description) will remain the same as we are making the API call by passing the Group ID.
WITH GroupDetails AS (
SELECT
id AS group_id,
customer_id,
name AS group_name
FROM
googledirectory.googledirectory_group
WHERE
customer_id = 'my_customer_id'
),
MemberDetails AS (
SELECT
id AS member_id,
group_id,
type AS member_type,
email AS member_email,
status AS member_status
FROM
googledirectory.googledirectory_group_member
)
SELECT
g.customer_id,
g.group_id,
g.group_name,
m.member_id,
m.member_type,
m.member_email,
m.member_status
FROM
GroupDetails g
JOIN MemberDetails m ON m.group_id = g.group_id;Please try pulling the changes from issue-88, build the plugin locally, and run the above query. Your feedback would be greatly appreciated.
Steps to build the plugin locally with the changes from issue-88:
- Clone the repository:
git clone https://github.com/turbot/steampipe-plugin-googledirectory.git - Change to the directory:
cd steampipe-plugin-googledirectory - Check out the branch:
git checkout issue-88 - Build the plugin:
make - Execute the query mentioned above.
Looking forward to your feedback. Thanks again!
laurensknoll
Is your feature request related to a problem? Please describe. When querying all googledirectory_group_member using the next query, the members are not queried in parallel, making it slow:
Describe the solution you'd like The preferred solution executes the
googledirectory_group_member.list-methods in parallel, using a configurable concurrency to reach the Admin SDK rate limit of 2400 req/min.Describe alternatives you've considered
Additional context In the plugin log, the group_member-calls are listed sequentially:
And, consequently, the Admin SDK API monitoring lists 1 to 2 req/second.