Google::Apis::CloudbillingV1::CloudbillingService#list_billing_accounts with fetch_all returns duplicated billing accounts

[minimum2scp]

I have 3 billing accounts.

% gcloud beta billing accounts list
ACCOUNT_ID            NAME                                 OPEN   MASTER_ACCOUNT_ID
000***-******-***CE2  ***********@gmail.com                True
01B***-******-***B3E  ********@gmail.com                   True
003***-******-***18B  Billing Account for my test project  False

CloudbillingService#list_billng_accounts returns 3 accounts, it's ok.

% cat script1.rb
#! /usr/bin/env ruby

require 'googleauth'
require 'google/apis/cloudbilling_v1'

service = Google::Apis::CloudbillingV1::CloudbillingService.new
service.authorization = Google::Auth.get_application_default(%w[https://www.googleapis.com/auth/cloud-billing.readonly])

ret = service.list_billing_accounts
ret.billing_accounts.each.with_index do |account, idx|
  puts "[#{idx}] #{account.name} #{account.display_name}"
end

% 
% bundle exec ./script1.rb
[0] billingAccounts/000***-******-***CE2 ***********@gmail.com
[1] billingAccounts/003***-******-***18B Billing Account for my test project
[2] billingAccounts/01B***-******-***B3E ********@gmail.com

But when I wrapped list_billing_accounts with fetch_all, it returns 6 billing accounts! (3 are duplicates)

% cat script2.rb
#! /usr/bin/env ruby

require 'googleauth'
require 'google/apis/cloudbilling_v1'

service = Google::Apis::CloudbillingV1::CloudbillingService.new
service.authorization = Google::Auth.get_application_default(%w[https://www.googleapis.com/auth/cloud-billing.readonly])

ret = service.fetch_all(items: :billing_accounts){ |token, s|
  s.list_billing_accounts(page_token: token)
}.to_a

ret.each.with_index do |account, idx|
  puts "[#{idx}] #{account.name} #{account.display_name}"
end

% 
% bundle exec ./script2.rb
[0] billingAccounts/01B***-******-***B3E ********@gmail.com
[1] billingAccounts/003***-******-***18B Billing Account for my test project
[2] billingAccounts/000***-******-***CE2 ***********@gmail.com
[3] billingAccounts/000***-******-***CE2 ***********@gmail.com
[4] billingAccounts/01B***-******-***B3E ********@gmail.com
[5] billingAccounts/003***-******-***18B Billing Account for my test project

I found next_page_token is empty string, not nil. I think this is the cause.

[7] pry(main)> ret = service.list_billing_accounts
=> #<Google::Apis::CloudbillingV1::ListBillingAccountsResponse:0x0000557309537f60
 @billing_accounts=
  [#<Google::Apis::CloudbillingV1::BillingAccount:0x0000557309535968 @display_name="Billing Account for my test project", @master_billing_account="", @name="billingAccounts/003***-******-***18B", @open=false>,
   #<Google::Apis::CloudbillingV1::BillingAccount:0x0000557309534770 @display_name="********@gmail.com", @master_billing_account="", @name="billingAccounts/01B***-******-***B3E", @open=true>,
   #<Google::Apis::CloudbillingV1::BillingAccount:0x0000557309a33aa0 @display_name="***********@gmail.com", @master_billing_account="", @name="billingAccounts/000***-******-***CE2", @open=true>],
 @next_page_token="">
[8] pry(main)> ret.next_page_token
=> ""

see also: https://github.com/googleapis/google-api-ruby-client/blob/c6266b6fe944a1e9cd5c392e000fed4c4f9df180/lib/google/apis/core/base_service.rb#L84

Environment details

• OS: Linux
• Ruby version: 2.7.2
• Gem name and version: google-api-client 0.47.0

[minimum2scp]

And one more.

When I add page_size paramter, list_billing_accounts with fetch_all enters infinite loop!

% cat script3.rb
#! /usr/bin/env ruby

require 'googleauth'
require 'google/apis/cloudbilling_v1'

service = Google::Apis::CloudbillingV1::CloudbillingService.new
service.authorization = Google::Auth.get_application_default(%w[https://www.googleapis.com/auth/cloud-billing.readonly])

idx = 0
service.fetch_all(items: :billing_accounts){ |token, s|
  s.list_billing_accounts(page_token: token, page_size: 1)
}.each do |account|
  puts "[#{idx}] #{account.name} #{account.display_name}"
  idx += 1
  sleep 1
end

% 
% bundle exec ./script3.rb
[0] billingAccounts/01B***-******-***B3E ********@gmail.com
[1] billingAccounts/000***-******-***CE2 ***********@gmail.com
[2] billingAccounts/003***-******-***18B Billing Account for my test project
[3] billingAccounts/01B***-******-***B3E ********@gmail.com
[4] billingAccounts/000***-******-***CE2 ***********@gmail.com
[5] billingAccounts/003***-******-***18B Billing Account for my test project
[6] billingAccounts/01B***-******-***B3E ********@gmail.com
[7] billingAccounts/000***-******-***CE2 ***********@gmail.com
[8] billingAccounts/003***-******-***18B Billing Account for my test project
[9] billingAccounts/01B***-******-***B3E ********@gmail.com
[10] billingAccounts/000***-******-***CE2 ***********@gmail.com
[11] billingAccounts/003***-******-***18B Billing Account for my test project
[12] billingAccounts/01B***-******-***B3E ********@gmail.com
[13] billingAccounts/000***-******-***CE2 ***********@gmail.com
[14] billingAccounts/003***-******-***18B Billing Account for my test project
[15] billingAccounts/01B***-******-***B3E ********@gmail.com
[16] billingAccounts/000***-******-***CE2 ***********@gmail.com
[17] billingAccounts/003***-******-***18B Billing Account for my test project
[18] billingAccounts/01B***-******-***B3E ********@gmail.com
[19] billingAccounts/000***-******-***CE2 ***********@gmail.com
[20] billingAccounts/003***-******-***18B Billing Account for my test project
^CTraceback (most recent call last):
    7: from ./script3.rb:12:in `<main>'
    6: from /tmp/b/vendor/bundle/ruby/2.7.0/gems/google-api-client-0.47.0/lib/google/apis/core/base_service.rb:63:in `each'
    5: from /tmp/b/vendor/bundle/ruby/2.7.0/gems/google-api-client-0.47.0/lib/google/apis/core/base_service.rb:63:in `loop'
    4: from /tmp/b/vendor/bundle/ruby/2.7.0/gems/google-api-client-0.47.0/lib/google/apis/core/base_service.rb:67:in `block in each'
    3: from /tmp/b/vendor/bundle/ruby/2.7.0/gems/google-api-client-0.47.0/lib/google/apis/core/base_service.rb:67:in `each'
    2: from /tmp/b/vendor/bundle/ruby/2.7.0/gems/google-api-client-0.47.0/lib/google/apis/core/base_service.rb:70:in `block (2 levels) in each'
    1: from ./script3.rb:15:in `block in <main>'
./script3.rb:15:in `sleep': Interrupt

[dazuma]

Thanks for the analysis! Yes, it looks like this is due to the next_page_token being unexpectedly the empty string rather than nil. We'll get a fix out.

[dazuma]

You can now update google-apis-core to version 0.4.1 for the fix.

Note: the google-api-client gem is deprecated. Please use the individual gem for the service you are using. For billing, this will be google-apis-cloudbilling_v1

[minimum2scp]

@dazuma Thank you!