bidder-data.csv has spaces, causing line-item-manager to not find line items

jrinco11 commented 1 year ago

line-item-manager version: 0.2.9
Python version: 3.11
Operating System: Mac 13.3.1 (Ventura)

Description

On Aug 8, I ran line-item-manager to create orders for a handful of bidders. On Aug 9, when trying to continue with other SSPs, I started to run into an issue where I would receive error output stating "Following items were not found after creation" with a list of line item names that are prepended with whitespace (a single space), and I'm only now getting around to having a look at why this is happening.

What I Did

Here is a sample of the command run (network-code removed for privacy):

python % line_item_manager create ./lineitemmanager/gumgum_8.01-10.99.yaml -k gam_creds_james.json -b gumgum --network-code [NETWORK_CODE_REMOVED] --network-name="Test GAM Account" INFO:line_item_manager.gam_config:################################################################################ INFO:line_item_manager.gam_config:Bidder: name=" GumGum", code="gumgum" INFO:line_item_manager.gam_config:Key: "hb_pb_gumgum", Values: ['8.01', '8.02', '8.03', ..., '10.97', '10.98', '10.99'] INFO:line_item_manager.gam_config:############################################################ INFO:line_item_manager.gam_config:Media Type: "banner" INFO:line_item_manager.gam_config:Line Items: CPMs(min=8.01, max=10.99, count=299) INFO:line_item_manager.gam_config:Line Item Creative Associations: Creative Count=13 ERROR:line_item_manager.cli:Unexpected result, Following items were not found after creation: '[' GumGum: HB $8.01', ' GumGum: HB $8.02', ' GumGum: HB $8.03', ' GumGum: HB $8.04', ' GumGum: HB $8.05', ' GumGum: HB $8.06', ' GumGum: HB $8.07', ' GumGum: HB $8.08', ' GumGum: HB $8.09', ' GumGum: HB $8.10', ' GumGum: HB $8.11', ' GumGum: HB $8.12', ' GumGum: HB $8.13', ' GumGum: HB $8.14', ' GumGum: HB $8.15', ' GumGum: HB $8.16', ' GumGum: HB $8.17', ' GumGum: HB $8.18', ' GumGum: HB $8.19', ' GumGum: HB $8.20', ' GumGum: HB $8.21', ' GumGum: HB $8.22', ' GumGum: HB $8.23', ' GumGum: HB $8.24', ' GumGum: HB $8.25', ' GumGum: HB $8.26', ' GumGum: HB $8.27', ' GumGum: HB $8.28', ' GumGum: HB $8.29', ' GumGum: HB $8.30', ' GumGum: HB $8.31', ' GumGum: HB $8.32', ' GumGum: HB $8.33', ' GumGum: HB $8.34', ' GumGum: HB $8.35', ' GumGum: HB $8.36', ' GumGum: HB $8.37', ' GumGum: HB $8.38', ' GumGum: HB $8.39', ' GumGum: HB $8.40', ' GumGum: HB $8.41', ' GumGum: HB $8.42', ' GumGum: HB $8.43', ' GumGum: HB $8.44', ' GumGum: HB $8.45', ' GumGum: HB $8.46', ' GumGum: HB $8.47', ' GumGum: HB $8.48', ' GumGum: HB $8.49', ' GumGum: HB $8.50', ' GumGum: HB $8.51', ' GumGum: HB $8.52', ' GumGum: HB $8.53', ' GumGum: HB $8.54', ' GumGum: HB $8.55', ' GumGum: HB $8.56', ' GumGum: HB $8.57', ' GumGum: HB $8.58', ' GumGum: HB $8.59', ' GumGum: HB $8.60', ' GumGum: HB $8.61', ' GumGum: HB $8.62', ' GumGum: HB $8.63', ' GumGum: HB $8.64', ' GumGum: HB $8.65', ' GumGum: HB $8.66', ' GumGum: HB $8.67', ' GumGum: HB $8.68', ' GumGum: HB $8.69', ' GumGum: HB $8.70', ' GumGum: HB $8.71', ' GumGum: HB $8.72', ' GumGum: HB $8.73', ' GumGum: HB $8.74', ' GumGum: HB $8.75', ' GumGum: HB $8.76', ' GumGum: HB $8.77', ' GumGum: HB $8.78', ' GumGum: HB $8.79', ' GumGum: HB $8.80', ' GumGum: HB $8.81', ' GumGum: HB $8.82', ' GumGum: HB $8.83', ' GumGum: HB $8.84', ' GumGum: HB $8.85', ' GumGum: HB $8.86', ' GumGum: HB $8.87', ' GumGum: HB $8.88', ' GumGum: HB $8.89', ' GumGum: HB $8.90', ' GumGum: HB $8.91', ' GumGum: HB $8.92', ' GumGum: HB $8.93', ' GumGum: HB $8.94', ' GumGum: HB $8.95', ' GumGum: HB $8.96', ' GumGum: HB $8.97', ' GumGum: HB $8.98', ' GumGum: HB $8.99', ' GumGum: HB $9.00', ' GumGum: HB $9.01', ' GumGum: HB $9.02', ' GumGum: HB $9.03', ' GumGum: HB $9.04', ' GumGum: HB $9.05', ' GumGum: HB $9.06', ' GumGum: HB $9.07', ' GumGum: HB $9.08', ' GumGum: HB $9.09', ' GumGum: HB $9.10', ' GumGum: HB $9.11', ' GumGum: HB $9.12', ' GumGum: HB $9.13', ' GumGum: HB $9.14', ' GumGum: HB $9.15', ' GumGum: HB $9.16', ' GumGum: HB $9.17', ' GumGum: HB $9.18', ' GumGum: HB $9.19', ' GumGum: HB $9.20', ' GumGum: HB $9.21', ' GumGum: HB $9.22', ' GumGum: HB $9.23', ' GumGum: HB $9.24', ' GumGum: HB $9.25', ' GumGum: HB $9.26', ' GumGum: HB $9.27', ' GumGum: HB $9.28', ' GumGum: HB $9.29', ' GumGum: HB $9.30', ' GumGum: HB $9.31', ' GumGum: HB $9.32', ' GumGum: HB $9.33', ' GumGum: HB $9.34', ' GumGum: HB $9.35', ' GumGum: HB $9.36', ' GumGum: HB $9.37', ' GumGum: HB $9.38', ' GumGum: HB $9.39', ' GumGum: HB $9.40', ' GumGum: HB $9.41', ' GumGum: HB $9.42', ' GumGum: HB $9.43', ' GumGum: HB $9.44', ' GumGum: HB $9.45', ' GumGum: HB $9.46', ' GumGum: HB $9.47', ' GumGum: HB $9.48', ' GumGum: HB $9.49', ' GumGum: HB $9.50', ' GumGum: HB $9.51', ' GumGum: HB $9.52', ' GumGum: HB $9.53', ' GumGum: HB $9.54', ' GumGum: HB $9.55', ' GumGum: HB $9.56', ' GumGum: HB $9.57', ' GumGum: HB $9.58', ' GumGum: HB $9.59', ' GumGum: HB $9.60', ' GumGum: HB $9.61', ' GumGum: HB $9.62', ' GumGum: HB $9.63', ' GumGum: HB $9.64', ' GumGum: HB $9.65', ' GumGum: HB $9.66', ' GumGum: HB $9.67', ' GumGum: HB $9.68', ' GumGum: HB $9.69', ' GumGum: HB $9.70', ' GumGum: HB $9.71', ' GumGum: HB $9.72', ' GumGum: HB $9.73', ' GumGum: HB $9.74', ' GumGum: HB $9.75', ' GumGum: HB $9.76', ' GumGum: HB $9.77', ' GumGum: HB $9.78', ' GumGum: HB $9.79', ' GumGum: HB $9.80', ' GumGum: HB $9.81', ' GumGum: HB $9.82', ' GumGum: HB $9.83', ' GumGum: HB $9.84', ' GumGum: HB $9.85', ' GumGum: HB $9.86', ' GumGum: HB $9.87', ' GumGum: HB $9.88', ' GumGum: HB $9.89', ' GumGum: HB $9.90', ' GumGum: HB $9.91', ' GumGum: HB $9.92', ' GumGum: HB $9.93', ' GumGum: HB $9.94', ' GumGum: HB $9.95', ' GumGum: HB $9.96', ' GumGum: HB $9.97', ' GumGum: HB $9.98', ' GumGum: HB $9.99', ' GumGum: HB $10.00', ' GumGum: HB $10.01', ' GumGum: HB $10.02', ' GumGum: HB $10.03', ' GumGum: HB $10.04', ' GumGum: HB $10.05', ' GumGum: HB $10.06', ' GumGum: HB $10.07', ' GumGum: HB $10.08', ' GumGum: HB $10.09', ' GumGum: HB $10.10', ' GumGum: HB $10.11', ' GumGum: HB $10.12', ' GumGum: HB $10.13', ' GumGum: HB $10.14', ' GumGum: HB $10.15', ' GumGum: HB $10.16', ' GumGum: HB $10.17', ' GumGum: HB $10.18', ' GumGum: HB $10.19', ' GumGum: HB $10.20', ' GumGum: HB $10.21', ' GumGum: HB $10.22', ' GumGum: HB $10.23', ' GumGum: HB $10.24', ' GumGum: HB $10.25', ' GumGum: HB $10.26', ' GumGum: HB $10.27', ' GumGum: HB $10.28', ' GumGum: HB $10.29', ' GumGum: HB $10.30', ' GumGum: HB $10.31', ' GumGum: HB $10.32', ' GumGum: HB $10.33', ' GumGum: HB $10.34', ' GumGum: HB $10.35', ' GumGum: HB $10.36', ' GumGum: HB $10.37', ' GumGum: HB $10.38', ' GumGum: HB $10.39', ' GumGum: HB $10.40', ' GumGum: HB $10.41', ' GumGum: HB $10.42', ' GumGum: HB $10.43', ' GumGum: HB $10.44', ' GumGum: HB $10.45', ' GumGum: HB $10.46', ' GumGum: HB $10.47', ' GumGum: HB $10.48', ' GumGum: HB $10.49', ' GumGum: HB $10.50', ' GumGum: HB $10.51', ' GumGum: HB $10.52', ' GumGum: HB $10.53', ' GumGum: HB $10.54', ' GumGum: HB $10.55', ' GumGum: HB $10.56', ' GumGum: HB $10.57', ' GumGum: HB $10.58', ' GumGum: HB $10.59', ' GumGum: HB $10.60', ' GumGum: HB $10.61', ' GumGum: HB $10.62', ' GumGum: HB $10.63', ' GumGum: HB $10.64', ' GumGum: HB $10.65', ' GumGum: HB $10.66', ' GumGum: HB $10.67', ' GumGum: HB $10.68', ' GumGum: HB $10.69', ' GumGum: HB $10.70', ' GumGum: HB $10.71', ' GumGum: HB $10.72', ' GumGum: HB $10.73', ' GumGum: HB $10.74', ' GumGum: HB $10.75', ' GumGum: HB $10.76', ' GumGum: HB $10.77', ' GumGum: HB $10.78', ' GumGum: HB $10.79', ' GumGum: HB $10.80', ' GumGum: HB $10.81', ' GumGum: HB $10.82', ' GumGum: HB $10.83', ' GumGum: HB $10.84', ' GumGum: HB $10.85', ' GumGum: HB $10.86', ' GumGum: HB $10.87', ' GumGum: HB $10.88', ' GumGum: HB $10.89', ' GumGum: HB $10.90', ' GumGum: HB $10.91', ' GumGum: HB $10.92', ' GumGum: HB $10.93', ' GumGum: HB $10.94', ' GumGum: HB $10.95', ' GumGum: HB $10.96', ' GumGum: HB $10.97', ' GumGum: HB $10.98', ' GumGum: HB $10.99']' INFO:line_item_manager.gam_config:Auto-archiving Orders: [3242729885]

Notice the bidder name is referenced as " GumGum", as well as the "missing items" all have that same leading space in their names above. When looking at how the line items were created in GAM, they interestingly do not have that whitespace included, presumably therefore the reason why they can't be found per the output above.

When investigating, I noticed that https://docs.prebid.org/dev-docs/bidder-data.csv, which is used by the script to get the list of bidder names/codes/etc, has a space present after each comma (field) for all data except the header row, such as:

bidder-code,bidder-name,banner,video,native,schain,dchain,tcfeu,coppa,gpp,usp,safeframes,deals,client-adapter,server-adapter,user-ids,mobile-apps,floors,fpd,prebid-member,ortb-blocking,multiformat ... ... ... gumgum, GumGum, yes, yes, no, yes, check with bidder, yes, check with bidder, yes, yes, check with bidder, check with bidder, yes, yes, unifiedId identityLink, yes, yes, check with bidder, no, check with bidder, check with bidder ... ... ...

If, however, I reference a local copy of bidder-data.csv with the leading spaces removed, such as below, the script runs as expected and no errors:

bidder-code,bidder-name,banner,video,native,schain,dchain,tcfeu,coppa,gpp,usp,safeframes,deals,client-adapter,server-adapter,user-ids,mobile-apps,floors,fpd,prebid-member,ortb-blocking,multiformat ... ... ... gumgum,GumGum,yes,yes,no,yes,check with bidder,yes,check with bidder,yes,yes,check with bidder,check with bidder,yes,yes,unifiedId identityLink,yes,yes,check with bidder,no,check with bidder,check with bidder ... ... ...

python % line_item_manager create ./lineitemmanager/gumgum_11.00-40.00.yaml -k gam_creds_james.json -b gumgum --network-code [NETWORK_CODE_REMOVED] --network-name="Test GAM Account" INFO:line_item_manager.gam_config:################################################################################ INFO:line_item_manager.gam_config:Bidder: name="GumGum", code="gumgum" INFO:line_item_manager.gam_config:Key: "hb_pb_gumgum", Values: ['11.00', '12.00', '13.00', ..., '38.00', '39.00', '40.00'] INFO:line_item_manager.gam_config:############################################################ INFO:line_item_manager.gam_config:Media Type: "banner" INFO:line_item_manager.gam_config:Line Items: CPMs(min=11.00, max=40.00, count=30) INFO:line_item_manager.gam_config:Line Item Creative Associations: Creative Count=13 INFO:line_item_manager.gam_config:Line Item Creative Associations: Writing 390 records... 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 390/390 [00:53<00:00, 7.33it/s] python % line_item_manager

It seems the bidder-data.csv from prebid should be updated to remove the whitespace in the first place, but perhaps line-item-manager should/could also account for unexpected whitespace like this? (potentially the bidder-data.csv changed between Aug-8 and Aug-9 since I only ran into this starting Aug-9?)

dshore commented 1 year ago

Thank you. I am reviewing this and will follow up.

dshore commented 1 year ago

I am going to treat this as a bug and strip leading/trailing whitespace from bidder names upon parsing. Additionally, I have reached out to prebid via slack and asked to possibly repost without the ambiguous whitespace.

Thx @jrinco11 for identifying this issue

dshore commented 1 year ago

This has been fixed in development (not yet released to pypi) in two ways:

Prebid has reposted the bidder metadata without spaces
The code now skips initial whitespace from the file

If you would like to test this you can follow these commands:

$ pip uninstall line-item-manager
$ pip install https://github.com/prebid/line-item-manager/archive/master.zip
$ line_item_manager --version
line-item-manager version 0.2.9-dev3

prebid / line-item-manager

bidder-data.csv has spaces, causing line-item-manager to not find line items #128

Description

What I Did