Improve ability to test bidder responses

pm-harshad-mane commented 4 years ago

Type of issue

Feature Request

Description

While debugging on a live inventory, ad-ops people need to request test bids from bidders. We have a to modify bid responses but it needs changes in the config http://prebid.org/dev-docs/troubleshooting-tips.html#modify-bid-responses-for-testing DistrictM has its own way, it looks for dmTest=true in page URL, but bidder specific test-query-param is not a scalable solution.

Proposal

All bidders read a common query param pbjs_testbids=true and if present in page URL then each bidder will send respective test-bids flag to their server.

pm-harshad-mane commented 4 years ago

Rather than each bidder reading the same param, this param can be made available through config.getConfig('testbids') // true/false

pm-harshad-mane commented 4 years ago

we can also support pbjs.setConfig({ testbids: true });

jaiminpanchal27 commented 4 years ago

@bretg @mkendall07 Any thoughts on this new feature request

pm-harshad-mane commented 4 years ago

Hello @bretg, @mkendall07 Could you please share your thoughts on this new feature request?

bretg commented 4 years ago

What is the bidder expectation here? Just avoid charging for the bid? Extra debug info? Fake creative? Really whatever the expectation, it's pretty impractical -- that would be a very long term goal.

You need to identify the actual use cases... what does the pub want to test:

header bidding ads high enough to win in the ad server?
ad server line items?
PBJS config like bidAdjustments?
a specific type of creative?

I don't think trying to herd 200+ bidder-cats into providing fake bid+creatives is going to happen anytime soon. And the requirements are too vague -- does the pub want a bid of a specific value to trigger a particular line item? Do they want to see bidderA win for once?

I think a much better approach would be to deepen the feature we already have documented at http://prebid.org/troubleshooting/troubleshooting-guide.html#modify-bid-responses-for-testing

One could imagine a Chrome extension that would help make it easier for AdOps people to identify an adunit and force things to happen... all within PBJS.

pm-harshad-mane commented 4 years ago

The expectation from the bidder is to serve a fake creative. This can help test the end-to-end setup, not all bidders respond with bids always. I agree that it will be difficult to have all bidders support it. The existing approach is useful only in a test environment where one has access to the PBJS config to modify bid responses.

bretg commented 4 years ago

The existing approach is useful only in a test environment

Not true - the approach uses the javascript console to set local storage and modify future bid responses. It's pretty cool. Try it! However, it does have the shortcoming there has to be a bidresponse to modify.

What we need is a way to inject bids that never happened. @snapwich - do you think we could extend debugging.js to have some no-bid behavior that when debugging is turned on?

1) Always inject an event handler for noBid 2) If the bidRequest that resulted in noBid matches the bidderCode+AdUnit defined 3) Create and inject a bidResponse from the bid value

Then, from the console, someone could define a creative. e.g.

> pbjs.setConfig({
  debugging: {
    enabled: true,
    bids: [{
      bidder: 'bidderA',
      adUnitCode: 'test-div',
      cpm: 1.5,
      adId: '111111',
      ad: '<html><body><img src="https://files.prebid.org/creatives/prebid300x250.png"></body></html>'
    }]
  }
});

snapwich commented 4 years ago

I kind of like the idea of having an actual test bid returned from the bidder; that makes it a true end-to-end test. While some bidders do accept returning test bids as a flag on bid.params, I do see an issue with it being global config as I would say the majority of bidders do not support test bids... which means, does it make sense to just leave that configuration as bidder specific on bid.params? I do see some value in making it global as a "build it and they will come" type approach, where maybe if the config exists more bidders might start implementing test bid support.

As for returning mocked bids, I almost feel that'd be better served by a mock bidder adapter. In-fact, I think we had one internally at rubicon... but maybe that's something worth adding to open source? The issue I have with stubbing bidders through the debugging module is while you can say what the bidder is returning, the bidder's implementation might return nothing like that (you hope they would, but you never know), so I think it could give you a false sense of security; getting an actual test bid back from the bidder seems a lot more valuable for an end-to-end test.

bretg commented 4 years ago

I think the use case here is mainly to confirm the publisher in-page code and the line item setup.

A mock bidder adapter would be fine if there were a runtime way to force/override aliases from the javascript console.

e.g.

1) page adunits are setup with bidderA, bidderB, bidderC 2) the goal is to test the actual page code and line item setup 3) tester runs a command to alias bidderA-->mockBidder, bidderB-->mockBidder, bidderC-->mockBidder 4) tester can then use the existing debugging feature to change bid values

pm-harshad-mane commented 4 years ago

Hello @bretg , @snapwich

Sorry for the late response, I was on a holiday trip.

I agree that the debugging related code can be added through the console and it doesn't need page access. Thank you for the sample code.

But I think the bid-modification works only if the bidder responds with a bid as the bid-response-hook is used. https://github.com/prebid/Prebid.js/blob/master/src/debugging.js#L37 Let me know if I have missed something.

Now I think having a common flag will be confusing as all active bidders will submit a tets-bid and it will be difficult to test a bidder specific setup (like line-item).

What do you think about making changes in the existing debugging module to add test-bids (even if a bidder does not respond) through the existing approach? we might need to use bidder-done-hook OR we can add a functionality to modify bidder request params so that test param can be added for required bidder at run time ( in this approach support from each bidder will be required).

bretg commented 4 years ago

@pm-harshad-mane - you're right that the debug feature only works for bidders that return bids. It's structural. Rich suggested a 'mock-bidder' adapter, but we would want that built directly into core. Perhaps a new localstorage-based feature that causes core to skip actual bids altogether and just uses a supplied bidresponse?

e.g.

> pbjs.setConfig({
  debugging: {
    enabled: true,
    bidOverride: [{    // processed before the auction rather than after
      bidder: 'bidderA',
      adUnitCode: 'test-div',
      cpm: 1.5,
      adId: '111111',
      ad: '<html><body><img src="https://files.prebid.org/creatives/prebid300x250.png"></body></html>'
     },{
      bidder: 'bidderB',
      adUnitCode: 'test-div',
      cpm: 1.7,
      adId: '22222',
      ad: '<html><body><img src="https://files.prebid.org/creatives/prebid300x250.png"></body></html>'
    }]
  }
});

[Edit] - we would want to avoid firing analytics adapters in this debug scenario.

GeneGenie commented 4 years ago

@bretg it is also a good idea to supply isDebugBid property in bid. So analytic adapters may skip/adjust reporting

Alejandro-MartinG commented 4 years ago

Hi @all,

What's the status of this feature? will be implement soon?

I'm interested too in have a way to test prebid really. Will be interesting have a tool or an agile way to request test bids from all my bidders, cause when you refresh the website prebid start to return no bids all time. And sometimes when I develop an "advance" feature which involve googleadmanager + prebid the only way to know if Prebid is working correctly is show statistics in the next days.

mmanashirov commented 4 years ago

A mock bidder adapter would be a great option to have. The existing debug config to change the bid response is of limited use because it requires a valid bid response in the first place which often won't be available in a testing environment.

gglas commented 3 years ago

Hey guys - revitalizing the thread, it seems to me that the core use case is to have a Mock Bidder available in order to process bids in an auction to make sure that the setup is working properly, rather than having bidder-specific end points that will regularly return test bids (which is a separate use case, one to test whether or not a bidder is responding as expected / that their endpoint is working). The core limitation currently being that 1. Bret's sample code might be difficult for basic users to implement and 2. it requires a bidder to have returned a bid at some point, which in certain situations can be difficult.

Is that a fair summary @pm-harshad-mane ? If so, I think this is a great project to collaborate on with @ChrisHuie and will surface at our next product planning meeting.

pm-harshad-mane commented 3 years ago

Thank you @gglas for summarizing. Yes the main purpose is to be able to test the setup for a bidder without waiting for a non-zero bid from the bidder.

bretg commented 3 years ago

I'm going to reiterate my proposal to create a "bid response override" feature

1) bidders aren't going to return ads 100% of the time. They just aren't. We have to deal with that. e.g. Rubicon has traffic shaping in place so even if our static test IDs are used, test requests can be filtered out in high traffic periods. 2) a 'mock bidder' would bloat all PBJS packages and be every bit as complex to configure 3) Professor Prebid is getting closer to reality. Once it's real, we can abstract away the complexity with a nicer UI.

bretg commented 2 years ago

Talked with Demetrio, he's going to build what I've suggested above:

> pbjs.setConfig({
  debugging: {
    enabled: true,
    bidOverride: [{    // processed before the auction rather than after
      bidder: 'bidderB',
      adUnitCode: 'test-div',
      bidResponseParams: {
          cpm: 1.7,
          adId: '22222',
          ad: '<html><body><img src="https://files.prebid.org/creatives/prebid300x250.png"></body></html>'
          meta: { blah }
          currency: 
          ... anything that can go into a bidResponse now or in the future should be able to go here ...
    }]
  }
});

Expected results:

if the a bid response comes in from bidderB
AND the adunit.code is test-div
THEN ignore bidderB's actual response (e.g. NOBID, TIMEOUT) and override it with bidOverride.bidResponseParams

Whether or not bidderB is actually called is irrelevant.

prebid / Prebid.js