Consider componentized architecture

[bretg]

During the Prebid Summit this week, there was a desire for the ability to call the auction "programmatically" - imported as a library and called within a memory space by another code base.

We need the community to weigh in on actual use cases here because we have concerns about surfacing internal APIs in a way that hamstrings future development. One use case, for instance, might be:

• Just let me invoke open source Prebid bid adapters directly from my SSAI server. I don't need stored requests, I don't need modules, I don't need ORTB responses or analytics - I just want to have the logic for the bid adapters in my product and forget about Prebid.

If you're a product owner that wants to cherry-pick pieces of Prebid Server into your product, please help us understand why.

If you want to use the whole Prebid Server workflow (from stored-request through analytics), then please help us understand why being a standalone server is a source of problems. For most of us on the committee, PBS is so high volume, complex, and specialized that it's hard to imagine wanting to glom it all inside a multi-purpose server.

[matthieularere-msq]

I don't know whom this desire comes from but I would be interesting by the exact use case you described. The reason is as use prebid-server from a network company working with various ssps. The publishers we are working with are relunctant to add our many ssps into their header bidding, so we've built a proxy server on top of prebid-server in order to appear as only one bidder in their prebid.js. Because only one prebid-server can be set in a prebid.js setup, it made sense not be take this slot.

This also simplifies the need we have to add additional features which feats our exact needs by adding them in the afore mentioned proxy. For example, filtering the creatives send based on blocklist or floors. We also have a system to manage the ssps associated which each ad placements, a logging event system... It was easier to put that outside prebid-server.

Thus, it means that our "proxy" needs to call prebid-server throught http. This had an impact on performance due to the amount of tcp connections our servers needed to keep open. We circumvented this issue by pushing a PR to prebid-server so that it supported unix socket. We still call prebid-server throught http but now it's using a unix socket and no more tcp. However we could probably improve performance by calling prebid-server auction mechanism throught an imported library. I am not sure how much it would improve performance, but I my guess is that avoiding to have a dedicated http server could save some ressources.

Obviously I need the cookie_sync endpoint too in order to get the cookies urls I need to sync my uids cookie with. But there again if it could be a library I load and not a http call to make, it could save ressources.

However, the truth is I am happy with the current product already. While it would be nice to be able to use prebid-server as a cherry picking library and not a full packaged http daemon, it fits my needs and the way I call it throught its endpoint already feels like an API I am hosting. Having it as an imported library would be for a me just a way to improve performance.

Does it make sense ?

[bretg]

Thanks for the thoughts @matthieularere-msq

I wonder if your use case could now be met by building the extra functionality using the module infrastructure?

e.g. you could hook into the Processed Auction Response stage and overwrite the actual bidder with whatever you want. Same with the "filtering the creatives send based on blocklist or floors" feature.

I get that you've invested a lot in your front-end server, but just saying that this is why we built the module system -- so that your business-specific needs can be integrated more easily for hosting fewer tiers of servers.

[matthieularere-msq]

While it could probably be mostly made using modules, I have concerns about it:

• I would have no interest in pushing such a thing through a PR here as I don't expect my specific needs to be use by anyone else, or at least not the way I do.
• I would have to comply with the way the module infrastructure is designed which is a constraint, even if I don't apply to the guidelines rules as I don't need to push here
• I probably don't want some competitors to be able to take advantage of features I developped.
• Thus, I would have to maintain a private fork and keep it updated with its upstream repo
• Any archicture changes in the upstream prebid-server repo could break my module, requiring some work to fix

I can be wonrg, but I see modules more as a way to be able to distribute components to external companies. For internal use I am unsure if it's appropriate. While I could indeed consider it, I don't know if it worth the shot. However, as I said, while I would be happy to use prebid-server as an imported library, I am already satisfied with the way I use it right now. I just see that as a way for performance improvement.

[bretg]

Glad you're satisfied with the state of things @matthieularere-msq .

For the record, private modules are totally fine. In fact, they're more common than public modules... Magnite has several non-open source modules with custom logic that wouldn't apply to others. It's how we keep our forked changes to nearly zero and make merges much easier.

Any architecture changes in the upstream prebid-server repo could break my module

Also for the record, the module interface is meant to be stable. We will aim to not break modules.

[SyntaxNode]

For internal use I am unsure if it's appropriate.

It's entirely appropriate.

Actually, it's a use case we specifically targeted. Modules are a stable way to extend PBS functionality in your fork. For PBS-Go, we made sure to minimize potential fork conflicts. There is only one file in the repository which needs to be edited when adding a private module, and we shipped a code generator for that file to make conflicts easy to solve. You could automate the code gen in a fork sync script if you'd like.

the module interface is meant to be stable

That's the goal. We're still in the early stages, so there might be some short term tweaks. Long term you can expect a stable interface.

[zhongshixi]

will start to add some detailed idea how to modularize different module

1. the bidder info-loading module has hard code dependency entirely on the local filesystem, I propose we modularize it by letting the user define how to load the bidder with common interface ( how to load it and where we can load it ) In my specific requirement, I would like to have it stored and loaded as embed.FS so when I import prebid server as library, those bidder info yaml files are part of the prebid-server binary.

2. I also propose we could provide a way for the user to build their server and load it in the prebid server without writing it in the prebid server's main code base - that means , a prebid server provides a module(hook) interface to for the consumer to load its hook code

[bretg]

Discussed in committee. No use cases have been found that would drive a major rearchitecture. We encourage the use of private modules to make merging open source changes easier.

We're closing this, but would be open to discussing new use cases.