search

prebid / prebid-server

Open-source solution for running real-time advertising auctions in the cloud.
https://prebid.org/product-suite/prebid-server/
Apache License 2.0
425 stars 720 forks source link

Refactor Vendor List Fetcher Package #3336

Open zhongshixi opened 9 months ago

zhongshixi commented 9 months ago

Hi @SyntaxNode, as promised I have compiled a list of changes I would like to happen in the vendor list fetch package so your team can cover it as part of the vendor list refactoring work.

Concurrency

  1. if I want to fetch an N vendor JSON file, I have to send an HTTP call for each file, this is not going to scale as we have a growing vendor list - I suggest we use concurrency so we can improve the performance. To not cause a network spike that produces connection error/timeout, we can either use a rate limiter or configure the http client with more live connections.

Configurable HTTP Client

  1. we want to have a configurable http client like timeout, max live connections or idle connections so that we can fine-tune the vendor list fetching performance

Customizable Retry

  1. we want to have a configuration to specify how long should we retry, and at which interval we should retry so that in case the vendor JSON is not available due to a network issue or endpoint issue, we can retry it with our own settings instead of a hard-code fixed number

Control the version range

  1. Instead of downloading all versions of the vendor JSON files, we want to control which versions to download, this also helps with improving the performance of the fetching logic by having fewer files to download.

Possible HTTP2

  1. if we can use http2 client to download the files from CloudFront, then it is even better, we can leverage its request multiplexing feature to significantly boost the downloading performance
bsardo commented 9 months ago

Hi @zhongshixi, thanks for sharing. I should be taking on the fetcher package refactor early next year and will consider all of your suggestions. I'll check back here when the project kicks off to further discuss.

zhongshixi commented 2 months ago

@bsardo @SyntaxNode I will wait for you guys discussion & ideas before I start the POC

zhongshixi commented 1 month ago

@bsardo, let me know if you have any updates.

I am going to do a short POC for my company's requirement to address the vendor list issue very soon; if you do have any updates from your side, definitely let me know so I can take them into consideration.