Unclear Account Config

[GLStephen]

can anyone clarify what the env var for cache size for PBS_ACCOUNTS_IN_MEMORY_CACHE_TYPE : lru should be? It seems like it should be PBS_ACCOUNTS_IN_MEMORY_CACHE_SIZE_BYTES but that does not seem to work. config/config.go has

v.SetDefault("accounts.in_memory_cache.type", "none")

but following that naming from stored_response and stored_request caches doesn't seem to work.

    v.SetDefault("stored_requests.in_memory_cache.request_cache_size_bytes", 0)
    v.SetDefault("stored_requests.in_memory_cache.imp_cache_size_bytes", 0)
    v.SetDefault("stored_requests.in_memory_cache.resp_cache_size_bytes", 0)

while the config for stored requests implies "size_bytes"

https://github.com/prebid/prebid-server/blob/439c7edeab04385d1116b0b97248a872a44bce40/config/stored_requests.go#L311

but that didn't seem to work either, and this isn't technically a stored_request, but it used the stored_request struct, anyway. stored_request has mapstructure:"ttl_seconds" and similar items.

The config error block implies "size_bytes" based on the stored_request config, is that accurate and this is being pulled from there even though it's an account setting?

https://github.com/prebid/prebid-server/blob/439c7edeab04385d1116b0b97248a872a44bce40/config/stored_requests.go#L353

It seems the following should work:

PBS_ACCOUNTS_IN_MEMORY_CACHE_TYPE:lru
PBS_ACCOUNTS_IN_MEMORY_CACHE_TTL_SECONDS:300
PBS_ACCOUNTS_IN_MEMORY_CACHE_SIZE_BYTES:500000

but only TYPE seems to be recognized. Any hints on what this config is exactly?

[VeronikaSolovei9]

Hello @GLStephen, thank you for submitting your question!

There are two types of cache can be initialized in Prebid Server: "lru" and "unbounded".

"unbounded" type is a cache where stored data is loaded at server start up in a thread safe map(*sync.Map). Data in "unbounded" cache updates with a frequency configured in

PBS_{STORED_RESPONSES}_{POSTGRES}_POLL_FOR_UPDATES_REFRESH_RATE_SECONDS: "60"

To enable "unbounded" cache type just set

PBS_ACCOUNTS_IN_MEMORY_CACHE_TYPE: "unbounded"

or configure fetchers:

PBS_STORED_RESPONSES_IN_MEMORY_CACHE_TYPE: "unbounded"

Data in this cache type suppose to stay in memory during server lifetime and can be deleted or updated after "poll for updates", hence ttl is not needed.

"lru" type is a cache with Least Recently Used optimization. It should work similar to "unbounded" cache type. I was able to make it work locally by adding this config:

PBS_ACCOUNTS_IN_MEMORY_CACHE_TYPE: lru
PBS_ACCOUNTS_IN_MEMORY_CACHE_SIZE_BYTES: 500000

[GLStephen]

@VeronikaSolovei9 from what I can tell these values are not initialized in Config.go.

PBS_ACCOUNTS_IN_MEMORY_CACHE_SIZE_BYTES: 500000

In addition, even once initializing them the server continually crashes. I spoke to @SyntaxNode on Slack and I'm available to do the dev to fix this, but there is a possibility of a deeper error. In the interim I'll use the refresh rate config. Are these available too? You only mentioned response and postgres.

PBS_ACCOUNTS_POLL_FOR_UPDATES_REFRESH_RATE_SECONDS: "60"
PBS_STORED_REQUESTS_POLL_FOR_UPDATES_REFRESH_RATE_SECONDS: "60"

[VeronikaSolovei9]

Right, config.go sets only default values. You need to configure your server based on what data storage you want to use.

I tested it locally using my local Postgres DB where I have table and data for stored responses only.

Prebid Server currently supports Postgres and MySQL data bases, file system or http.

Note these values for each fetcher in configs:

v.SetDefault("stored_video_req.http_events.endpoint", "")
v.SetDefault("stored_video_req.http_events.refresh_rate_seconds", 0)
v.SetDefault("stored_video_req.http_events.timeout_ms", 0)

v.SetDefault("stored_video_req.filesystem.enabled", false)
v.SetDefault("stored_video_req.filesystem.directorypath", "")

"Poll for updates" works for DB data providers only.

Please note PBS doesn't support DB fetcher for accounts.

It's hard to say what exactly is not right in your config, it's a trick set up. If you can share more about it, we may try to figure out the best way to satisfy your needs. I'll confirm with @SyntaxNode if I can do anything in the meantime.

[GLStephen]

Right, config.go sets only default values.

That's not exactly accurate. The default values also prepare Viper to load the values from ENV vars versus yaml. Without the defaults in config the values don't load from env vars. That's the initial bug that I encountered.

After properly setting those values, the server crashes. I'm still investigating that.

I don't need help with configuring the server though my question was phrased that way. I was being a bit deferential in my post and inquiring about the config because I wasn't sure if I was missing something. Our server is running fine, other than wanting to configure this particular cache differently. There is an issue in this config related to accounts. First, that the default vars aren't loaded so they can't be loaded from ENV vars. Second, that once set it leads to a crash.

[VeronikaSolovei9]

The default values also prepare Viper to load the values from ENV vars versus yaml. Without the defaults in config the values don't load from env vars.

This is correct too. We discussed this with @SyntaxNode, it looks like you found some missing variables related to cache. This should be fixed, yes, good point! It seems like you made it work. Please let us know if you need anything else.

[bretg]

@VeronikaSolovei9 - could you please summarize what needs to happen for this issue? I believe it's documentation.