GET /units-for-slot/{slotId}: the "supermarket"

Problem

There's a few things about the current market that can be easily optimized/improved:

the adview manager needs to request info about the slot first, and then all campaigns; this can be done in one request
we can do the targeting server-side and only return relevant campaigns; some targeting rules can only be applied on the client side (e.g. AdEx Profile, frequency capping) - so those rules will simply be ignored (see AIP31)
because of those limitations, we have to set cache times high, which leads to this issue, which happens because campaigns are still returned for some time after they've exhausted/expired

Solution

A new route that allows to get all units matching a certain ad slot. We will build it as a separate component called the "supermarket" and we'll run it separately, and route on a NGINX level

This gives us 2 advantages:

🚄 Speed: by doing 1 request instead of 2, and using a in-memory data structure, and Rust, we'd be able to cut cache times down to only a few seconds and deliver fresher data; furthermore, AdEx ads will load faster, and less KB will be sent over network
🧐 Traceability: if there are no viable ad units, it will return the precise reasons, which means easier debugging of "why are my ads not showing"; also allows better internal stats

Functionality:

[x] implements a route /units-for-slot, which returns all ad units which match this slot; it will apply targeting as per AIP31 https://github.com/AdExNetwork/adex-supermarket/issues/9
[x] pulls up-to-date campaign data often and directly from validators to avoid trailing impressions Supermarket impl
[ ] applies targeting logic server-side using the adview manager rust code
[] (separated in own issue #170) returns issues and/or stats: a list of possible reasons why there are no returned units: NO_ACTIVE_CAMPAIGNS, CAMPAIGNS_NOT_SOUND, NO_DEPOSITASSET_CAMPAIGNS, NO_UNITS_FOR_SIZE, NO_UNITS_FOR_TARGETING, NO_UNITS_FOR_ADSLOTRULES, SLOT_NOT_VERIFIED (if acceptedReferrers.length == 0)

Tech design

The supermarket will function in-memory, without a database. It will pull all data it needs from the validators.

Recommendations for internal data structures:

active: HashMap<CampaignId, CamapignWithInfo> where CamapignWithInfo holds the channel, latest balance, latest status and etc.
finalized: Set<CampaignId> - a set of finalized campaigns (exhausted/expired, whatever)

That way only active (non-finalized) campaigns are kept in memory and updated, and once a campaign becomes inactive (which is irreversible), it will be flagged in finalized. Note that unsound (Unhealthy, Invalid, etc.) campaigns are not finalized - only Exhausted/Closed/Withdraw/Expired are finalized.

On start-up and every few minutes, we will get all known campaigns from a configurable list of validators. Every few seconds (10-20), we will update our active campaigns from the validators (update their latest balance tree/messages/status).

Ad slots can be retrieved from the market on demand without a cache: this means every request to /unit-for-slot will first request the slot from the market. The market endpoint will be configurable. Later on, we can cache that too if needed.

Applying earning limits

There are a few concepts of earning limits within AdEx: the limits recommended by the Market per slot. Those are based on the Alexa rank of the website in question and may include quick account limits too in the future.

We cannot apply earning limits because we can't compute all lifetime earnings because we drop finalized campaigns from memory (also, we only crawl active ones from validators). But the Market can apply limits at an ad slot level, by sneaking in { onlyShowIf: false } within adSlot.rules.

Prerequisites

https://github.com/AdExNetwork/aips/issues/31 and it's matching engine - although that's mostly implemented

AmbireTech / adex-market