More & improved stats: funnel filtering etc.

[rvagg]

We have a funnel now:

But it's naive. We don't include filtering so we can't tell why retrievals are dropping off - the number at the bottom isn't a reflection of potential. So we should capture what we need to better reflect that so we can figure out where the next lowest-hanging fruit is.

• Indexer filtering - we're currently filtering on concurrency, although we're discussing dropping that
• Query filtering - currently filtering on paid asks
• Retrieval attempts - a "request" can have multiple failures and one success, we don't currently have a way of representing this. We can do success/(success+failure) but that's not necessarily right if we want to group by retrievals. Need to come up with ideas about (a) how to represent these and (b) what the most important numbers are.

[hannahhoward]

Yes this is all super helpful.

Seems like with retrieval, there should be a one to one relationship between "successful queries" (I think this happens after the query filter) and retrieval success + failure. The only question we might need to answer is whether we need a "did not attempt" category for no attempt due to simultaneous retrieval limits. in that case success + failure + did not attempt = successful queries

[kylehuntsman]

It's still a little unclear what the actionable item is here. Are we talking about being able to visualize the number of filtered out items as we pass through the funnel?

I'm in agreement with successful retrieval + failed retrieval + did not attempt = successful queries.

[kylehuntsman]

Confirmed offline with team that we want to be able to view the number of requests in each step pre and post filter.

[rvagg]

Coming back here after looking at #145 and thinking through what we really want to see. I think now that what we are needing in the immediate term is a view of what part of the drop-off is artificial and what is due to network conditions. So:

• Indexer - it's interesting to know how many random DHT CID requests get looked up in the indexer, but currently we're not showing that because we're filtering on concurrency.
  • Requests with Indexer Candidates (increment counter if we got >0, pre-filter)
  • Requests with Usable Indexer Candidates (increment counter if we got >0 post-filter - what we have now)
• Query - same deal but with paid query responses
  • Requests with Successful Queries (increment counter if we got >0, pre-filter)
  • Requests with Usable Queries (increment counter if we got >0, post-filter - what we do now)

Additionally, we want insight into counts at each of these phases but that's not something we can put in the funnel, but we can do histograms of distributions at the phases we care about. So let's set up view.Distribution() type metrics for:

• Indexer Candidate Count - pre-filter count of indexer candidates for each request (I don't think post-filter is as useful here, we could add it later if needed)
• Successful Query Count - pre-filter count of number of query-asks we got back
• Usable Query Count - post-filter count of number of query-asks that aren't paid
• Failed Retrieval Count - this is on a per-request basis, it's the number of retrievals we attempt in that loop before we get a successful one

This will all become more complicated when we do #136 of course, but I think we can still make use of most of this in roughly the same way, you just have to interpret the data slightly differently.