Add more summary data for donors based on donee and donation amount distributions

vipulnaik / donations

Donations list website (DLW): a repository for keeping track of public donations by some people I (arbitrarily) decide to track

https://donations.vipulnaik.com

Creative Commons Zero v1.0 Universal

20 stars 2 forks source link

Add more summary data for donors based on donee and donation amount distributions #108

Closed vipulnaik closed 5 years ago

vipulnaik commented 5 years ago

I was thinking of information like:

Quantiles of grants (what the biggest grant size was, what the smallest grant size was, what grant sizes at the 10th, 20th, ... percentiles were)
Quantiles of total amounts received by grantee
The above, broken down by cause area
For each grant, provide its "rank" among grants.

Some of these measures make sense only when the number of grants is large, and also how finely we quantile depends on how many grants there are

Also displaying some very summary version (e.g., median grant size) of this sort on the homepage would allow for easier sorting based on those metrics

riceissa commented 5 years ago

Does the following work, for the first three bullet points?

(the example is Open Phil)

riceissa commented 5 years ago

For each grant, provide its "rank" among grants.

What measure of rank do you want to use? percentile? or just "n out of N"?

vipulnaik commented 5 years ago

The screenshot looks good, and matches what I want

I think n out of N is easiest for rank.

riceissa commented 5 years ago

The stuff in your bullet point list is now done.

The only thing left is:

Also displaying some very summary version (e.g., median grant size) of this sort on the homepage would allow for easier sorting based on those metrics

I'm not sure what to do about this one. For per-donor or per-donee statistics, I am just pulling all the amounts into a PHP array (which takes one additional query per page) and calculating the percentiles using that. Doing this for the main page where there are many donors and donees (especially the donors and donees with the most number of associated donations) might be tricky (high memory usage? long load time?). There might be a way to do this in SQL (there are some Stack Overflow questions about it but I haven't looked into it in depth).

Thoughts?

vipulnaik commented 5 years ago

@riceissa for the home page piece, I'm wondering if it's better to run a precomputation like our similarity computation, then store the results in a SQL table and use those for rendering the home page.

vipulnaik commented 5 years ago

@riceissa for "Amount rank" I prefer if the rank is inverted, i.e., the biggest donation gets rank 1. That is more consistent with naive notions of rank.

vipulnaik commented 5 years ago

@riceissa your linked commit above only fixes the donor donation list page. Can you make similar fixes for other donation list pages, such as donee donation list and donor-donee donation list?

vipulnaik commented 5 years ago

@riceissa in cases where there is a cause area filter, can you provide ranks relative to donations that match the filter (rather than relative to all donations)? If there are significant technical complexities to doing so, please discuss.

riceissa commented 5 years ago

@riceissa your linked commit above only fixes the donor donation list page. Can you make similar fixes for other donation list pages, such as donee donation list and donor-donee donation list?

Done. I actually hadn't added the amount rank for donor-donee pages, so I also did that.

riceissa commented 5 years ago

@riceissa in cases where there is a cause area filter, can you provide ranks relative to donations that match the filter (rather than relative to all donations)? If there are significant technical complexities to doing so, please discuss.

This is also done.