search

gratipay / gratipay.com

Here lieth a pioneer in open source sustainability. RIP
https://gratipay.news/the-end-cbfba8f50981
MIT License
1.12k stars 308 forks source link

implement funds #449

Closed chadwhitacre closed 11 years ago

chadwhitacre commented 11 years ago

The idea here it to provide a way to lump individuals together into groups and projects for funding purposes. A "fund" here is a set of weighted percentages for individuals and other funds (funds can be nested). Then we use these funds as levels of indirection when shuffling money around each week on payday.

Was: group tips together for simultaneous adjustment

I believe this is what @joelmccracken is asking for on #434. The coarse-grained case would be adjusting all tips at once (#448). This ticket is for grouping tips together so that you could adjust 10 tips at once without touching the other 20. The language that Joel used was "a single tip covering multiple people."

carsomyr commented 11 years ago

@MikeFair I think your descriptions of to_me and from_me map perfectly onto the payout percentages and initial user allocations I was talking about. Is this what you had in mind?

  1. For my personal payout vector v, if I'm the ith entity, then v[i] corresponds to the percentage I keep in the to_me fund before forwarding on the funds.
  2. The initial user purse is actually the from_me fund, all of which is distributed in the first round. I'll need to adjust my model to normalize for that initial payout round.
  3. Are the to_me and from_me payout proportions the same? Are we going to allow the users to tip in a different proportion for money they receive at tips? I am guessing "no".
MikeFair commented 11 years ago

I think your descriptions of to_me and from_me map perfectly onto the payout percentages and initial user allocations I was talking about. Is this what you had in mind?

Yes.
It's also creating complete separation between a participant and a resource allocator (a fund). A participant has many resource allocators that will get used in different contexts. A participant has multiple resource pools (custodians/banks who actually hold the assets) and they also have multiple allocators. Allocators are a function that describe how to move resources between the various resource pools (banks of resources).

The resources aren't ever actually coming to or from the participant directly, they are going from one of the participant's resource pools into other compatible resource pools and using one of the participant's allocation functions to determine how to do apply those movements.

I'm thinking the from_me and to_me funds are simply the first of these allocation functions.

The initial user purse is actually the from_me fund, all of which is distributed in the first round. I'll need to adjust my model to normalize for that initial payout round.

Like you're seeing, from_me contains percentages just like every other fund out there. But it's likely you won't actually need to change the algorithm.

I'm considering that the initial top level fund could be the list of all participants and their contributed balances, creating one giant GitTip Fund at the very top.

So the top level fund that would get input into the function would be: P1_from_me: P1_balance_amount , P2_from_me: P2_balance_amount , ...

Each participant would get a distinct row, there'd be no duplicates at the top level, and your matrix should just do the right thing with it.

Are the to_me and from_me payout proportions the same? Are we going to allow the users to tip in a different proportion for money they receive at tips?

The from_me and to_me funds have nothing to do with each other and are only connected in the sense that they are controlled by the same participant. So yes they can absolutely be different.

The two funds will have different identifiers. Think of the fund's IDs as being "P1.from_me" and "P1.to_me" whereas before we've just been putting "P1". As far as the matrices are concerned, they two funds are as different as A6 and Z10. There's nothing special you'll need to do.

"P1.balance" will be the collection pool for P1's allocations. This makes generating that top level "GitTip" fund easy because it's just a list of participant's and their balances.

We might want to keep track of the changes in the balance amounts to each participant, but I don't see that as necessary at this time.

chadwhitacre commented 11 years ago

I have totally not digested this thread since my last post to it, but I did want to record the thought that we might allow funds to be public iff they only contain other funds. So ... all funds are private by default, and funds that only contain other funds can opt-in to being public. If you add a participant to it, it becomes private again, and you have to re-opt-in if and when you trim it back to funds only. (Or maybe that's automatic: funds of funds are public. If you want it private, add an individual to it.)

MikeFair commented 11 years ago

@whit537 Like we said before, the public/private debate is somewhat orthogonal to the implementation.

I see the need for both but I believe for very different reasons than you've put forth so far. In any case, I see the use cases for a permissions framework over who has access to see, edit, and use funds (read, write, execute).
This framework makes privacy a configuration issue, not an implementation issue.

Once we actually have working code to have a discussion about; where we can actually see it in practice; that's when I think we'll get traction in being able to resolve the privacy issues more succinctly

I opened issue #594 so the privacy conversation can be had there instead.

MikeFair commented 11 years ago

@carsomyr What progress?

carsomyr commented 11 years ago

@MikeFair Sorry, I've been caught up with work stuff all last week. I'll create a fork of the project on Tuesday and try to get a demonstrable sparse array implementation of the tipping algorithm.

MikeFair commented 11 years ago

Have fun with it! I've been doing some discovery on the least intrusive way to create a one participant->many funds framework and thinking about the first draft of the UI.

My thoughts for the moment are that a participant will have a changeable default fund, settable on their main profile page. Then as they tip people, this will add that person to whatever the current default fund is.

On their main profile page they can have more control over the construction of all these funds. Able to change the weights away from the default predetermined tipping amounts.

This way very little UI code has to change and it's pretty much all concentrated into one the main profile screen.

Table wise, I've added one table called "collections" which maps one entity_id to another fund_id (both entity and fund are foreign keys to the participant table for the moment), with a string array called "privileges". The string array contains elements of 'read', 'write', and/or 'execute'. I'm sure I'll change that part before I'm done. I wanted to assume that we don't know in advance all the privileges we might need on these things, or even that gross top level privileges are all we need. But rather than actually figure everything out, but in some flexibility but focus on existing known needs.

Talk again soon. On Feb 4, 2013 1:25 AM, "Roy Liu" notifications@github.com wrote:

@MikeFair https://github.com/MikeFair Sorry, I've been caught up with work stuff all last week. I'll create a fork of the project on Tuesday and try to get a demonstrable sparse array implementation of the tipping algorithm.

— Reply to this email directly or view it on GitHubhttps://github.com/zetaweb/www.gittip.com/issues/449#issuecomment-13068006.

chadwhitacre commented 11 years ago

Had a great conversation with @carsomyr today about this. What we realized is that we can use the infinite sum to mix funds into the existing payday algorithm, like so:

As it stands today, this is where we wrap up: the only entities that have money in the pending column at this point are participants, so we simply clear pending to balance in a SQL transaction and then move on to payouts. But now we're going to add funds as an intermediary, so at this point both participants and funds will have money assigned to them in the pending column. This is the cool, new part: we use the infinite sum function to compute the ratio in which to transfer money from funds to participants. After applying the results of that function, we have a pending column that has zeros in it for funds and varied amounts in it for participants. After we allocate money from funds in this manner, we clear pending to balance and move on to payouts as usual.

Damn elegant, gentlemen.

!m @MikeFair

!m @carsomyr

chadwhitacre commented 11 years ago

photo-1

chadwhitacre commented 11 years ago

photo

MikeFair commented 11 years ago

@carsomyr, @whit537

There's an even easier way to bypass the need for that second pass through the funds and simply use the infinite sum algorithm directly in the first place; you just have to view the whole of GitTip's database as one big giant fund.

Step 1) Loop over all participants Pull money amounts required to cover tips in to balance (may fail) End Loop At this point only participants will have balances and all their pending amounts will be zero. Balances may end up being more or less than the full amount they have allocated in their total list of tippees.

Step 2) Query all participants that have non-zero balances and positive tippees as one big fund.
Since a balance might be higher or lower than the aggregate of all their tips, query out minimum between their balance and the sum of their tip amounts (or test in the code to make sure that the code doesn't allocate more than the sum of their tips).

select participant, least(balance, (select sum(amount) from tips where tipper = p.id)) from participants p

Think of this query as if it's creating one big fund of everyone, then the infinite sum matrix allocation algorithm can be directly applied to this master nameless fund.

The equivalent would be looping over every participant and treating each participant's set of tips as if it was a single fund; but this does it all in parallel at once.

At the end of step 2 here; participants will have their pending column filled.

Here's an example assuming: Mike tips total $12 Chad tips total $20 Roy tips total $50

Each respective participant can be thought of as a single fund with their tippees as the constituents.

Roy's infinite sum algorithm is expecting dollar amounts as the top level input. So if you did it the loop way, you could simply apply the infinite sum algorithm to each participant, one at a time.

1) Mike $12 -> [infinite sum] -> resulting participant's pending 2) Chad $20 -> [infinite sum] -> resulting participant's pending 3) Roy $50 -> [infinite sum] -> resulting participant's pending

However when you see Mike, Chad, and Roy as one giant nameless fund of everything, then you can do this:

$12 - Mike Fund | $20 - Chad Fund } - [infinite sum] - { resulting participant's pending } $50 - Roy Fund |

This is the exactly the same as treating the fund's pending column as its contribution amount in the original second pass through phase, but it backs the algorithm up to start two levels higher and does it one pass.

MikeFair commented 11 years ago

@carsomyr what news?

chadwhitacre commented 11 years ago

Here's what we ended up with in MATLAB:

payout = [0 .8 .2  0 0;
          0  1  0  0 0;
          0 .9  0 .1 0;
          0  0  0  1 0; 
         .1 .5 .2 .2 0];

initial = [10000 10 0 200 0]';

payout_d = diag(diag(payout));

n_rounds = 3;
if n_rounds == 0
    res = initial' * inv(eye(size(payout, 1)) - (payout - payout_d)) * payout_d;
else
    acc1 = zeros(size(payout, 1));
    acc2 = eye(size(payout, 1));

    for i = 1:n_rounds,
        acc1 = acc1+acc2;
        acc2 = acc1*(payout - payout_d);
    end

    acc1 = acc1 * payout_d + acc2;

    res = initial' * acc1;
end
carsomyr commented 11 years ago

@whit537 For starters, n_round should be n_rounds.

MikeFair commented 11 years ago

Is the plan to have python write out to a file for Octave to process so we can reuse this code? Or is it to use Python's matrix processing utilities? On Feb 22, 2013 9:57 PM, "Roy Liu" notifications@github.com wrote:

@whit537 https://github.com/whit537 For starters, n_round should be n_rounds.

— Reply to this email directly or view it on GitHubhttps://github.com/zetaweb/www.gittip.com/issues/449#issuecomment-13985833.

carsomyr commented 11 years ago

@MikeFair My plan is to use SciPy and do the calculations in-process. There's nothing magical about Octave or MATLAB: One just needs to do a bit more work in Python to get things working.

chadwhitacre commented 11 years ago

@carsomyr If you're not aware of it already, be sure to check out pandas: http://pandas.pydata.org/.

carsomyr commented 11 years ago

@whit537 @MikeFair An initial go at the fund payout module can be found at 2589bd9. Some notes:

  1. I haven't done the matrix inverse version, which represents the exact answer. Perhaps a sparse matrix solver can be employed.
  2. The module is self-contained and can be run as a script. Just type python -- steady_state.py.
  3. You will need SciPy to run it. Unfortunately, satisfying SciPy's dependencies is kind of hard. Do consider getting MacPorts and typing sudo -- port install py27-scipy.
chadwhitacre commented 11 years ago

Thanks Roy. Two immediate questions, while I install SciPy:

carsomyr commented 11 years ago

@whit537 It's on my fork. How GitHub made the association, I don't know. I'll be changing the license shortly. What should the header look like?

chadwhitacre commented 11 years ago

Interesting. It's also available here: https://github.com/carsomyr/www.gittip.com/commit/2589bd9, namely:

https://github.com/carsomyr/www.gittip.com/commit/2589bd9

We haven't been enforcing copyright disclaimers on individual files, on the understanding that the global COPYING file covers all files in the repo. Are you comfortable with that?

chadwhitacre commented 11 years ago

I was able to install SciPy like so:

Success:

[gittip] $ python steady_state.py 
[[ 0.   0.8  0.2  0.   0. ]
 [ 0.   1.   0.   0.   0. ]
 [ 0.   0.9  0.   0.1  0. ]
 [ 0.   0.   0.   1.   0. ]
 [ 0.1  0.5  0.2  0.2  0. ]]
[[ 100.    0.  100.    0.  100.]]
[[   0.   265.8    0.    34.2    0. ]]
[gittip] $
carsomyr commented 11 years ago

@whit537 Sanity check: If you make every entity a fund, the script will blow up.

chadwhitacre commented 11 years ago

Since SHAs are universal, I wonder if you can reference any commit in the context of any repo on GitHub?

Answer: No, only on forks. Here's random one from rails: 61b91c4c55bcbd5a2ec85d6e1c67755150653dff (original: https://github.com/rails/rails/commit/61b91c4c55bcbd5a2ec85d6e1c67755150653dff).

Here's a random one from another Gittip fork: 0c1b33ca2534d96c73c26a0b38819dddaa9fdcff

chadwhitacre commented 11 years ago

Confirmed. :-)

[gittip] $ python steady_state.py 
[[ 0.   0.8  0.2  0.   0. ]
 [ 0.1  0.   0.2  0.3  0.4]
 [ 0.   0.9  0.   0.1  0. ]
 [ 0.6  0.3  0.1  0.   0  ]
 [ 0.1  0.5  0.2  0.2  0. ]]
[[ 100.    0.  100.    0.  100.]]
Traceback (most recent call last):
  File "steady_state.py", line 101, in <module>
    sys.exit(main())
  File "steady_state.py", line 96, in main
    print(initial * SteadyState.converge(payouts).todense())
  File "steady_state.py", line 77, in converge
    raise RuntimeError("The payout matrix failed to converge")
RuntimeError: The payout matrix failed to converge
[gittip] $
chadwhitacre commented 11 years ago

This is great, man. Thank you. May I add you as a collaborator on the Zetaweb team now?

https://www.gittip.com/on/github/zetaweb/

As mentioned at OpenHack tonight I'd like to move payday to a separate repo, but for now I say we take a pull request from you for this repo once you modify the license.

chadwhitacre commented 11 years ago

BTW, I'm more than happy to keep the __author__ attribution in there.

chadwhitacre commented 11 years ago

Outta steam, will check back tomorrow.

!m carsomyr

carsomyr commented 11 years ago

@whit537 Sure, add me to the Gittip organization. I also dedicated the steady state payday library to the public domain in b8e9784. Next up: Trying to solve for the exact solution using the spsolve method.

chadwhitacre commented 11 years ago

+1 from @adelevie via Twitter.

chadwhitacre commented 11 years ago
chadwhitacre commented 11 years ago

Softest launch ever:

https://www.gittip.com/for/gittip/

Neglected to merge the branch. :)

chadwhitacre commented 11 years ago

Okay, deployed:

https://www.gittip.com/for/gittip/

:)

chadwhitacre commented 11 years ago

Convo re: soft launch here:

https://botbot.me/freenode/gittip/msg/2681474/

chadwhitacre commented 11 years ago

This shit is released, people! Will reticket ... everything else I didn't do yet. :)

mvdkleijn commented 11 years ago

Sorry for commenting on a closed issue... I'm confused. How are funds implemented? Or are Teams supposed to be funds? I don't see options to donate to communities and I don't see funds in the UI anywhere. The closest thing I can find is Teams?

chadwhitacre commented 11 years ago

@mvdkleijn Right, funds ended up morphing into Teams (via "open groups"). Down the line we can revisit the idea of funds as originally described on this ticket, but I don't want to get into that right now. Let's stay focused on cleaning up what already exists.

chadwhitacre commented 11 years ago

This took a left turn into Teams. Reviving the original proposal as #1493.