Bring back takes for Team Gratipay

[chadwhitacre]

This is the main ticket for the "Bring Back Take-What-You-Want for Team Gratipay" milestone.

Done

This ticket/milestone is done when we successfully distribute Gratipay Team revenue to ~users as part of payday. The six five ~users we are targeting for the first distribution are: @kaguillera @rohitpaulk @mattbk @aandis @clone1018 @whit537.

Steps

National Identity

• [x] deploy schema and Python for storing national identity for ~users: #3976
  • [x] implement symmetric encryption: #3998
  • [x] add countries: #3999 → #4027
  • [x] implement encrypted national identities: #4000 → #4028
  • [x] [un]verifying identities: #4001 → #4031
  • [x] clearing identities: #4002 → #4032
  • [x] has_verified_identity: #4003 → #4033
• [x] implement fake data for participant identities: #4006
• [x] deploy UI for publicly displaying country info of ~users: #4007
• [x] deploy UI for storing national identity for ~users: #4008 → #4034
• [x] 👾 bonus PRs: #4035 #4036

  Takes

• [x] make room for team changes: #4037
• [x] 👾 convert client-side tests to Splinter: #4079
• [x] make more room for team changes 😁 : #4068 #4070
• [x] bring back the old UI behind a flag: #4072
• [x] start fixing the UI: #4073
• [x] migrate _mixin_team.py: #4074
• [x] cherry-pick #4023: #4075
• [x] cherry-pick #4024: #4076
• [x] update some Python APIs: #4099
• [x] bring back the takes UI: #4077
• [x] 👾 bonus PRs: #4108 #4109 #4110
• [x] bring back payday: #4102
• [x] include team members in MassPay: #4111
• [x] review computed values algorithms → reticketed #4120
• [x] review docs for new mixins → reticketed #4121
• [x] fill out the test suite for the mixins, prune old tests → reticketed #4122

  Finish!

• [x] use UI to add identities
• [x] use Python API to verify identities
• [x] decide on https://github.com/gratipay/inside.gratipay.com/issues/747
• [x] use SQL to set available for Team Gratipay
• [x] use UI to add members to Team Gratipay
• [x] use UI to set takes
• [x] run payday!

[chadwhitacre]

Alright, I've laid out a plan in the description. We're off and running! 💃

[chadwhitacre]

@kaguillera and I are planning to work on this today.

[chadwhitacre]

@aandis You around to help with review? I anticipate that we're gonna spit out a bunch of PRs today. 💃

[kaguillera]

[aandis]

Sorry, I am caught up with my college exams for a week. :disappointed: Will be here after that. Excited this is happening though:grey_exclamation: :tada: :dancer:

[chadwhitacre]

Sorry, I am caught up with my college exams for a week. :disappointed:

No worries, I thought that might be the case. Do well on your exams! ⚾️ 🔋

[chadwhitacre]

As I'm working through https://github.com/gratipay/gratipay.com/pull/4005 on my way to UI for storing and modifying participant identities, I'm really feeling like we want to keep a lid on this. I think we want to throttle who we accept identity information from, because it's highly likely that we'll need to make changes to this system as we start to gain real-world experience with it. The more people who have entered their identity info, the more friction we'll have against changes. Security researchers will surely hit it (which we want), but it'll be hard to sort out researchers from non-researchers. We could discard data for researchers, but that looks amateur if we discard real users' data.

I think at the very least we should avoid putting the identity form in the nav. That should prevent most folks from finding it and therefore filling it out. A step further would be to put a nonce in front of it so that only people with a unique link could access it—i.e., people we invite (nonces would be single-use and tied to participants).

[chadwhitacre]

Basically, once we have a few identities in there, the next step down this path will be to start exploring programmatic identity verification (#2449) in earnest. That's a big undertaking for us, and I will be very surprised if we don't need to reenter identity info based on whatever system we end up implementing.

But! #2449 is step 4.i on our product roadmap/priorities ... and we're still on step 1. As deep as my head is in thinking about payroll right now while working on this ticket, I want to remember that the plan is to set this all aside once we've got payroll back for Team Gratipay. We aren't going to get back to generalizing payroll until after we've a) finished cleaning up after the Gratipocalypse, and b) hit payments out of the park. Realistically we're looking at 2017, given that we also have other Radars besides Product to keep up with—Tech and Accounting loom large, per "Making it Right," as well as Security.

Which is all to say that I think we should limit entering of identity information to just those of us who are going to be on Team Gratipay itself. It's probably enough to keep it out of the nav. Anyone who finds the form by reading our source code should probably be on the Team anyway. :-)

[chadwhitacre]

by reading our source code

... or reading this thread. 😚

[chadwhitacre]

next steps:

• [x] add filtering on is_verified to list_identity_metadata: https://github.com/gratipay/gratipay.com/pull/4001#issuecomment-216038499
• [x] propagate the change through subsequent PRs under #3976 and #4005
• [x] propagate rebases of #3998 do for each PR as it comes on deck
• [x] salvage https://github.com/gratipay/gratipay.com/pull/4005 for parts
  • [x] factor out fake_data clean-up and additions (maybe also fix this fake_data bug with missing elsewheres?) → #4006
• [x] insert a PR for has_verified_email (https://github.com/gratipay/gratipay.com/pull/3976#issuecomment-216093098) moved this to a punchlist on #4000

[chadwhitacre]

ttfn

[chadwhitacre]

Worked through the timeline for this over bad 🍻 with @hurlothrumbo. Basically, we need to front-load the work.

Step next is merging https://github.com/gratipay/gratipay.com/pull/3998. It's appropriate to pause for a bit of review before landing that, since it's the heart of our crypto implementation. I think it would be best for @rohitpaulk or possibly @aandis to merge that one once we're ready, given its weighty nature.

Step N+1 is to slam out the rest of the work, which at this point looks like will mostly fall on my shoulders to do—but! I need a reviewer to help me keep the pipeline flowing. @kaguillera and I are expecting to cowork on Wednesday and Thursday as usual (yes?). My goal is to get as many PRs as possible lined up before Wednesday noon, sooo ...

Wednesday and Thursday @kaguillera and I can work through the review queue. That said, I'm not going to hesitate to keep the PRs moving, with or without @kaguillera. We need to get this done!

On Friday we'll come up for air. I'll have to put in time for COS. It's also Mother's Day weekend over here. But, basically whatever is left to do, I'll do.

My goal is to have all of the code deployed by Sunday night.

That gives us Monday through Wednesday to shake out the bugs and sign us all up!

Thursday we run payroll for the first time since the Gratipocalypse ... and then we celebrate!

Who's ready? You can help by reviewing PRs before and as they come up, also rebasing downstream ones as we land the one on deck. Let's do this!

[chadwhitacre]

You can help by reviewing PRs before and as they come up, also rebasing downstream ones as we land the one on deck.

... and of course by taking the PR on deck, reviewing, merging, and deploying it!

[chadwhitacre]

Hey Peter, hope tax season went well for you. :-)

We are making a push to get our payroll product back online—for our own use only at this point. We are aiming to have six of us for the initial rollout. U.S. and India we know how to handle. The third we need for launch is Trinidad & Tobago. When you have a chance, could you look into the tax treaty there?

Secondly: are we okay to use the term "payroll" for this product, even though we're primarily talking about independent contractors? Is payroll general enough to include contractors, or is it specific to employees?

Thanks, Peter!

[chadwhitacre]

Alright, I'm cutting invites out of the scope here. We don't need that for the six of us to get paid by Gratipay, we can add us manually.

Cut/altered these lines in the description:

• [ ] add schema and Python for tracking Team permission to invite ~users: #4010

Invitations

• [ ] add schema and Python for tracking invitations to join a Team: #4015
• [ ] deploy UI for accepting an invitation to join a Team
• [ ] deploy UI for inviting a ~user to join a Team (sends email)

Finish!

• [ ] use UI to invite ~users to become members of Team Gratipay
• [ ] use UI to accept invitations to become members of Team Gratipay

[chadwhitacre]

So ... what's that? Six open PRs under #3976, plus eight others on here—we've got 14 open PRs between us and our first Payroll Take-What-You-Want 2.0 next Thursday. Only 14 PRs! 😀 🔔 🔨

[chadwhitacre]

... and some of the PRs are even mostly written! 💃

[chadwhitacre]

Payroll is specifically used for employees. You should avoid that term for contractors.

The treatment for Trinidad & Tobago is the same as India. Contractors receiving income there, so long as they do not spend longer than 3 months in the US, then treatment is the same.

[kaguillera]

😱

[chadwhitacre]

Alright! Been deep-diving on crypto (#3998) the past few days, and we're not outta the woods yet. I'm gonna go ahead and trim another PR from this milestone:

• [ ] deploy UI for modifying Team takes: #4014
• [ ] use UI to set takes

At this point I will be thrilled if we can move six five (@kaguillera is out pending conversation with a lawyer about his work situation) pennies next Thursday! ¢¢¢¢¢

[chadwhitacre]

Picking up from https://github.com/gratipay/inside.gratipay.com/issues/325#issuecomment-217335638 ...

One of the main problems with the take-what-you-want pilot was that specifying absolute dollar amounts made for busy work keeping up with fluctuations in money available to share. In true fashion, @pjz wrote a small bash script (okay—Python, actually ;) to solve the problem that "it's difficult for me to express something like 'take an even percentage that maxes out at $X.'" @YesCT and @alexpott reported that under DrupalCoreGittipTeam's highly variable income fluctuation, absolute share values were a significant source of friction.

We have a chance here to revise the rules. What are we gonna doooo?

[chadwhitacre]

Proposal

1. Every Team has 1,000 "takes."
2. Takes are kinda like shares—except that they don't indicate legal ownership.
3. The weekly distribution algorithm works like this:
   1. A manager-set "baseline," an absolute dollar amount, comes out first and goes to the ~owner.
   2. Anything above the baseline is split amongst Team members, prorated according to takes.
   3. Any unclaimed takes also go to the ~owner.
4. Any member can claim unclaimed takes at any time.
5. Only you can set your take (still).
6. Managers (anyone!?) can add people at one take.
   1. Requires that at least one take be unclaimed.
   2. You can always drop a take, in order to give it to someone else. :)
7. Managers can remove people.

Benefits

Benefits relative to TWYW 1.0:

1. Ratio rather than absolute means less friction adapting to variation in funding flows.
2. Ratio rather than absolute means it doesn't matter who "eats first" (practically speaking, we would go from smallest takes to largest, so that rounding errors accumulate in the larger end—whatever we do in MassPay; I think that's it).
3. The baseline guarantees that the Team bank account gets what it needs for ops and capital and even baseline salary for the owner(s). Each Team ~owner can decide for themselves how high to set this. I have in mind https://github.com/gratipay/team-review/issues/11#issuecomment-130494815 and https://github.com/gratipay/gratipay.com/issues/3399#issuecomment-112186121.
4. There is no throttling. Under 1.0, we had to throttle to prevent newcomers from taking everything. With this proposal the most a newcomer could take is the surplus for that week.
5. We still get takes as information about Team social balance. When takes get out of balance, the Team needs to talk, and then takes will rebalance.

[chadwhitacre]

1,000 takes means that 1 take is 0.1%, or $10 at $10,000 per week.

I like the simplicity of "1000 takes." We'd have to keep an eye on where the pressure points show up when Teams get big. Maybe we end up needing a facility for doing take splits, or issuing more takes.

[chadwhitacre]

I think this would largely address @traverseda's suggestion that we have "a bunch of different methods for managing team payments, including what [we] have now and completely central management by the team owner."

• A fully closed Team would simply set their baseline at infinity: all revenue would go to the ~owner, to pay salaries or whatever out-of-band (though Gratipay could offer that on-site, too; eep).
• A fully open Team would set their baseline to zero: all revenue would go to the members.

We'd expect a spectrum to emerge from "closed" to "open," with most Teams ending up somewhere in the middle.

Put another way: this seems to move us in the direction of decoupling payments and payroll. Hmmm ...

[chadwhitacre]

Alright, redid stub PRs around takes. Brought back the final UI PR because dang it I want to show it off in France! 🇫🇷

[Changaco]

Interesting proposal @whit537. FYI this is how Liberapay Teams currently work:

• no owners
• members set their own takes, obviously
• we call those user-input numbers "nominal takes", and the amounts of money users really receive are called "actual takes"
• nominal takes are "absolute" amounts except that they act as shares when the team's income isn't enough to cover all the takes
  • in other words, nominal takes are maximums, there's no "kids eat first"
  • this system allows setting aside spare income that new team members can take without eating into the income of existing members, and stabilizes the income of members in general
  • it also avoids having to ask everyone to lower their takes when income decreases, which was one of the problems the Drupal team ran into
  • the downside is that every member has to up their take when income increases (payday email notifications kinda act as reminders to do that)
• takes are throttled, the algorithm is more flexible than the old Gratipay throttling, but still not flexible enough (https://github.com/liberapay/liberapay.com/issues/287)
• any member can invite more people

[chadwhitacre]

Thanks for the explanation, @Changaco. I've poked around at Liberpay teams a little bit, but apparently not that much, because, while I did notice the nominal vs. actual takes, I hadn't found the documentation. We are going to have some interesting conversations in a few weeks, I think. :)

[chadwhitacre]

From https://github.com/liberapay/liberapay.com/issues/287#issuecomment-217378424:

the downside is that actual take amounts would not be stable, they'd go up and down as income changes.

We could resolve this with a "cap" as well as a "baseline." While income stays above the cap, there'd be stable distribution week to week. Excess could go to the ~owner, or be distributed to members, framed as a "bonus" (this latter could work for Liberapay as well).

I almost included a cap in my proposal, but thought it added too much complexity for little benefit since an ~owner can always up the baseline as a workaround (it's just that this isn't automated, i.e., a parallel problem to absolute takes).

If distributing supra-cap income to members, on the other hand, I think psychologically it'd be hard for members to distinguish between "baseline" and "bonus" from week to week. I expect it'd still feel like a single, variable sum.

[chadwhitacre]

this system allows setting aside spare income

@Changaco Who owns this money, the spare income retained within Liberapay?

[chadwhitacre]

to distinguish between "baseline" and "bonus" from week to week

Or maybe the bonuses are tracked and distributed quarterly or annually. Dipping into any accumulated float in sub-cap weeks would dampen ~member income variability.

[chadwhitacre]

Or maybe the bonuses are tracked and distributed quarterly or annually. Dipping into any accumulated float in sub-cap weeks would dampen ~member income variability.

But we don't need to land that before I go to France. :-)

[Changaco]

The "spare income" belongs to the donors and stays in their wallets. Barring any intentional or unintentional change by the donors to the amount of the team's income, the extra money is there again the following week should a team member wish to take it.

One thing I didn't mention above is that Liberapay teams are not legal entities, thus they cannot hold money, and the donations go directly from the team's donors to its members.

[chadwhitacre]

Liberapay teams are not legal entities

Right, saw that in the docs: "Teams are for peer-to-peer donations, they can't be used by a legal entity to distribute donations to its member."

[Changaco]

Well, we say "can't", but actually it's more "you probably shouldn't", and in fact we kinda have one team doing it right now as an experiment.

[chadwhitacre]

Alright, I've got the PRs under #3976 topped up and ready to go. I would reeeeaaalllllyy love someone besides @kaguillera or me to merge #3998. Of course it'd be better for @rohitpaulk @aandis to merge subsequent PRs as well, but apart from the crypto one I'm comfortable with @kaguillera and me merging them together on Wednesday. That means I've got today and tomorrow to get the eight remaining PRs ready to go. Onward! 🗻

[chadwhitacre]

Okay! I finished a first pass on #4008, and #4009 is trivial. That means we only have four more PRs to rough in! Schedule:

• Tuesday
  • Rough in four remaining PRs. 😬
  • Land #3998?!
  • Publish gratipay/inside.gratipay.com#551.
• Wednesday
  • Land thirteen remaining PRs. 😵
  • Put first identity info into the system!
• Thursday
  • Put a little more identity info into the system.
  • Use the Python API to turn on takes for Team Gratipay.
  • Set our takes!
  • Run payday!
  • Celebrate! 💃
  • Crash. 😴

!m *

[chadwhitacre]

Okay! One down, thirteen to go. 🌺

[chadwhitacre]

Every Team has 1,000 "takes."

Drupal had 200 people eligible to participate. Assume 20% of the people do 80% of the work. The minimum number of takes is 1, so let's say that's 160 takes for the long tail. Is 840 takes enough to split between the remaining 40?

I had thought that it might not be, that we'd need need to offer 10,000 takes out of the gate. Now that I work it through, though, I think 1,000 takes could be enough for even a mature project such as Drupal. We may end up saying that when you run out of takes, it's time to split the Team itself.

[chadwhitacre]

Alright. Today is the day.

[chadwhitacre]

This is the best and cleanest I think we've ever done a piece of work. There's definitely friction in holding a dozen+ PRs open at once (I landed in the weeds last night and just spent the first hour this morning mucking about in the reflog), but overall I think we're ending up with something much stronger. I love documenting the Python API as we go along, and it feels really right to be able to go back and fix API in earlier PRs based on the realities that arise in later ones. 👍

[chadwhitacre]

I'm learning to prefix git push --force with make pytest &&. ;-)

[chadwhitacre]

... because one of the problems with the way this is working is that Travis is not testing the PRs that are made against something other than master. Do we have to choose between that, and duplicate checks for PRs against master (#3187)?

[chadwhitacre]

Okay! PR chain cleaned up and pushed up. The two PRs at the end still need to be roughed in, and then we can start going over the whole thing from the top, reviewing and merging. My goal is to get the two roughed in this morning before meeting @kaguillera at Blue Canary. Hopefully he and I can start merging things this afternoon. Whatever we don't get done, I'll try to land tonight and tomorrow early. I'd like to at least get the identity work up through #4008 deployed today so that we all have at least a little time to enter our deets(!). Once deets are in I can verify identity and add yinz to the team without further input from you. You'll just be at one take tomorrow if you're not able to update your takes before payday. :)

Oooh ... we also need to implement the baseline https://github.com/gratipay/gratipay.com/issues/3994#issuecomment-217340995 ...

[chadwhitacre]

Alright, I'm half-way through #4025. It's going well! @kaguillera just arrived at my place. We're going to work here instead of Blue Canary to minimize distraction.

[chadwhitacre]

Okay! We've got #4023 roughed in. We're going to leave #4026 for later (tonight? tomorrow early?) and focus for the rest of the afternoon on reviewing, merging, and deploying the identity work.

🐫 🍰 🐪 🌵

[chadwhitacre]

Back from supper.

[chadwhitacre]

Dang. Alright, folks. We have national identity numbers in Gratipay. 😯 🙈 🙉 🙊

[chadwhitacre]

May as well verify myself. :)

[gratipay] $ heroku run python
Running python on gratipay... up, run.5584
Python 2.7.9 (default, Dec 11 2014, 17:18:51) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from gratipay import wireup
>>> env = wireup.env()
>>> db = wireup.db(env)
>>> from gratipay.models.participant import Participant
>>> w = Participant.from_username('whit537')
>>> from gratipay.models.country import Country
>>> US = Country.from_code('US')
>>> w.set_identity_verification(US, True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "gratipay/models/participant/mixins/identity.py", line 208, in set_identity_verification
    """, dict(locals(), participant_id=self.id))
  File "/app/.heroku/python/lib/python2.7/site-packages/postgres/cursors.py", line 105, in one
    out = self._some(sql, parameters, lo=0, hi=1)
  File "/app/.heroku/python/lib/python2.7/site-packages/postgres/cursors.py", line 127, in _some
    self.execute(sql, parameters)
  File "/app/.heroku/python/lib/python2.7/site-packages/psycopg2/extras.py", line 288, in execute
    return super(NamedTupleCursor, self).execute(query, vars)
psycopg2.ProgrammingError: can't adapt type 'Country'
>>> w.set_identity_verification(US.id, True)
>>>

[chadwhitacre]

Looks like we've got another code2 mole to whack ...

[chadwhitacre]

👍