Bring back takes for Team Gratipay

[chadwhitacre]

This is the main ticket for the "Bring Back Take-What-You-Want for Team Gratipay" milestone.

Done

This ticket/milestone is done when we successfully distribute Gratipay Team revenue to ~users as part of payday. The six five ~users we are targeting for the first distribution are: @kaguillera @rohitpaulk @mattbk @aandis @clone1018 @whit537.

Steps

National Identity

• [x] deploy schema and Python for storing national identity for ~users: #3976
  • [x] implement symmetric encryption: #3998
  • [x] add countries: #3999 → #4027
  • [x] implement encrypted national identities: #4000 → #4028
  • [x] [un]verifying identities: #4001 → #4031
  • [x] clearing identities: #4002 → #4032
  • [x] has_verified_identity: #4003 → #4033
• [x] implement fake data for participant identities: #4006
• [x] deploy UI for publicly displaying country info of ~users: #4007
• [x] deploy UI for storing national identity for ~users: #4008 → #4034
• [x] 👾 bonus PRs: #4035 #4036

  Takes

• [x] make room for team changes: #4037
• [x] 👾 convert client-side tests to Splinter: #4079
• [x] make more room for team changes 😁 : #4068 #4070
• [x] bring back the old UI behind a flag: #4072
• [x] start fixing the UI: #4073
• [x] migrate _mixin_team.py: #4074
• [x] cherry-pick #4023: #4075
• [x] cherry-pick #4024: #4076
• [x] update some Python APIs: #4099
• [x] bring back the takes UI: #4077
• [x] 👾 bonus PRs: #4108 #4109 #4110
• [x] bring back payday: #4102
• [x] include team members in MassPay: #4111
• [x] review computed values algorithms → reticketed #4120
• [x] review docs for new mixins → reticketed #4121
• [x] fill out the test suite for the mixins, prune old tests → reticketed #4122

  Finish!

• [x] use UI to add identities
• [x] use Python API to verify identities
• [x] decide on https://github.com/gratipay/inside.gratipay.com/issues/747
• [x] use SQL to set available for Team Gratipay
• [x] use UI to add members to Team Gratipay
• [x] use UI to set takes
• [x] run payday!

[chadwhitacre]

I modified my identity now that address 2 is optional ... and of course it reset to unverified. :)

Iterating ...

#!/usr/bin/env python2
from __future__ import absolute_import, division, print_function, unicode_literals

from gratipay import wireup
from gratipay.models.country import Country
from gratipay.models.participant import Participant

env = wireup.env()
db = wireup.db(env)

p = Participant.from_username('whit537')
c_id = Country.from_code('US').id
p.set_identity_verification(c_id, True)

[chadwhitacre]

Really coming down to the wire to get this in before https://github.com/gratipay/inside.gratipay.com/issues/618. We don't need the UI for that. Gratipay can issue 5 takes and set a high-ish baseline and we should be fine with equal splits for now (1 take each). That means we need to get through #4037 #4023 #4024 and #4025 in order to make https://github.com/gratipay/inside.gratipay.com/issues/618.

[chadwhitacre]

I'm working on writing update_taking as a PL/pgSQL function, because I don't see how to write it in Python without too many database calls.

[chadwhitacre]

Kinda waiting for @kaguillera to get here. I think we can get this done in two or three hours. Not sure if that's too late for https://github.com/gratipay/inside.gratipay.com/issues/618. :-(

[chadwhitacre]

Well, "this" won't be "done", but if we can get through #4025 that should be enough for today's payday.

[chadwhitacre]

Okay! @kaguillera is here. :)

Looks like we can do it in Python and write back to the database once per team with this mechanism.

[chadwhitacre]

🐻 👈

[chadwhitacre]

Alright, we're calling it, here. Gonna punt. 😶

Summit fever. 🗻

[chadwhitacre]

"We did pretty well, though." —@kaguillera

Oh well. I was thinking I'd hate to leave yinz with a risky payday next week as well. I guess we'll pick up with this when I get back. I'm not gonna be able to keep working at the rate I have been, but hopefully we can land it in June. 😖

"Let's do a little more work, and then we'll go drink!" 💃

[chadwhitacre]

I'm having second thoughts about the proposal to use relative takes instead of absolute amounts. The main drawback is that it's not as inclusive. With absolute takes it's always possible to add another person at 1¢, but if the Team has to issue takes then takes become a scarce resource with a disincentive to add new people to the Team. I'm now leaning more towards Liberapay's model where (absolute) takes are treated as ratios when there's not enough money to cover all takes.

[chadwhitacre]

The main drawback is that it's not as inclusive.

Another drawback is that it's more complex. When I set my take I'm thinking about things like my living expenses and the time I want to spend, and it's most natural to think in terms of absolute amounts—"If I can take $500/wk I can work a day or two a week"—and that'd be harder to estimate if I have to translate from takes to dollar amounts.

[mattbk]

that'd be harder to estimate if I have to translate from takes to dollar amounts.

It would also fluctuate from week to week.

I'm now leaning more towards Liberapay's model where (absolute) takes are treated as ratios when there's not enough money to cover all takes.

That makes sense to me. Although a takes system seems like it would be easier to be "fair", the truth is probably that the fairest system lets people set their takes according to how much work they do--in real currency. A system with 1,000 takes seems overly contrived or gamelike.

[chadwhitacre]

I'm giving up on trying to get #4064 before this.

[chadwhitacre]

Looking at this again with @kaguillera. Things on our plate:

1. What are all the things in the old _mixin_team.py doing and do we still need them?
2. How are we going to do payday? Is now a chance to split payouts from payins?
3. What protections are we going to put in place? Throttling? Or something else? I have an idea for this.

[chadwhitacre]

Alright! Over lunch/coffee @kaguillera and I started looking at (1) and ended up hashing out (2) and (3). We were looking at what from _mixin_team.py we had already reimplemented in #4023 and #4024, and what we hadn't yet. set_take_for is where throttling used to be implemented (takes capped to a certain amount), so that got us to talking about (2) and (3).

The basic idea we had is to invert the baseline: rather than having an amount that the Team owner withholds prior to payouts, the owner sets an explicit amount that is available for payouts.

This contributes to stability (cf. https://github.com/liberapay/liberapay.com/issues/287#issuecomment-217378424). As long as a Team can deliver on the amount they're expecting to have available for payouts each week—and if they can't, there are some management issues going on!—then Team members have a more solid basis for their own financial planning than if their income is tied to a potentially volatile weekly Team income.

(This queues us up in the future to decouple payins from payouts, but we won't fully tackle that on this ticket.)

Then, we process distributions from smallest amount to largest. This shifts abuse protection from take setting time to take distribution time. Everyone sets their take to whatever they want whenever they want (fixing Drupal's problem where they brought a mature team to Gratipay and it was annoying to not be able to set appropriate takes immediately). Someone who tries to swoop in and take $1,000,000 in a week is limited by the amount that the Team has made available for the week. The absolute worst they could do is steal the top taker's take (by setting their take to $1 less than the top taker). Three additional controls could mitigate this:

• Implementing a Wednesday cut-off for payout changes.
• Sending a weekly notification (also on Wednesday) to all Team members showing payout changes.
• Making Team joining and leaving public history contributes to the possibility of due diligence (a la, "list your last three jobs and why you left").

It also puts pressure on the top taker, who is probably a Team manager or core member, to vet new Team members to their own personal satisfaction (aligning the manager's personal interest with the Team's).

[chadwhitacre]

We've also got the old UI to refer to. We need to think about the additional plumbing required not just to set/get takes, and update computed values, but also to support the UI details we need: showing changes from last week, and now showing clipping (cf. throttling previously).

[chadwhitacre]

How should we structure the PRs?

[chadwhitacre]

Can we bring back and modify the existing files instead of starting from scratch?

[chadwhitacre]

I like having the permissions handling down in the Python library instead of in the JSON simplates. However, we do need i18n.

[chadwhitacre]

We can use a teams.weekly_distribution field to toggle the availability of the members UI (distributing will be the sum of takes or weekly_distribution, whichever is less). weekly_distribution will default to zero, and we'll manually set it to $50 or whatever for Gratipay. That will allow us to bring back the existing UI even in a broken state, shielding it from everyone but ourselves. We can iterate on that without having to batch up a major mess of PRs. Let's do it!

[chadwhitacre]

We can implement the payday changes last. We can keep improving the plumbing and UI until we've got the data we need in the database to support payday, and then we can implement distribution in payday. Small batch size!

[chadwhitacre]

@aandis @rohitpaulk You ready to make a push on this? How's the next couple weeks look for ya?

[chadwhitacre]

I think @kaguillera and I are going to start self-merging some of these PRs next week, to keep this moving along.

FYI, we've got a couple marketing events coming up week after next (Code & Supply, OpenSource.com). Abstractions is coming up next month. I'd love to be able to say during these that we have distributions back in place for the Gratipay team, at least. Let's see how far we can get! :-)

[chadwhitacre]

I'm getting ready to merge #4076 and everything it includes. @kaguillera is out this week (we're planning to hang out on Thursday with our families but not work). @aandis and @rohitpaulk are working full-time. We've got to get some economic incentive back here so we can all afford to work on Gratipay! 😡

[chadwhitacre]

And here I'm self-merging major chunks of work right after documenting for @nashe that that's not how we operate. 😞

At least @kaguillera has seen most of this code so far.

[chadwhitacre]

Whoosh! Hear that? It's the sound of #4068 #4070 #4072 #4073 #4074 #4075 and #4076 landing in one fell swoop. 👍

[chadwhitacre]

Deploying ...

[chadwhitacre]

👍

[chadwhitacre]

The old takes table is in backups if we ever need that info.

gratipay::MAROON=> select count(*) from takes;
┌───────┐
│ count │
├───────┤
│  2387 │
└───────┘
(1 row)

gratipay::MAROON=> select count(*) from takes;
┌───────┐
│ count │
├───────┤
│     0 │
└───────┘
(1 row)

gratipay::MAROON=>

[chadwhitacre]

We should make sure I'm available during the first few paydays that we operate with this. It's my off month this month, and I'm actually going to be on vacation for the 11th, and at the Abstractions conference for the 18th. Therefore, let's aim to run with payouts for the first time in early September. That gives us another month to deploy the UI and payday changes, and I'm the payday pilot in September, so it'll fall to me to deal with any bugs (fair since I will mostly likely have written the bugs :).

[chadwhitacre]

Okay! We have UI for adding members and setting takes! 🎃

The payday changes are underway in https://github.com/gratipay/gratipay.com/pull/4102. Other than that it's just a matter of shaking out bugs and final clean-up. Here's to Team Gratipay payouts on September 1!

[dragon788]

Have you guys considered working with Keybase.io to verify "trusted" identities for developers and/or supporters? I know they are in beta, but they seem like a great advance on the PGP servers that support the bulk of open source currently.

[chadwhitacre]

@dragon788 That's an interesting idea, but we need national identity, like, for where you pay your taxes.

[dragon788]

I don't know if it's in the roadmap for Keybase/Gratipay, but I would love to see some kind of permissions model around your identity similar to, but STRONGER than Facebook/Google/LinkedIn, where you could allow other parties access to more of your information than just your public description. You could use this as an online "backup" of your personal information, encrypted via your private keys into your profile, and the site generates subkeys to allow a new user to access bits of information about you, eg. your bitcoin wallet address or see your EIN, if you are operating a development business and it is required for a form like the 1099 if you are contracting somewhere.

[chadwhitacre]

You could use this as an online "backup" of your personal information, encrypted via your private keys into your profile, and the site generates subkeys to allow a new user to access bits of information about you, eg. your bitcoin wallet address or see your EIN, if you are operating a development business and it is required for a form like the 1099 if you are contracting somewhere.

Yes! That's exactly the sort of use case we can explore once we have identity verification in place. It will take us a while to get there but I hope we can some day! :-)

[chadwhitacre]

Okay! PRs for payday (#4102) and MassPay (#4113 + #4111) are ready to go! I'd like to go back over the whole thing to make sure our computed values are good, and tidy up docs and the test suite. But basically those PRs are the last ones we need in order to run payouts for Team Gratipay and close this ticket and milestone! We are on track to do that next week! 😱

[chadwhitacre]

@aandis @clone1018 @mattbk @nashe @rohitpaulk @TheHmadQureshi + @gratipay ...

Now would be a good time to submit your identity information, if you'd like to participate in the distribution next week! 💃

[chadwhitacre]

So lonely! :)

[mattbk]

I am unverified. :disappointed:

[chadwhitacre]

I looked at your info (and @clone1018's! you should both have gotten an email), but I forget how to verify ... :)

[chadwhitacre]

use Python API to verify identities

Ah. :)

@clone1018 Your id has fewer digits than I'd expect. Is it right? (Do we have a bug, like maybe an int to text conversion loosing zeros bug or something?)

[chadwhitacre]

@mattbk Verified! 🇺🇸 💃

Why is the flag not showing up in the banner on your profile, though? 😕

(Also, your username is missing from the banner on the identities page.)

[clone1018]

I created it incorrectly as a Tax ID and changed it to drivers license, but it won't let me save it as a DL, I'll try and delete and recreate.

[chadwhitacre]

D'oh! Sorry. Another bug! :)

[clone1018]

Created it as drivers license, shows as Tax ID, I assume the saved value isn't being translated to the form.

[chadwhitacre]

Could be. Hard to tell since your info is encrypted in the database. :-)

Verified! 💃

[chadwhitacre]

Reticketed #4115 and #4116.

P.S. Are yinz getting emails when I look at your info? You should be.

[clone1018]

Yes, I've received two.

[chadwhitacre]

Cool. Okay! So it looks like I won't be the only one taking from Team Gratipay next week. :)

!m @mattbk @clone1018

Any other takers? :D

[mattbk]

I got an email.