turn the ship

[chadwhitacre]

Last week, Gratipay stopped processing payments because of legal concerns with our old terms of service that became apparent after our processor, Balanced, announced that they're going out of business, and Stripe wouldn't take us. So in addition to migrating our processing infrastructure (#3377), we also have to change our business model at the same time. No :sweat:, right? :)

We hashed out our new business model on https://github.com/gratipay/inside.gratipay.com/issues/180 and https://github.com/gratipay/inside.gratipay.com/issues/192. This ticket is to track implementation. See also the Pivot milestone.

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[chadwhitacre]

I'm stuck thinking about the relationship between individuals, groups, and projects. I think #3337 is the cleanest way to go:

Users optionally have Projects. Projects optionally have Payroll.

Projects would be what you give money to.

But right now Users are what you give money to. We don't want to drop existing funding streams if we can at all help it. We want to give Users a way to migrate to a Project. Forcing migration before we process payments is okay since we're requiring acceptance of our new terms anyway.

[chadwhitacre]

I'm not sure how to pull off this pivot without #3337. That's a fair chunk of work.

[chadwhitacre]

Let's think about Participants (Users above) and Teams (Projects above).

Currently Teams are a kind of Participant (those with plural number and > 0 takes), but I think we should promote Teams to a first-class citizen with a new teams table.

A Team is a group of people (some of whom may even be Participants) working together to deliver a given product or service.

Teams have Members (Participants of ours who are part of the Team). Some Members are Owners.

(Note that Participants can still be singular or plural.)

Right now, Tips attach to Participants. How do we migrate Tips from Participants to Teams? We basically need Participants to create a new Team, to which we will reattach their Tips.

Since giving to Participants has been ambiguous in regard to products or services delivered, migrating giving to a single-purpose Team will be a judgement call for our users. If I have 100 supporters, and 60 would say they were giving to me because of one product or service I provide, and 30 because of another, and 10 just because they love me, then how should I migrate? I don't even know who my supporters are, so it's not like I can manually review and make assignments ("Oh, this person is a Fizzbuzz user, that's probably why they've been supporting me. Let's move their payment to the new Fizzbuzz Team"). No: I have to guess at the 60-supporter product/service, and migrate all of my giving to a new Team dedicated to that. Then I can make a second Team for the 30-supporter product/service.

[chadwhitacre]

Can a Team give to another Team?

[chadwhitacre]

Can a Team give to another Team?

I think not, but I think the Owner of a Team can give to other Teams. Let's say that Exchange Routes still attach to Participants. The Owner of a Team is a Participant, and that's where the leftover money drains after Payroll, so we still have the chance to offset their giving with their receiving in a given cycle.

[chadwhitacre]

The approach I'm considering is to make new tables teams, subscriptions, payroll, payments that are parallel to participants, tips, takes, transfers. We no longer write to the old tables, but they're available to use for migrating. Migrating happens when a Participant takes the initiative; we're not migrating all tips at once.

[chadwhitacre]

Where do we put Teams in our URL structure? And in our IA, for that matter?

Do we demote Participants? https://gratipay.com/whit537 → https://gratipay.com/~whit537. Does that redirect? Almost certainly, or we'd break lots of inbound links.

But then does https://gratipay.com/~Gratipay redirect back to https://gratipay.com/Gratipay? Do we have a single namespace for Participant and Team slugs, even if we differentiate them in the URL? Probably. It doesn't seem right to allow someone to register https://gratipay.com/~Gratipay. On the other hand, we're saying that we want an Owner Participant for a Team, so I guess we should allow ~Gratipay to be the Owner of Gratipay. Right?

[chadwhitacre]

In terms of IA, I think we're looking at the following changes as the absolute minimum to pull this off this week:

• /$USERNAME/ → /~$USERNAME/
• Add $TEAM at /$TEAM/, with a Profile at top level.
• /$USERNAME/members/ → /$TEAM/payroll/.

[chadwhitacre]

How do Participants register new Teams?

[chadwhitacre]

How about adding /new for that? Then we add /dashboard/teams for the new Teams review queue.

[chadwhitacre]

The first time a Participant with Tips creates a new Team, we should migrate all their Tips to Subscriptions for the Team (with prompting and confirmation, of course).

[chadwhitacre]

This feels good. I like how this is coming together, it's an evolution of Teams, a pretty natural step.

[chadwhitacre]

• [ ] replace UI over tips & takes with UI over subscriptions & payroll (#3400)
• [ ] implement UI for making/migrating to a new team
• [ ] revise payday to use subscriptions & payroll instead of tips & takes

[chadwhitacre]

• [x] unroll the interaction of number w/ takes
• [x] text fields on teams want to be i18n'd like statement
• [ ] implement is_approved
• [ ] implement closing a team
• [ ] sort out interaction with take_over
• [ ] don't forget about the API, especially tips.json
• [ ] should be able to search over teams
• [ ] don't forget about db self-checks
• [ ] don't forget OG headers
• [ ] don't forget about fake_data

[chadwhitacre]

After bashing around on #3400 all day yesterday, I'm starting to get a better idea of how to approach this. I've made three clean PRs:

• 3402 makes room in the URL structure for Teams

• 3403 cuts out pledging

• 3405 cuts out goals

We should be able to clean up tip limits based on number as well. Is there any other code we can simply cut out?

Beyond that, here's a review of work based on looking at a profile (hard, medium, easy):

• [ ] Dashboard - Quarter-baked as it is. Copy is all wrong but nothing besides that.
• [ ] Profile - Teams section will need to work with new Teams.
• [ ] Members - This will be moved entirely to Teams. I'm not sure how anonymous_receiving interacts with this but that's probably out of scope for this ticket.
• [ ] Receiving - Still want this, but it should only show payments from teams under the new system. It should also continue to respect anonymous_receiving.
• [ ] Giving - Still want this, but it should only show payments to teams under the new system. It should also continue to respect anonymous_giving.
• [ ] History - This should show history under both the old and new systems.
• [ ] Widgets - Widgets are going to have to change somehow as part of this transition. How?
• [ ] Identity - We almost certainly want identity information for any Team Owner. Right now we store this info at Balanced (they verify it with some API we don't know about). Do we need to start storing identity info locally as part of this ticket, even if we don't programmatically verify it yet?
• [ ] Settings - I think this is pretty much fine. Notifications looks goofier than every without the "notify patrons" option, but I'm not going to worry about it here.
• [x] Events - No references to other tables here, so I think we're good.

[chadwhitacre]

What should we do with tips.json? I think we should make a new API that uses team id instead, and just let tips.json go 404. How are we supposed to interpret tips to usernames from the old tipping system under the new system?

[chadwhitacre]

In which case, I think I'll rip out tips.json for this week, and reticket bringing back something similar.

[chadwhitacre]

URL structure

1. [x] move /username/ to /~username/ (#3402)

Reduce Surface Area

1. [x] remove pledging entirely (#3403)
2. [x] turn off giving in the UI/JSON (#3406)
3. [x] remove goal (#3405)
4. [x] turn off taking in the UI/JSON (#3407)

Rewrite Payday

1. [ ] make new subscriptions, payroll, and payments tables
2. [ ] promote _mixin_team to a Team class
3. [ ] rewrite payday to use new teams (#3414)

Wire Up Teams

1. [ ] application/approval/migration for teams (#3412)
2. [x] set payment to team
3. [ ] set take from team
4. [ ] rewrite History

[chadwhitacre]

I have the feeling that I'm going to be doing most of this work myself. I'm not even sure we'll be able to wait for proper review before merging and deploying. I'm at least going to keep using PRs, even if I end up merging them myself.

[chadwhitacre]

I went ahead and merged/deployed #3402, to get the ball rolling.

[chadwhitacre]

From #3406, here's a census of the parts of the user experience that touch the tips table:

• closing, merging, and archiving accounts
• updating team takes (#3407)
• adding/updating exchange routes
• setting tips (#3406)
• the pricing, stats, receiving, and giving pages
• tips.json (#3406)
• the old button widget (#3406)

Here's the details:

  _check_tips
  _check_orphans_no_tips

  Participant.update_number
  Participant.clear_tips_giving - close
  Participant.clear_tips_receiving - close
  Participant.update_giving - team takes, exchange routes, setting tips, take over
  Participant.update_receiving - team takes, setting tips, take over
  Participant.set_tip_to - {tip,tips}.json, close, throughout the test suite
  Participant.get_tip_to - tip widget, old button widget, pricing
  Participant.get_tip_distribution - stats, receiving page
  Participant.get_giving_for_profile - giving page, tips.json, close
  Participant.get_tips_receiving - unused!
  Participant.get_current_tips - tips.json
  Participant.take_over - take over
  Participant.final_check - archive, close

  www/about/stats.spt
  www/about/tip-distribution.json.spt

[chadwhitacre]

I'm deploying #3403.

[chadwhitacre]

Looks like we have a regression from #3402: the tip widget is posting to /foo/tip.json, which is 302 to a GET and the POST body is lost. I'm going to proceed with #3406, which simply makes tip.json 404 ahead of the move to /team/tip.json.

[chadwhitacre]

Okay, I'm pumped. :boom: :exclamation:

#!/usr/bin/env python
"""Sandbox for exploring the new Payday algorithm.
"""
from __future__ import print_function, unicode_literals

from collections import defaultdict

# Classes

class _Thing(object):
    def __init__(self, name):
        self.name = name
        self.values = list()
    def __setitem__(self, k, v):
        self.values.append((k,v))
    def __repr__(self):
        return(self.name)
    def __str__(self):
        return '\n'.join(['{} {}'.format(repr(k),v) for k,v in self.values])

class Participant(_Thing):
    pass

class Team(_Thing):
    owner = None

# Universe

a, b, c, d, e = [Participant(x) for x in 'abcde']
A, B, C, D, E = [Team(x) for x in 'ABCDE']

# subscriptions

a[A] = 1
a[B] = 1
a[C] = 1
a[E] = 1

b

c[A] = 1
c[C] = 1
c[E] = 1

d[D] = 1

e[D] = 1

# payroll

A[b] = 1
A[c] = 1
A.owner = c

B.owner = c

C.owner = a

D.owner = d

E[c] = 1
E.owner = e

def payday(participants, teams):
    """Given a list of participants and a list of teams, return a list.

    The list we return contains instructions for funds transfer, both card
    captures (positive) and bank deposits (negative).

    """
    t_balances = defaultdict(int)
    p_balances = defaultdict(int)
    p_holding = defaultdict(int)

    for p in participants:
        for t, amount in p.values:
            t_balances[t] += amount
            p_holding[p] += amount

    for t in teams:
        for p, amount in t.values:
            t_balances[t] -= amount
            p_balances[p] += amount

    for t in teams:
        p_balances[t.owner] += t_balances[t]
        t_balances[t] -= t_balances[t]

    assert sum(t_balances.values()) == 0

    return [(p, p_balances[p] - p_holding[p]) for p in participants]

for participant, instruction in payday([a,b,c,d,e], [A,B,C,D,E]):
    print("{} {:2}".format(participant.name, instruction))

It looks like throwing a Team abstraction into payday makes the algorithm much nicer. It seems like it will enable us to easily settle the graph completely each cycle, and have no escrow! Who knew!? :dancer:

Any guesses what the above spits out? :-)

(The "4" is wrong. It's zero, taking the sign into account [charge/deposit].)

[chadwhitacre]

@rohitpaulk Do you think there's any chance you'll be able to work on this before tomorrow's deadline? It'd be great to find a way to deal you in here. Maybe we can quickly stub out a teams table and Team class, and then you can work on the UI side while I work on the payday side?

[rohitpaulk]

@whit537 - Sure, sounds great. Let's cherry pick stuff out of #3400 and get a few more PRs out of the way.

[rohitpaulk]

Ah, looks like you're already on it :dancer:

[rohitpaulk]

Let's follow our common pattern - You start PRs, I'll build on them.

[chadwhitacre]

Let's follow our common pattern - You start PRs, I'll build on them.

Perfect. I actually have to disappear for an hour or so to get kids out the door. I see you've merged #3410. Can you start on wiring up the UI for creating teams? I've started #3412 for that.

[chadwhitacre]

!m @rohitpaulk

[rohitpaulk]

On it. !m @whit537

[chadwhitacre]

@rohitpaulk Hopefully you're topped up with #3412 and #3413 for a bit. I'm going to dive in on #3414. It occurs to me that we don't even need the ability to modify tips before tomorrow. I expect to develop the new payments-related tables on #3414. Either in #3412 or in a separate PR we can implement migration from the old tips and takes tables to the new tables, and then after Payday 154 we can start bringing back the UI to work with payments and payroll.

[chadwhitacre]

We have until 3:30pm US/Pacific tomorrow to complete payday. We do want to get a couple people signed up first, though. I'm still going to spend some time right now working on payday (#3414), because it's risky and challenging and I need to work on it in the morning while I'm fresh, but we can finish coding that tomorrow (I do have a 10:00 meeting with Citizens). We really should land the terms (#3408) and the team sign-up process (#3412) today if we can, in order to give @dcht00 @fenryxo @silverhook @ehmatthes @mixolidia et al. a little time to get in the door for tomorrow's payday (#3415).

[chadwhitacre]

cc: @copiesofcopies

[chadwhitacre]

Okay! We're making progress on the terms (#3408). We've got a batch of 12 small proposals/questions to sort out, after which we'll hit two slightly larger questions, one about Open Work, and another about how Teams register. How are we doing with #3412, @rohitpaulk?

[chadwhitacre]

Oh, right. You already told me it was ready for review. :-)

!m @rohitpaulk

[chadwhitacre]

I've topped up #3412. Next step is to move #3414 far enough forward that we have our new payments schema; #3412 is blocked on that and #3408. Less important but easier to tackle is #3413.

@rohitpaulk: If you come back around before I'm able to make significant progress on #3412 or #3414, I see two things you could work on:

• Try moving #3412 as far forward towards with migrations as you can. There's surely a bit of UI/testing work that could be done without actually having the schema present.

• 3413

[chadwhitacre]

I just made a pass through support and replied to people who have been waiting to hear from us, asking them to wait a little longer ...

[chadwhitacre]

Alright, I realized we don't need to block sign-ups on migration, only on terms. Here's the current state of the sign-up form:

It limits based on verified email and bank account or PayPal, and here's the success message (owner's email is displayed):

[chadwhitacre]

Alright, the pressure is on #3408. Once we land that we can start signing up Teams! After that it's all about #3414 (with a little bit of #3413 and some migration work, which could fall under #3415).

[rohitpaulk]

3413 is ready for review, and I've closed out support tickets that weren't related to the pivot.

[chadwhitacre]

Awwwweeeesssssoooommmmeee.

!m @rohitpaulk

[chadwhitacre]

Made it through support. Now to land #3408 and #3412 ...

[chadwhitacre]

3408 landed. Almost there with #3412 ...

[chadwhitacre]

Okay! Applications for new teams are now open!

https://gratipay.com/new

[mattbk]

By "new teams" you mean anyone who wants to migrate, correct? Not just new people to the service.

[jiri-janousek]

I got error "400 Bad Request: You must attach a bank account or PayPal to apply for a new team." although ~NuvolaPlayer does have PayPal account attached. Should I create a new ticket for this?

[ehmatthes]

I applied for introtopython.org. I used my personal Gratipay account to do so; I'm not sure if that was the correct approach, but I only have one Gratipay account.

Also, I see a message that my credit card is working but out of date. I tried to update the credit card, but the information didn't seem to save. I'm not sure where this fits in with all the recent changes.

[chadwhitacre]

By "new teams" you mean anyone who wants to migrate, correct? Not just new people to the service.

@mattbk Yes, that is correct.

I got error "400 Bad Request: You must attach a bank account or PayPal to apply for a new team." although ~NuvolaPlayer does have PayPal account attached.

Bah, sorry. Looking into it ...

I applied for introtopython.org.

Hooray! Introtopython.org is our first team! :dancer: !m @ehmatthes

I tried to update the credit card, but the information didn't seem to save.

Yeah, that's no doubt due to regressions from migrating to Braintree. We are starting to store new cards there even though we're going to still be processing on Balanced today. You can make a ticket if you want, or look for a recent one that might be related ...

[chadwhitacre]

Back from Citizens (#3366). Todo:

• [x] fix paypal bug
• [x] get people signed up, specifically:
  • [x] @ehmatthes
  • [x] @fenryxo
  • [x] @silverhook
  • [x] @mixolidia
  • [x] Sudo Room
  • [x] Sudo Mesh
• [x] rewrite payday (#3414)
• [x] migrate tips to subscriptions
• [x] run Gratipayday 154! :open_mouth: (#3415)
• [x] :cocktail: :beer: :poultry_leg: :love_hotel: