Integrate npm - Githubissues

chadwhitacre commented 7 years ago

✈️ This is the flight deck for the Integrate npm project. ✈️

Current open-source crowdfunding options (Kickstarter, Patreon, Gratipay, OpenCollective, etc.) are consumer-grade. Our hunch is that a business-grade product with better aggregation can better serve the companies that want to pay for open source, because companies use hundreds or thousands of open source packages, not just a few.

Picking up from https://github.com/gratipay/gratipay.com/pull/4135#issuecomment-255122149 and https://github.com/gratipay/inside.gratipay.com/issues/852#issuecomment-255098337 ...

For wider context see:

JavaScript is the most popular language in open source and npm is the most popular package manager for JavaScript. A good first concrete step towards helping companies pay for open source (#4135), therefore, will be to add the ability to pay for any package on npm. Once we have npm deployed, we will have enough experience to inform a partnership with Libraries.io for the rest of the package managers.

Target

~~Our goal is to announce this feature in my lightning talk on Thursday, October 26 at Red Hat's All Things Open conference (https://github.com/gratipay/inside.gratipay.com/issues/757).~~

Our goal is to incrementally improve this feature throughout the first half of 2017, with an eye towards OSCON and $ustain in May.

Package names to test with

From https://github.com/gratipay/gratipay.com/pull/4135#issuecomment-262672635:

Todo

Prerequisites

[x] Land the Decouple milestone—https://github.com/gratipay/inside.gratipay.com/issues/432
[x] Finish Project Basics—https://github.com/gratipay/gratipay.com/milestone/18

Checkpoint 1: Inert `/on/npm/foo/` Pages

[x] wrap marky-markdown (#4154)
[x] load up npm (#4153, #4158)
[x] load up npm readmes (#4159, #4160, #4161, #4162, #4164, #4176, #4179, #4180, #4181, #4182)
[x] stop storing readmes after all (#4211)
[x] add inert (can't actually give to them) /on/npm/foo pages (#4212)

Checkpoint 2: Giving to Packages

[ ] link packages to teams (#4155)
[ ] opt in or out from /on/npm/foo/ pages via email verification
[ ] add payment widget on /on/npm/foo/ pages

Checkpoint 3: Easy Sign-up

[ ] bulk manage payments from /on/npm/ pages

Nice to Have

[x] move search into {% content %} (leaving social network jump in sidebar)
[ ] merge current result types together (~user via username, ~user via profile statement)
[ ] add Teams to local index for inclusion in results
[ ] mix npm packages into search (#4147)
[ ] pledging to as-yet-unclaimed packages
[ ] load up npm incrementally (reticket from #4153)
[ ] deal intelligently with unpublished packages (reticket from https://github.com/gratipay/gratipay.com/issues/4148#issuecomment-256525647)
[ ] we seem to get two H1s on https://www.npmjs.com/package/resize-img
[ ] syntax highlight

Promotion

[ ] update the "★ announcement ★" link (#4149)
[ ] Publish a blog post.
[ ] Promote the heck out of it!

✈️ This is the flight deck for the Integrate npm project. ✈️

chadwhitacre commented 7 years ago

@mattbk You wanna draft a blog post?

chadwhitacre commented 7 years ago

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

chadwhitacre commented 7 years ago

I've started a task list in the ticket description.

aandis commented 7 years ago

@aandis How about I start hacking on this today and tomorrow, and I will leave notes for you over the weekend based on how far I get. Ya?

Yep. That works. A quickstart on https://github.com/gratipay/gratipay.com/pull/4147 will help in implementing the exact functionality you have in mind.

chadwhitacre commented 7 years ago

@aandis What's your availability? I was assuming Saturday and Sunday would be the time you could work on this but maybe I'm wrong?

aandis commented 7 years ago

@whit537 I am free over weekend, yes. I'll be keeping a close eye on this ticket for the next couple of days.

chadwhitacre commented 7 years ago

I've reached out to npm, Inc. in private email to let them know about our plans and invite them to give feedback and/or cross-promote with us next week.

chadwhitacre commented 7 years ago

Is there a rate limit on npm registry API requests? Can it be up'd with authentication? I find no docs on this.

chadwhitacre commented 7 years ago

We should think about possibly merging existing teams into these.

chadwhitacre commented 7 years ago

The code that npm uses to talk to the registry.

https://github.com/npm/npm-registry-client

aandis commented 7 years ago

http://nipstr.com

aandis commented 7 years ago

@whit537 I am not seeing any official well documented api for searching packages over npm. Best I could find is https://github.com/wires/npm-keywordsearch through this.

chadwhitacre commented 7 years ago

@aandis Let's discuss search on #4147.

chadwhitacre commented 7 years ago

Hashed this out a bit with @kaguillera. The problem we're solving is companies funding open source software. That's a market. The two sides are companies (buyers) and projects (sellers). We need to make it as easy as possible for companies to give to open source software; that's #4135. This is the other side, making it as easy as possible for developers to receive payments for open source.

`/on/npm/`

One thing we realized is that a single developer will have multiple projects. "Easy as possible" means letting them configure multiple projects at once. Doug Wilson is maintaining 103 packages. We can't make him go through 103 separate verification workflows. He needs one worfklow.

We're thinking that that workflow should live at /on/npm/.

-anon—Sign in to configure payments for your npm packages. -auth—Verify the email you use on npm to configure payments your npm packages. -no packages—We didn't find any npm packages with alice@example.com as an author or maintainer. -packages—We found 103 npm packages with alice@example.com as an author or maintainer.

We should account for the case where another ~user has already claimed the package.

Added to todo. ^^^ Bring this over to a PR when the time comes.

Going all in?

The other thing we realized is that the 400,000 packages on npm dwarf the 200 projects currently on Gratipay. We should consider redoing the homepage to contain a search box (for givers and the curious) and a link to /on/npm/ (for developers ready to start receiving payments). We don't need to list "our Teams" on the homepage. With this change, "our Teams" instantly explodes to the size of npm.

chadwhitacre commented 7 years ago

Oh! And we should bring back pledging: givers should be able to set up payment instructions to projects before the projects opt-in. This is where opt-out starts to matter, because it prevents pledging.

chadwhitacre commented 7 years ago

We're getting back to the same dynamic as Gittipay 1.0, but with a focus on packages on open source software repositories (→ Teams [Projects]) rather than users of social media networks (→ ~users).

chadwhitacre commented 7 years ago

Also, it seems unlikely that we'll get this all done before next Thursday. Let's do what we can ... and have fun doing it! :-)

chadwhitacre commented 7 years ago

Alright, @aandis, I've stubbed out a few PRs and updated the todo in the description here. Hopefully that gives you something to go on before I get online tomorrow!

!m @aandis

chadwhitacre commented 7 years ago

@aandis Can we use this thread to communicate and coordinate about what specifically we're each working on? I'm seeing work from you on #4154 and #4155 (!m @aandis). What are you working on now? Do you think the todo on this ticket is right and in the right order or do we need to modify it?

I'm going to start at the top of the todo with #4117 and then move down to join you on the actual coding bits. :-)

Let's do this! 💃 🌻

aandis commented 7 years ago

@whit537 I am gonna wrap up #4154 and then work on #4153 and #4155 to make #4151 possible. I think we should move pledging out of #4155 into it's own ticket and focus on mvp to be able to give to a package. Pledging is a nice to have feature but only when the groundwork is laid.

chadwhitacre commented 7 years ago

@aandis Cool. Strictly speaking, #4155 isn't necessary for #4151. In other words, we could have /on/npm/foo/ pages without any possibility of giving to them. Is it worth batching #4154 #4153 and #4151 as our first checkpoint, and tackle #4155 after that first checkpoint is deployed?

chadwhitacre commented 7 years ago

{#4154, #4153, #4151} feels like something we could potentially get deployed before the end of the day today. Deployment itself will take some effort because of the need to spin up a worker dyno—something we haven't done before—to handle the continual npm syncing. Sure, inert /on/npm/foo/ pages won't be much to brag about, but it'll at least give us a feeling of win. :)

Thoughts?

aandis commented 7 years ago

Sounds good. :)

chadwhitacre commented 7 years ago

@aandis Cool. I've updated the todo.

chadwhitacre commented 7 years ago

Alright, sooooooo ... it's emerging that the Decouple milestone, which should really get done before this (so we are free to accept all of the new Teams → projects we're hoping to get from this), is going to have some code implications. In particular we really ought to reflect a couple name changes in our codebase to avoid drift and tech debt.

Teams → projects
members → collaborators

I guess that's just a heads-up that we're looking at some potentially serious rebases of the work under here once the dust settles under https://github.com/gratipay/inside.gratipay.com/issues/432.

mattbk commented 7 years ago

once the dust settles under gratipay/inside.gratipay.com#432

This will be a happy day.

chadwhitacre commented 7 years ago

Maybe Tuesday? 😳

chadwhitacre commented 7 years ago

I guess that's just a heads-up that we're looking at some potentially serious rebases of the work under here once the dust settles under gratipay/inside.gratipay.com#432.

Scratch that, @aandis. It doesn't make sense to block this on the full codebase renames on gratipay/inside.gratipay.com#432. We can still get the important stuff done (Terms of Service update, in particular), without renaming tables.

chadwhitacre commented 7 years ago

@aandis Heads up that I'm going to be offline today. I've been going pretty hard and need a break before I fly out tomorrow for https://github.com/gratipay/inside.gratipay.com/issues/757. Whatever you get done under this will be awesome! Tomorrow afternoon/evening I should be able to pick up where you leave off.

Also, I think it may be better to focus on bulk sign-up on /on/npm/ instead of adding npm packages to search. The former is of more value to developers, which is who we're aiming at here. The latter is for companies/givers, which is less of the focus this week.

chadwhitacre commented 7 years ago

!m @aandis 💃

aandis commented 7 years ago

@whit537 I am just gonna finish what I can on https://github.com/gratipay/gratipay.com/pull/4153 since that's anyway going to be needed for email signups. I'll try to find time this week for more. Good luck for https://github.com/gratipay/inside.gratipay.com/issues/757 :)

chadwhitacre commented 7 years ago

Thank you! :-)

chadwhitacre commented 7 years ago

Also, I think it may be better to focus on bulk sign-up on /on/npm/ instead of adding npm packages to search.

Todo updated to reflect this.

I just compiled my schedule for this week. I should have a little time tomorrow and Tuesday, and then I should be free from lunchtime onwards on Wednesday until I fall asleep. I would really rather not be trying to hack on Thursday morning before the lightning talk, so my goal would be to get everything done by Wednesday night.

Please forgive me, @aandis @rohitpaulk et al.: I may well cut lots of corners and self-merge PRs and whatnot in order to get this done. I really really want to demonstrate something real and interesting on Thursday! :rage4: Let's go!

chadwhitacre commented 7 years ago

I'm gonna pause on decoupling since that's pretty far along. The remaining changes to the About pages are low-risk. I'm going to focus here over the next couple days and come back to decoupling on Wednesday.

mattbk commented 7 years ago

The remaining changes to the About pages are low-risk.

If I get some time I'll dip into https://github.com/gratipay/gratipay.com/pull/4157 to move it along.

chadwhitacre commented 7 years ago

Okay! #4153 is deployed, and I've scheduled a daily job for 15:00 UTC.

screen shot 2016-10-26 at 10 45 12 am

chadwhitacre commented 7 years ago

Gonna get ready to head over to https://github.com/gratipay/inside.gratipay.com/issues/757, I'll check Papertrail before I leave. I won't be back for ~6 hours, at which point I plan to start hacking on the rest of this ticket.

chadwhitacre commented 7 years ago

Your account has reached 100% log data transfer for this period. Log processing has been paused.

End of the month! Would you believe we got cut off, like, two hours ago? 😞

chadwhitacre commented 7 years ago

Nothing yet. 😕

gratipay::MAROON=> select count(*) from packages;
┌───────┐
│ count │
├───────┤
│     0 │
└───────┘
(1 row)

chadwhitacre commented 7 years ago

😕

[gratipay] $ heroku logs -a gratipay | grep scheduler
[gratipay] $

chadwhitacre commented 7 years ago

"Last run" is now Oct 26 15:00 UTC. Maybe I will try it from heroku run bash ...

chadwhitacre commented 7 years ago

[gratipay] $ heroku run -a gratipay bin/sync-npm.sh > first-run.log 2>&1

[gratipay] $ tail -f first-run.log 
Running bin/sync-npm.sh on gratipay... starting, run.1126 (Hobby)
Running bin/sync-npm.sh on gratipay... connecting, run.1126 (Hobby)
Running bin/sync-npm.sh on gratipay... up, run.1126 (Hobby)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 24 27.2M   24 6784k    0     0   946k      0  0:00:29  0:00:07  0:00:22  825k

chadwhitacre commented 7 years ago

It's processing packages ...

econds
processed 2000 packages in   3 seconds
  0  201M    0 1167k    0     0   253k      0  0:13:32  0:00:04  0:13:28  253kprocessed 3000 packages in   4 seconds
  1  201M    1 2063k    0     0   367k      0  0:09:21  0:00:05  0:09:16  367kprocessed 4000 packages in   5 seconds
processed 5000 packages in   5 seconds
  1  201M    1 2895k    0     0   435k      0  0:07:52  0:00:06  0:07:46  503kprocessed 6000 packages in   6 seconds
processed 7000 packages in   6 seconds
  1  201M    1 3727k    0     0   490k      0  0:07:00  0:00:07  0:06:53  573kprocessed 8000 packages in   7 seconds
  2  201M    2 4431k    0     0   513k      0  0:06:41  0:00:08  0:06:33  809kprocessed 9000 packages in   8 seconds
  2  201M    2 5263k    0     0   549k      0  0:06:15  0:00:09  0:06:06  821kprocessed 10000 packages in   9 seconds
  2  201M    2 5967k    0     0   560k      0  0:06:07  0:00:10  0:05:57  776kprocessed 11000 packages in  10 seconds
  3  201M    3 6415k    0     0   549k      0  0:06:14  0:00:11  0:06:03  699kprocessed 12000 packages in  11 seconds

chadwhitacre commented 7 years ago

Oh heck yeah. Streaming! 💃

If I'm reading that right, it's sending data to postgres as it's discovered, not waiting to read the whole file.

chadwhitacre commented 7 years ago

Well, sorry. It's serializing as it downloads. I don't think it's uploading as it serializes.

chadwhitacre commented 7 years ago

Though some additional logging would help confirm that.

P.S. This play-by-play should maybe still be on #4153. :o)

chadwhitacre commented 7 years ago

Back from https://github.com/gratipay/gratipay.com/pull/4158. Rescheduling the sync ...

chadwhitacre commented 7 years ago

Scheduled job has kicked off!

chadwhitacre commented 7 years ago

2016-10-27T00:35:37.400471+00:00 heroku[scheduler.9462]: Process exited with status 0

chadwhitacre commented 7 years ago

Okay, we're live with the npm syncer.

gratipay::MAROON=> select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 372271 │
└────────┘
(1 row)

gratipay::MAROON=>

gratipay / gratipay.com

Integrate npm #4148

Target

Package names to test with

Todo

Prerequisites

Checkpoint 1: Inert `/on/npm/foo/` Pages

Checkpoint 2: Giving to Packages

Checkpoint 3: Easy Sign-up

Nice to Have

Promotion

`/on/npm/`

Going all in?

gratipay / gratipay.com

Integrate npm #4148

Target

Package names to test with

Todo

Prerequisites

Checkpoint 1: Inert /on/npm/foo/ Pages

Checkpoint 2: Giving to Packages

Checkpoint 3: Easy Sign-up

Nice to Have

Promotion

/on/npm/

Going all in?

Checkpoint 1: Inert `/on/npm/foo/` Pages

`/on/npm/`