search

nice-registry / all-the-package-names

🔤 A list of all the public package names on npm. Updated daily.
242 stars 28 forks source link

This is going to be bigger and bigger #3

Closed IonicaBizau closed 6 years ago

IonicaBizau commented 8 years ago

@zeke Is there any way to keep the size of the repo the same while time is passing on? This is already pretty big and it will be bigger and bigger.

zeke commented 8 years ago

I'm open to ideas..

IonicaBizau commented 8 years ago

@zeke Maybe run the extract script after installing (a postinstall script)?

zeke commented 8 years ago

run the extract script after installing

☝️ This is a good technique, and one I use elsewhere for things like all-the-packages

But this package is different. I want users to have access to a dependency-free list of package names that is "accurate enough". If we were to add a postinstall script, it would require a full streaming of the contents of the registry on every install, which would take a long time per user. The hourly re-releasing of the package takes that burden off the user and onto the publisher (a bot).

I wrote about the technique I'm using to automate this at http://zeke.sikelianos.com/npm-and-github-automation-with-heroku/

Also, this module is kind of a :trollface: to illustrate the fact that there's no documented public API for the npm registry. There is an undocumented, unmaintained, and incomplete couch database. There is the npm/public-api repo, but it's been mostly quiet in the 18 months since its inception.

IonicaBizau commented 8 years ago

@zeke Makes sense!

What if we set up the bot to override the commits so it won't keep all the world history in this git repo? Because in the end, that's the problem, that the git repo itself gets bigger. 😁

billiegoose commented 7 years ago

Also, this module is kind of a :trollface: to illustrate the fact that there's no documented public API for the npm registry. There is an undocumented, unmaintained, and incomplete couch database. There is the npm/public-api repo, but it's been mostly quiet in the 18 months since its inception.

^so many feels

zeke commented 7 years ago

yes. many feels.

Here's my current work to share more consumable registry metadata:

https://github.com/zeke/dependent-packages https://github.com/zeke/package-stream https://github.com/zeke/nice-package https://github.com/zeke/all-the-packages https://github.com/zeke/electron-npm-packages

billiegoose commented 7 years ago

What ARE you building with all these tools? 🤨

zeke commented 7 years ago

Working toward an understanding of the most popular and useful userland tools for Electron. Hoping to display this data on Electron's website soon.

zeke commented 6 years ago

Yes this is getting bigger, but so far that hasn't been a problem. We can reopen this when and if the size becomes a problem.

IonicaBizau commented 6 years ago

I agree! 😁