Create a central package database and website with all the trimmings

[brson]

As part of the Rust user experience we need package discovery to be very easy. Let's make a single place to find packages, docs, metrics, and whatever else is important.

Let's try not to require a lot of dynamic content, and instead rely on other services like Travis CI and GitHub pages wherever we can. Dynamic web sites require operational support that makes them much more difficult to set up and maintain.

Just copy http://npmjs.org.

cc #9875

Nominating.

[brson]

This will likely incorporate or replace http://docs.octayn.net/ and http://rust-ci.org/rust-ci/

[emberian]

docs.octayn.net was a "it's the weekend, let's procrastinate homework with something useful!" project. it just fetches and runs rustdoc on a bunch of repos.

[emberian]

It'd be cool to integrate benchmark graphs with test run data, and view how it changes over time.

[brson]

cc #1453 #1642

[catamorphism]

1.0, high

[liigo]

Let users vote on existing packages would be help to find great packages. See reddit and stackoverflow.

[gsemet]

+1, love this idea. How will this integrate if the developer environment? and how will dependencies be distributed along with the software? For instance, pip install everything in /usr/, or in the virtualenv if used. However, one might want its dependency project for development and only distribute the binary. How about versionning? It's a field where I have lot of idea and some experience (in python, C++, a little bit of java), and I always wanted to do something for my C++ projects, but maybe rust is the place to do something :) I really like the simplicity of pypi, and hated the complexity of maven and generally all java-related dependency management tools.

[runvnc]

I think you can release a central database and website without fully implementing dependency management and package versioning, and I hope you will.

I think the first step is to link to the wiki or Travis CI page on the top of the Rust home page, and that would have saved me about six or seven minutes.

Anyway I am currently primarily a Node.js user. I think that the node/npm semantic versioning and dependency management (http://semver.org/, https://github.com/isaacs/node-semver) is the state-of-the-art, and I hope that Rust will consider adopting something similar.

I am pretty excited about Rust, even more so than Go, because of the straightforward syntax, type inference, performance, low-level capabilities, etc. I think that semantic versioning and dependency management similar to Node/npm would make Rust the obvious choice for many types of server-side development.

But I think that semantic versioning/dependency management is so complex that it at least deserves its own issue.

[thehydroimpulse]

I agree with @ithkuil, in that Rust should take some ideas from Node in terms of NPM. I think having a local, by default, install path would be best. Maybe under $PWD/.rust/crates or something. Something akin to node_modules. I think those systems who adopted a global install path, by default, made things much harder in the end.

[tarcieri]

I would strongly suggest looking at The Update Framework:

http://theupdateframework.github.io/

It provides a set of metadata formats (based off of JSON) upon which you can implement secure software update systems.

It's designed to provide secure code signing with multisignature trust in a way that developers can opt-in to signing their own packages, otherwise they're signed automatically by the update system. This makes it unobtrusive while still providing good security guarantees.

I have been working on implementing it as part of RubyGems. It's also being integrated into PyPI. I'd be interested in working on an implementation for Rust too!

[brson]

cc #1642

[tj]

FWIW I'm building a tool called versions.io that will remove the need for people to keep re-inventing package management solutions. They're all effectively identical, so I'd like to house them under one roof, so versions.io will have the concept of platforms built right in (some packages can be shared between multiple), and we'll have organization support and tight github integration. Backed by s3 it will be much more robust than npm etc, no need to copy all of that. If anyone finds that appealing I'd love to collaborate and take suggestions!

It's worth noting that you could still have say crates.rust-lang.org CNAME rust.versions.io, so from a "branding" perspective it wouldn't be a loss either.

[tarcieri]

@visionmedia what security model are you using?

On Thursday, January 23, 2014, TJ Holowaychuk notifications@github.com wrote:

FWIW I'm building a tool called versions.io that will remove the need for people to keep re-inventing package management solutions. They're all effectively identical, so I'd like to house them under one roof, so versions.io will have the concept of platforms built right in (some packages can be shared between multiple), and we'll have organization support and tight github integration. Backed by s3 it will be much more robust than npm etc, no need to copy all of that.

— Reply to this email directly or view it on GitHubhttps://github.com/mozilla/rust/issues/10041#issuecomment-33130674 .

Tony Arcieri

[tj]

I'm not far into the project yet, just started the API/s3 integration a few days ago, certainly open to suggestions! I'd like to also facilitate pre-compiled attachments for releases, but that's probably more applicable to executables and addons for platforms like node/ruby

[tarcieri]

@visionmedia I'd suggest checking out TUF which I mentioned earlier: http://theupdateframework.github.io/

[tj]

@tarcieri will do

[thestinger]

@visionmedia: Please keep in mind that Zero Install is an existing solution to build on with features like package signing and integration with system package managers. It handles side-by-side installation of multiple versions fine too.

[tj]

@thestinger I'm not familiar with Zero Install - http://0install.net/? Not quite what I had in mind but feature-wise it seems pretty complete.

[thestinger]

Yes, that's it. I became familiar with it when Rust was considered as a possible language for the rewrite. Rust is of course nowhere near stable enough yet, and the ultimate decision was OCaml.

[puredevotion]

cc me; stupid question, but do we want this written in pure rust? Although don't really have time now, I'll follow this and think along :)

[thestinger]

I don't think we should reinvent the wheel. Do we want to deal with issues like package signing and handling dependencies in other languages ourselves? That's why I'm pushing for 0install as it's already a full solution with existing usage. Rust just needs to layer on some Rust-specific sugar.

[puredevotion]

Yes, or as suggested fork npm or pypi.

[thestinger]

How do they compare to Zero Install? I think we have some hard requirements:

• support for binary packages
• support for retrieving package source
• no conflicts between packages or multiple versions of packages side-by-side
• dependency handling and upgrading
• package signing
• tooling for managing repositories
• cross-platform, without a requirement of cygwin (like Nix)
• support for packages written in other languages (for dependencies)

As a bonus, Zero Install gives us...

• GUI versions of the tools
• integration with system package managers (can use system libraries)

[tj]

IMO reinventing the wheel isn't so bad when that wheel is a lot nicer to use than the last wheel. Definitely -1 for npm fork for me.

[puredevotion]

• support global and local installation of packages (preferably without a tool like virtenv)
• version management (part of dependencies i guess)
• some sort of registration file, like grunt.json (?)
• some sort standardized readme, like readme.md (?)

Interesting point is the implementation: centralized or decentralized? Do we want that developers send new versions in, and we keep that version somewhere stored? or do we just want a repo-URL registered, and use something like version-tags in git to keep automagically up-to-date with versions?

@visionmedia to be honest, not really sure what is the state-of-the-art of package management, and I'm a sucker for ideas like your versions.io (is it on github?)

[tj]

I have github.com/versions, I'll push the code up soon if I have some more time to hack on it. I'm definitely a sucker for first impressions, so I'd like to get the API as nice as I can. Each "platform" can store arbitrary metadata per package so it should facilitate the needs of most of these different languages/systems, basically a glorified S3 with auth/signing/org/search etc.

I think storing the packages in S3 would be good, we've had issues with both github and npm being down, and while S3 isn't perfect it holds up pretty well, and using a pluggable store would make it trivial for people to fork this thing and boot up a private instance. There's definitely pros and cons of centralized / decentralized, though personally I think decentralized is covered just fine already, it's easy to write a package manager that just supports tarballs/git etc, but then it's really annoying when someone's site goes down, or you're trying to find where the damn package lives and find out it it's in bitbucket or something annoying to work with haha

[lambda-fairy]

I think we should be careful not to over-engineer this.

For example, why store packages in S3, or separately at all? Git supports versioning (via tags), signatures, generating tarballs (git-archive), and mirroring (obviously). Find a decent Git host -- hey, don't we have GitHub? -- and we get all these features for free.

If you're a "sucker for first impressions", then the fact that Git does all this already should ring alarm bells. If people are tagging their versions, signing their commits, maintaining mirrors already, then why would they care if some magical package manager does it as well? What would really impress them (and me) would be if the program determined the version automatically, based on existing tags -- which is exactly what rustpkg does. Maybe even silently switch to a mirror when the main site goes down.

We want less work for ourselves, not more. If our solution ends up just a thin layer around existing tools, that's a good thing. Let's not forget that.

[brson]

Nominating for removal from milestone. Cargo is not a 1.0 blocker, neither is this.

[pnkfelix]

Removing from 1.0 milestone; leaving at P-high.

[japaric]

Can we close this issue, now that we have crates.io? Any issue or feature request should be filed against its repo.

cc @alexcrichton