Secure Distributed Registry for deno

[ameerthehacker]

Deno is goal to create a distributed package management solutions but

1. How do we ensure the immutability of those packages?
2. How to we take care about the security vulnerability in them (or we don't do anything about it)
3. Since we have a set of distributed packages from various packages, how do deno users know which source to use for the same package

Thoughts on the solution

Is is possible that we create a registry system that anyone can host to share their packages (we do security checks, immutability support in the service) and is there is a way by which deno can detect that we are downloading packages from our service hosted registries securely without being fooled

[lucacasonato]

I'm going to compare this with Go which has a similarly distributed module system as Deno:

1. Go has a service that handles module immutability: https://proxy.golang.org/. It is enabled by default in Go 1.13.
2. Go doesn't have security reviews for modules.
3. Always download it from the location the module author is publishing it. If it is on deno.land/x then that is the place to download it from. You can also use raw.githubusercontent.com URLs with commit hashes, that is pretty immutable. There are also services like https://denolib.com.

Also Go uses standard https:// and ssh:// git operations to fetch modules. That means if you trust the domain you can trust that the communication to that domain is secure because of https.

[ameerthehacker]

thanks @lucacasonato that gives a lot of clarity. Are we planning to build similar proxy system for deno too or is it already built?

[lucacasonato]

It doesn't exist right now, and I don't know if it is something that is planned. @ry?

[kitsonk]

https://deno.land/x/ is effectively the equivalent to the proxy and documents on that page how other packages can be added. The optional support for lock files (#3231) was the solution for sub-resource integrity to compliment this.

[lucacasonato]

Well no because it doesn't do the immutability caching. Go Proxy does do that (pretty much the entire point of go proxy).

But yeah, pretty unimportant if you use the lock files.

On Wed, 8 Jan 2020 at 00:12, Kitson Kelly notifications@github.com wrote:

https://deno.land/x/ is effectively the equivalent to the proxy and documents on that page how other packages can be added. The optional support for lock files (#3231 https://github.com/denoland/deno/pull/3231) was the solution for sub-resource integrity to compliment this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/denoland/deno/issues/3616?email_source=notifications&email_token=AB3XNVIOYUCRE3KHR6MDWADQ4UD5ZA5CNFSM4KD3HEH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIKTWGI#issuecomment-571816729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3XNVMKJRPJNVMV3A6DH4TQ4UD5ZANCNFSM4KD3HEHQ .

[rsp]

@ameerthehacker @lucacasonato, those are interesting issues worth thinking about.

The lock files can ensure that I always get the same dependencies that I get the first time when I installed them, but it doesn't ensure that I get the right files for the first time to begin with.

Let's say that a malicious user creates a useful library with no bad code. Something that helps with handling credit cards. It's all nice, people audit the code and use it with lock files. One day, the author decides to steal the data and suddenly publishes a new version and changes the tags on GitHub so that it looks like the old version. From that point in time the following happens:

1. For those who used lock files - their code stops working on CI and deploys.
2. For those who didn't use lock files - their data is stolen.
3. New users who create lock files now - they have data stolen and a false sense of security.
4. For new users who don't use lock files are equally screwed as No. 3.

This is a problem of mutability of GitHub version tags.

Possible solution would be to have deno.land/x or a service like deno.land/x to either:

1. Cache the contents of files instead of redirecting to GitHub
2. Redirect to URLs with commit hashes instead of version tags

Currently, deno.land/x seems to return the file contents instead of redirecting to GitHub. I am almost sure that it used to redirect to GitHub before.

I don't know if the current behavior is getting the original file once with a guarantee of immutability, or is it just a caching for convenience. I suppose it has no guarantees or otherwise the code for master branches would never get updated.

In any case, what you are proposing here might be done as a third party service with no need to be centralized. In fact, it would be interesting to see other services like deno.land/x to demonstrate different ideas to implement a service like this.

[fakoua]

Also it is very important to add statistics to help developers to pick top modules

[kitsonk]

The quality or usefulness of a module is not determined by its statistics, for example https://www.npmjs.com/package/isarray.

[hayd]

😮

[rsp]

@kitsonk Good point :smile:

return toString.call(arr) == '[object Array]'

https://github.com/juliangruber/isarray/blob/master/index.js#L4

[mudivili]

I install git(hub/lab), etc. hosted modules over SSH. Is it possible for deno now or later, to provide such ability to load modules that are protected by SSH keys?

[kitsonk]

Using an ssh key only validates that you are who you say you are. Why is this a requirement for being able to download a module?

[mudivili]

@kitsonk, the requirement is to download private modules. In a distributed software architecture, some modules could need to be reused in many or all services.

So far, I had used github, gitlab, bitbucket along with ssh keys to reuse those modules.

I am wondering how to securely distribute private packages across services in deno.

[kitsonk]

SSH really isn't conducive for single file fetches. GitHub supports passing an auth token as part of the URL which would give you access to a private repo (and I assume it would work with other services as well). Still that is likely not sufficient as it requires you embedding your token in the code. I opened #5239 which would mean it would be passible on the command line.

[ghost]

We discussed options for a distributed / decentralized registry for Deno in a different issue, if you're interested, have a look at https://github.com/denoland/deno_website2/issues/406.

I've started working on a registry a while ago, which should solve most of the problems we talked about. For a more detailed explanation, have a look at https://github.com/denoland/deno_website2/issues/406#issuecomment-631008536 or check out the website https://deno.to/.

It's still a work in progress, but I hope to get a first version up and running soon.

[boskiv]

Please don't forget about private registries. Many companies want to use their own private modules in projects. There is an important part of the development process. Will Deno support authorisation to load modules from private repositories ?

[t8]

I might be a little late to this conversation, and I apologize for that. I think that it would be helpful for me to mention a module registry and CDN (of sorts) that my team and I developed: https://nest.land. Nest.land solves several of the reasons why this issue was brought about. Let me address each one:

Q: How do we ensure the immutability of those packages? A: Nest.land's core lies on the blockchain. All module code is sent straight to the Arweave blockchain through something the Arweave team likes to call the "permaweb." We're looking at code as bundles of transactions now, rather than just as files. In a blockchain, deleting this transaction data is physically impossible, as it is distributed across hundreds of nodes around the world, and counting.

Q: How to we take care about the security vulnerability in them (or we don't do anything about it) A: Module code is uploaded to the permaweb, but registry data is not. This gives us a certain degree of control over what users see on our registry. Though a malicious module would be permanently accessible, we don't have to display it to the user. We're also working on archive and audit systems. The nest.land archive will be a place where registry data is "backed up" at a set interval so that if our service goes down, the registry can continue to function. The audit feature that can warn a user if a module they are importing is malicious from our CLI.

Q: Since we have a set of distributed packages from various packages, how do Deno users know which source to use for the same package A: This isn't necessarily a problem that any one person or entity can solve. In fact, I think that a major goal of Deno is to give the user flexibility with the module that they want to import and from where they want to import it. Similar to capitalism, different registries can and will coexist. I believe that the competition between this coexistence will bring better services, encourage innovation, and will benefit the end user for the lifespan of Deno.

A free, open source, and decentralized module registry/CDN is simply my take on what the Deno ecosystem could benefit from.

[lucacasonato]

deno.land/x has gotten a major upgrade recently (https://deno.land/posts/registry2), with the main change being that all code being served is now immutable. Also there are other registries like nest.land that provide even stricter immutability guarantees. I think this is mostly a userland issue and deno provides the necessary tools to prevent rogue modules (secure by default, lock files). @ameerthehacker can this be closed?

[ameerthehacker]

awesome @lucacasonato this looks greats, thanks