pypi / warehouse

The Python Package Index
https://pypi.org
Apache License 2.0
3.52k stars 945 forks source link

Security Notification Systems for Python Packages #798

Open philippeowagner opened 8 years ago

philippeowagner commented 8 years ago

Having the following scenario: We are open sourcing apps from time to time and maintaining them as well ;-). Our most popular Django app is django-hijack. Days back we had a little security vulnerability in hijack, nothing really problematic but it was an issue. We had no real plan how to communicate to all the users that runs the app it in production.... However, it's not just us – others have this problem too!

Fixing a security vulnerability is one thing, but communicating it is something different. Honestly we had no real tools for that, something like a mailing list for example. It feels wrong using mailing lists for this use-case anyway. How to organise them? One for all our apps, one for the GitHub organisation, one for each app/project we open source?

A great thing would be to automatically subscribe to the apps/packages I do "pip install", "subscribe" or download from the CheeseShop. In case an app has a serious security vulnerability I would be notified by my channel of choice linked to my PyPI account. Without talking about the technical details how this could be done, do you Donald et al. think this is a legit use-case that could be added to this platform and adds value for others?

I look forward for your feedback.

nlhkabu commented 8 years ago

@dstufft I have been talking to @philippeowagner about this by email. I personally think it is a great idea, so long as users are provided with enough control to manage their notifications, both when they install a package and later via Warehouse. :+1:

sephii commented 8 years ago

I would be very interested by this feature. Also at the moment there's no way to tell whether a version of a package you're using has any known security vulnerability or not. The only information you can currently get is "am I running the latest version of this package?" and not "am I running a secure version of this package?".

This would also help security checks made by tools such as requires.io, which as far as I know rely on manual checking at the moment.

sigmavirus24 commented 8 years ago

Along similar lines, perhaps some metadata along the lines of This release fixes CVE-201Y-ABCD. And the ability to search by CVE-ID?

ewdurbin commented 8 years ago

Could a feature be introduced into pip which blocks pip from installing a release marked as vulnerable on an opt-in basis?

Similar to the way --allow-external was introduced.

Explicitly declaring that vulnerable packages are installed would give a method for application developers to be quickly alerted... assuming sane CI environments.

Anecdotally... I have learned a bunch about the consistent progress made with setuptools and pip by adding python -m pip install -U setuptools pip to my Travis builds and waiting for a red build :)

ewdurbin commented 8 years ago

To clarify, I feel that the external hosting issue is a weaker security concern that fits in the same realm as "known bad" releases.

They are a kind of META-meta-data that are perfect candidates for mutable state

apollo13 commented 8 years ago

Yes, we just discussed this at "django under the hood" and it would be great to have something in that area!

brainwane commented 6 years ago

Thanks for your note and sorry for the slow response!

The folks working on Warehouse have gotten funding to concentrate on improving and deploying Warehouse, and have kicked off work towards our development roadmap -- the most urgent task is to improve Warehouse to the point where we can redirect pypi.python.org to pypi.org so the site is more sustainable and reliable. Since this feature isn't something that the legacy site has, I've moved it to a future milestone.

Thanks and sorry again for the wait.

eirnym commented 5 years ago

@brainwane I think task to redirect old PyPi and use a new one is done and many people would prefer to have a security and vulnerability management as it was done in npm, FreeBSD pkgng or other systems.

This would save a world from problems to keep tracking many projects out from security problems.

brainwane commented 5 years ago

From December 2017 till the end of April 2018, PyPI had funding to get the new site up and running and perform the switchover. We did finish that task. The grant ran out and we have, as far as I know, no one paid to work on PyPI; volunteers are maintaining and improving the software and infrastructure sides of things, but we need dedicated funding to add complex features. The Packaging Working Group is seeking donations and applying for further grants to fund more design work, more and faster development and code review, and requisite project management. One of the grants we've applied for is specifically to fund security work. Sorry for the wait.

brainwane commented 5 years ago

Good news! The Packaging Working Group, seeking donations and further grants to fund more work, got some new funding from the Open Technology Fund, and an auditable event log -- tracked in #5863 -- is part of the current grant-funded project.

To do security notifications on PyPI packages the right way, we should wait till we have #5863 implemented, so we can draw on the event logging and use it to trigger this kind of notification.

eirnym commented 5 years ago

This is the excellent news, @brainwane! I'm looking forward for the implementation!

brainwane commented 5 years ago

@SantiagoTorres Per our conversation earlier this week, would you like to share a thought about the possibility of using in-toto for this?

brainwane commented 4 years ago

Now unblocked! See #5863 for related issues and requests for notifications of specific kinds of events, and #6339 for events currently being logged, and #5714 for related conversation.

MVrachev commented 4 years ago

I would be very interested by this feature. Also at the moment there's no way to tell whether a version of a package you're using has any known security vulnerability or not. The only information you can currently get is "am I running the latest version of this package?" and not "am I running a secure version of this package?".

There are a multiple difficulties and questions when regarding the question "is this package secure". Questions like:

There is the other possibility as suggested in the begging of this issue - let the maintainers have a flag to mark their packages as vulnerable. But then there is the other question: If a package is not marked as vulnerable from its maintainers does it mean it's safe?

This approach could be misleading for the users of the packages if there isn't clear information that a package could be marked as vulnerably only from the maintainers and if it's not marked it doesn't mean it's safe.