Source tarball with absolute or parent-traversing paths not handled

mbargull commented 6 years ago

Actual Behavior

If a source tarball includes absolute paths or paths like sub/../../file-in-parent-dir that traverse parent directories higher than the tarball's root dir, conda-build extracts those files and/or fails ungracefully.

Expected Behavior

Don't extract files outside of extraction directory.
Remove absolute prefixes.
Catch and gracefully fail if a resolved path of, e.g., untar-dir/sub/../../some-thing-outside-untar-dir, lies outside of untar-dir, i.e., maybe check for something like commonprefix([untar_dir, join(untar_dir, normpath(tar_member_path))]) == untar_dir

Steps to Reproduce

``` docker run --rm --entrypoint '' -it condaforge/linux-anvil /bin/bash -c " . /opt/conda/bin/activate set -x conda update -qy conda && conda update -qy --all >/dev/null && conda install -qy conda-build=3 mkdir -p /evil-build/{recipe,content} cd /evil-build/content touch /root.evil ../parent.evil ../parent-from-sub.evil tar -czPf ../content.tar.gz /root.evil ../parent.evil cd .. tar -tPf content.tar.gz cat > recipe/meta.yaml <<'EOF' package: name: evil-build source: url: file:///evil-build/content.tar.gz build: script: touch some-file EOF find / -name '*.evil' -exec rm {} \; conda-build recipe find / -name '*.evil' 2>/dev/null " ```

``` + conda update -qy conda # All requested packages already installed. # packages in environment at /opt/conda: # conda 4.3.34 py36_0 conda-forge + conda update -qy --all + conda install -qy conda-build=3 # All requested packages already installed. # packages in environment at /opt/conda: # conda-build 3.8.0 py36_0 conda-forge + mkdir -p /evil-build/recipe /evil-build/content + cd /evil-build/content + touch /root.evil ../parent.evil ../parent-from-sub.evil + tar -czPf ../content.tar.gz /root.evil ../parent.evil + cd .. + tar -tPf content.tar.gz /root.evil ../parent.evil + cat + find / -name '*.evil' -exec rm '{}' ';' find: `/proc/tty/driver': Permission denied + conda-build recipe Adding in variants from internal_defaults INFO:conda_build.variants:Adding in variants from internal_defaults Attempting to finalize metadata for evil-build INFO:conda_build.metadata:Attempting to finalize metadata for evil-build BUILD START: ['evil-build--0.tar.bz2'] Source cache directory is: /opt/conda/conda-bld/src_cache No hash (md5, sha1, sha256) provided. Source download forced. Add hash to recipe to use source cache. WARNING:conda_build.source:No hash (md5, sha1, sha256) provided. Source download forced. Add hash to recipe to use source cache. Downloading source to cache: content.tar.gz Downloading file:///evil-build/content.tar.gz Success Extracting download Traceback (most recent call last): File "/opt/conda/bin/conda-build", line 6, in sys.exit(conda_build.cli.main_build.main()) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main execute(sys.argv[1:]) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute verify=args.verify) File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 199, in build notest=notest, need_source_download=need_source_download, variants=variants) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 2105, in build_tree notest=notest, File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1310, in build try_download(m, no_download_source=False, raise_error=True) File "/opt/conda/lib/python3.6/site-packages/conda_build/render.py", line 470, in try_download source.provide(metadata) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 623, in provide timeout=metadata.config.timeout, locking=metadata.config.locking) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 157, in unpack folder = os.path.join(tmpdir, flist[0]) IndexError: list index out of range + find / -name '*.evil' /opt/conda/conda-bld/parent.evil /root.evil ```

``` docker run --rm --entrypoint '' -it condaforge/linux-anvil /bin/bash -c " . /opt/conda/bin/activate set -x conda update -qy conda && conda update -qy --all >/dev/null && conda install -qy conda-build=3 mkdir -p /evil-build/{recipe,content} cd /evil-build/content touch /root.evil ../parent.evil ../parent-from-sub.evil mkdir sub tar -czPf ../content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil cd .. tar -tPf content.tar.gz cat > recipe/meta.yaml <<'EOF' package: name: evil-build source: url: file:///evil-build/content.tar.gz build: script: touch some-file EOF find / -name '*.evil' -exec rm {} \; conda-build recipe find / -name '*.evil' 2>/dev/null " ```

``` + conda update -qy conda # All requested packages already installed. # packages in environment at /opt/conda: # conda 4.3.34 py36_0 conda-forge + conda update -qy --all + conda install -qy conda-build=3 # All requested packages already installed. # packages in environment at /opt/conda: # conda-build 3.8.0 py36_0 conda-forge + mkdir -p /evil-build/recipe /evil-build/content + cd /evil-build/content + touch /root.evil ../parent.evil ../parent-from-sub.evil + mkdir sub + tar -czPf ../content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil + cd .. + tar -tPf content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil + cat + find / -name '*.evil' -exec rm '{}' ';' find: `/proc/tty/driver': Permission denied + conda-build recipe Adding in variants from internal_defaults INFO:conda_build.variants:Adding in variants from internal_defaults Attempting to finalize metadata for evil-build INFO:conda_build.metadata:Attempting to finalize metadata for evil-build BUILD START: ['evil-build--0.tar.bz2'] Source cache directory is: /opt/conda/conda-bld/src_cache No hash (md5, sha1, sha256) provided. Source download forced. Add hash to recipe to use source cache. WARNING:conda_build.source:No hash (md5, sha1, sha256) provided. Source download forced. Add hash to recipe to use source cache. Downloading source to cache: content.tar.gz Downloading file:///evil-build/content.tar.gz Success Extracting download Traceback (most recent call last): File "/opt/conda/bin/conda-build", line 6, in sys.exit(conda_build.cli.main_build.main()) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 420, in main execute(sys.argv[1:]) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 411, in execute verify=args.verify) File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 199, in build notest=notest, need_source_download=need_source_download, variants=variants) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 2105, in build_tree notest=notest, File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1310, in build try_download(m, no_download_source=False, raise_error=True) File "/opt/conda/lib/python3.6/site-packages/conda_build/render.py", line 470, in try_download source.provide(metadata) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 623, in provide timeout=metadata.config.timeout, locking=metadata.config.locking) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 143, in unpack tar_xf(src_path, tmpdir) File "/opt/conda/lib/python3.6/site-packages/conda_build/utils.py", line 632, in tar_xf t.extractall(path=dir_path) File "/opt/conda/lib/python3.6/tarfile.py", line 2008, in extractall numeric_owner=numeric_owner) File "/opt/conda/lib/python3.6/tarfile.py", line 2050, in extract numeric_owner=numeric_owner) File "/opt/conda/lib/python3.6/tarfile.py", line 2112, in _extract_member os.makedirs(upperdirs) File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs mkdir(name, mode) FileExistsError: [Errno 17] File exists: '/opt/conda/conda-bld/tmppuh33cyv/sub/../..' + find / -name '*.evil' /opt/conda/conda-bld/parent.evil /root.evil ```

similar output for conda-build 2

``` docker run --rm --entrypoint '' -it condaforge/linux-anvil /bin/bash -c " . /opt/conda/bin/activate set -x conda update -qy conda && conda update -qy --all >/dev/null && conda install -qy conda-build=2 mkdir -p /evil-build/{recipe,content} cd /evil-build/content touch /root.evil ../parent.evil ../parent-from-sub.evil tar -czPf ../content.tar.gz /root.evil ../parent.evil cd .. tar -tPf content.tar.gz cat > recipe/meta.yaml <<'EOF' package: name: evil-build source: url: file:///evil-build/content.tar.gz build: script: touch some-file EOF find / -name '*.evil' -exec rm {} \; conda-build recipe find / -name '*.evil' 2>/dev/null " ```

``` + conda update -qy conda # All requested packages already installed. # packages in environment at /opt/conda: # conda 4.3.34 py36_0 conda-forge + conda update -qy --all + conda install -qy conda-build=2 Package plan for installation in environment /opt/conda: The following NEW packages will be INSTALLED: pycrypto: 2.6.1-py36_1 conda-forge The following packages will be DOWNGRADED: conda-build: 3.8.0-py36_0 conda-forge --> 2.1.18-py36_0 conda-forge + mkdir -p /evil-build/recipe /evil-build/content + cd /evil-build/content + touch /root.evil ../parent.evil ../parent-from-sub.evil + tar -czPf ../content.tar.gz /root.evil ../parent.evil + cd .. + tar -tPf content.tar.gz /root.evil ../parent.evil + cat + find / -name '*.evil' -exec rm '{}' ';' find: `/proc/tty/driver': Permission denied + conda-build recipe BUILD START: evil-build--0 Source cache directory is: /opt/conda/conda-bld/src_cache Downloading source to cache: content.tar.gz Downloading file:///evil-build/content.tar.gz INFO:fetch.start:('content.tar.gz', 135) INFO:fetch.update:135 INFO:fetch.stop:None Success Extracting download Traceback (most recent call last): File "/opt/conda/bin/conda-build", line 6, in sys.exit(conda_build.cli.main_build.main()) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 342, in main execute(sys.argv[1:]) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 333, in execute noverify=args.no_verify) File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 97, in build need_source_download=need_source_download, config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1524, in build_tree config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1064, in build assert not need_source_download, "Source download failed. Please investigate." AssertionError: Source download failed. Please investigate. + find / -name '*.evil' /opt/conda/conda-bld/evil-build_1522678903456/parent.evil /root.evil ```

``` docker run --rm --entrypoint '' -it condaforge/linux-anvil /bin/bash -c " . /opt/conda/bin/activate set -x conda update -qy conda && conda update -qy --all >/dev/null && conda install -qy conda-build=2 mkdir -p /evil-build/{recipe,content} cd /evil-build/content touch /root.evil ../parent.evil ../parent-from-sub.evil mkdir sub tar -czPf ../content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil cd .. tar -tPf content.tar.gz cat > recipe/meta.yaml <<'EOF' package: name: evil-build source: url: file:///evil-build/content.tar.gz build: script: touch some-file EOF find / -name '*.evil' -exec rm {} \; conda-build recipe find / -name '*.evil' 2>/dev/null " ```

``` + conda update -qy conda # All requested packages already installed. # packages in environment at /opt/conda: # conda 4.3.34 py36_0 conda-forge + conda update -qy --all + conda install -qy conda-build=2 Package plan for installation in environment /opt/conda: The following NEW packages will be INSTALLED: pycrypto: 2.6.1-py36_1 conda-forge The following packages will be DOWNGRADED: conda-build: 3.8.0-py36_0 conda-forge --> 2.1.18-py36_0 conda-forge + mkdir -p /evil-build/recipe /evil-build/content + cd /evil-build/content + touch /root.evil ../parent.evil ../parent-from-sub.evil + mkdir sub + tar -czPf ../content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil + cd .. + tar -tPf content.tar.gz /root.evil ../parent.evil sub/../../parent-from-sub.evil + cat + find / -name '*.evil' -exec rm '{}' ';' find: `/proc/tty/driver': Permission denied + conda-build recipe BUILD START: evil-build--0 Source cache directory is: /opt/conda/conda-bld/src_cache Downloading source to cache: content.tar.gz Downloading file:///evil-build/content.tar.gz INFO:fetch.start:('content.tar.gz', 160) INFO:fetch.update:160 INFO:fetch.stop:None Success Extracting download Traceback (most recent call last): File "/opt/conda/bin/conda-build", line 6, in sys.exit(conda_build.cli.main_build.main()) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 342, in main execute(sys.argv[1:]) File "/opt/conda/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 333, in execute noverify=args.no_verify) File "/opt/conda/lib/python3.6/site-packages/conda_build/api.py", line 97, in build need_source_download=need_source_download, config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1524, in build_tree config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/build.py", line 1062, in build config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/render.py", line 86, in parse_or_try_download source.provide(metadata, config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 519, in provide unpack(metadata, config=config) File "/opt/conda/lib/python3.6/site-packages/conda_build/source.py", line 91, in unpack tar_xf(src_path, config.work_dir) File "/opt/conda/lib/python3.6/site-packages/conda_build/utils.py", line 344, in tar_xf t.extractall(path=dir_path) File "/opt/conda/lib/python3.6/tarfile.py", line 2008, in extractall numeric_owner=numeric_owner) File "/opt/conda/lib/python3.6/tarfile.py", line 2050, in extract numeric_owner=numeric_owner) File "/opt/conda/lib/python3.6/tarfile.py", line 2112, in _extract_member os.makedirs(upperdirs) File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs mkdir(name, mode) FileExistsError: [Errno 17] File exists: '/opt/conda/conda-bld/evil-build_1522679008835/work/sub/../..' + find / -name '*.evil' /opt/conda/conda-bld/evil-build_1522679008835/parent.evil /root.evil ```

`Output of conda info`

Current conda install:

               platform : linux-64
          conda version : 4.3.34
       conda is private : False
      conda-env version : 4.3.34
    conda-build version : 3.8.0
         python version : 3.6.5.final.0
       requests version : 2.18.4
       root environment : /opt/conda  (writable)
    default environment : /opt/conda
       envs directories : /opt/conda/envs
                          /root/.conda/envs
          package cache : /opt/conda/pkgs
                          /root/.conda/pkgs
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /root/.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.34 requests/2.18.4 CPython/3.6.5 Linux/4.14.29-1-MANJARO CentOS/6.9 glibc/2.12    
                UID:GID : 0:0

mbargull commented 6 years ago

xref: https://github.com/bioconda/bioconda-recipes/pull/8492, https://github.com/alekseyzimin/masurca/issues/12

msarahan commented 6 years ago

PRs would be welcome here. Otherwise, this is a pretty crazy edge case, and I'm inclined to say that conda-build should not be expected to be completely robust against any pathological tarball (or other source) input.

mbargull commented 6 years ago

I think it would be nice to make sure conda-build handles the input it processes itself (i.e., excluding everything happening in the build scripts, etc.) cautiously. But I won't be able to look into this anytime soon. @jamesabbott, if you have the means and capacity to look into this, I think you'd want to look at conda_build.utils.tar_xf. There, t.extractall(...) could take an additional members=some_func(...) argument where some_func iterates over the t, while modifying/checking the members' paths. Maybe, conda_build.utils.unzip and other functions have to be checked, too; not sure, though.

jamesabbott commented 6 years ago

I'll take a look but can't promise anything...I'm not familiar with the conda internals but will have a stab at it. It doesn't sound like there is much of a push for the pathological tarballs to be fixed.

jamesabbott commented 6 years ago

Sorry for the delay - been moving house from one end of the country to the other...

I have a PR (#2822) which I think will work for tarballs, and had something similar for zip files but have not actually worked out how to create zipfiles with evil paths in them since zip seems to sanitise these, so haven't included this in the PR since it was completely untested.

The approach is iterate through each tar member if it's path is absolute, replace with a path relative to '/' if the 'realpath' doesn't start from cwd then strip any occurrences of '../' from the path if the 'realpath' still doesn't start from cwd due to some circumstance I haven't thought of, then die Once all paths have been checked, the normal 'extractall' method can be called on the tarfile object.

This could result in the relative locations between different components of the extracted tarball differing to that which was intended, but I think is far safer than allowing things to be written outside the intended destination.

The test suite runs as well as it did before I changed anything (17 failures on my machine...) so I don't think I've broken anything...

github-actions[bot] commented 2 years ago

Hi there, thank you for your contribution!

This issue has been automatically locked because it has not had recent activity after being closed.

Please open a new issue if needed.

Thanks!

conda / conda-build