2.7rc1 tarfile.py: `bltn_open(targetpath, "wb")` -> IOError: Is a directory

[5bf59da1-86ec-4384-896c-68455c40bd5f]

BPO	8958
Nosy	@gustaebel, @vstinner

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/gustaebel' closed_at = created_at = labels = ['library', 'type-crash'] title = '2.7rc1 tarfile.py: `bltn_open(targetpath, "wb")` -> IOError: Is a directory' updated_at = user = 'https://bugs.python.org/srid' ``` bugs.python.org fields: ```python activity = actor = 'srid' assignee = 'lars.gustaebel' closed = True closed_date = closer = 'lars.gustaebel' components = ['Library (Lib)'] creation = creator = 'srid' dependencies = [] files = [] hgrepos = [] issue_num = 8958 keywords = [] message_count = 7.0 messages = ['107419', '107479', '107493', '107540', '107798', '107803', '107805'] nosy_count = 3.0 nosy_names = ['lars.gustaebel', 'vstinner', 'srid'] pr_nums = [] priority = 'normal' resolution = 'wont fix' stage = None status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue8958' versions = ['Python 2.7'] ```

[5bf59da1-86ec-4384-896c-68455c40bd5f]

1. Find an OSX 10.5.8 machine 
2. wget http://hntool.googlecode.com/files/hntool-0.1.1.tar.gz
3. $ python2.7 -c "import tarfile as T; t=T.open('hntool-0.1.1.tar.gz'); t.extractall()"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 2046, in extractall
    self.extract(tarinfo, path)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 2083, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 2159, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/tarfile.py", line 2198, in makefile
    target = bltn_open(targetpath, "wb")
IOError: [Errno 21] Is a directory: './hntool-0.1.1/hntool'

...

Note that, when extracted open via other tools, hntool-0.1.1.tar.gz contains within it ... both a directory named "HnTool" (note the case) and a file called "hntool".

[460c80ff-bdb7-4416-a811-ee58995bd9ed]

Unfortunately I do not have access to an OS X machine. Is this problem specific to 2.7rc1 or are other versions affected as well? I thought the OS X filesystem was case sensitive ...

[5bf59da1-86ec-4384-896c-68455c40bd5f]

On 2010-06-10, at 1:06 PM, Lars Gustäbel wrote:

Is this problem specific to 2.7rc1

Yes.

or are other versions affected as well?

Nope, at least ... I know that 2.6 doesn't have this problem.

[460c80ff-bdb7-4416-a811-ee58995bd9ed]

I found the problem. As of r76780 the default for the TarFile.errorlevel argument changed from 0 (suppress errors and write them to the debug log instead) to 1 (raise exceptions for fatal extraction errors). This change was not backported to the 2.6 branch back then (it was blocked in r76781).

This means, that Python 2.6 does not succeed either, but the error is simply suppressed.

Ergo, this is no regression but a simple filesystem issue.

[5bf59da1-86ec-4384-896c-68455c40bd5f]

I am curious as to why this should still "fail" as the OSX filesystem is case sensitive. Finder has no problems with extrating this particular tarball. Do you think this is a (separate) bug, or is this by design (why?)?

If this is by design, does it make sense to raise a custom exception from TarError, instead of the generic IOError?

[460c80ff-bdb7-4416-a811-ee58995bd9ed]

a) The point is: the operation simply wouldn't fail on a case-sensitive filesystem. There is no platform-specific or otherwise special code in TarFile.makefile(). It simply tries to extract the file and the filesystem layer says no, because it believes there is already a directory with the same name. The same thing happens on a Windows filesystem BTW. The problem boils down to this:

>>> os.mkdir("A")
>>> open("a", "w")
IOError: [Errno 21] Is a directory: 'a'

And IIUC, the Mac OS X filesystem is case-preserving(!) by default, with the possibility to create new filesystems as case-sensitive. As I said, my Mac OS X expertise is almost nonexistent, you might as well ask someone with more knowledge on Python on Mac OS X.

b) I don't know what Finder does with that archive, but I cannot think of any other way than either not to extract the file at all or to extract it under a different name or to remove the directory first and then extract the file. Could you please examine how Finder extracts this archive?

c) IMHO the IOError exception is perfectly fine, because this kind of issue is outside of tarfile's scope. We hit a filesystem limit here. Also, there is no decent way to work around this problem, and I think there is no need to either.

[5bf59da1-86ec-4384-896c-68455c40bd5f]

On 2010-06-14, at 10:43 AM, Lars Gustäbel wrote:

Lars Gustäbel \lars@gustaebel.de\ added the comment:

a) The point is: the operation simply wouldn't fail on a case-sensitive filesystem. There is no platform-specific or otherwise special code in TarFile.makefile(). It simply tries to extract the file and the filesystem layer says no, because it believes there is already a directory with the same name. The same thing happens on a Windows filesystem BTW. The problem boils down to this:

>>> os.mkdir("A") >>> open("a", "w") IOError: [Errno 21] Is a directory: 'a'

And IIUC, the Mac OS X filesystem is case-preserving(!) by default, with the possibility to create new filesystems as case-sensitive. As I said, my Mac OS X expertise is almost nonexistent, you might as well ask someone with more knowledge on Python on Mac OS X.

Ah, I see. I didn't know about this compatibility quirk.

b) I don't know what Finder does with that archive, but I cannot think of any other way than either not to extract the file at all or to extract it under a different name or to remove the directory first and then extract the file. Could you please examine how Finder extracts this archive?

I was wrong about Finder extracting the tarball; actually I used the Finder on my macbook (10.6), where open("a", "w") succeeds. Whereas this issue seems to be reproducible in 10.5 or 10.4 only.

I did try to use tar this time, and it failed:

$ tar zxf hntool-0.1.1.tar.gz 
tar: hntool-0.1.1/hntool: Cannot open: File exists
tar: Error exit delayed from previous errors

c) IMHO the IOError exception is perfectly fine, because this kind of issue is outside of tarfile's scope. We hit a filesystem limit here. Also, there is no decent way to work around this problem, and I think there is no need to either.

In this case, yes ... I agree.