regression: closes not-owned handle (Trac #2178)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/2178 on 2012-07-02 by atmention:yarikoptic, assigned to unknown.

While troubleshooting http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=669466 which used to build/test just fine I ran to conclusion that it is numpy's regression. Following snippet

#!/usr/bin/python
import numpy as np
print "Numpy version ", np.__version__
from tempfile import mktemp
outfile_name = mktemp(suffix='.np')
outfile = open(outfile_name, 'w')
np.savez(outfile, data='LOVELY LOAD')
outfile.close()

print outfile_name
outfile = open(outfile_name, 'r', 10000)
outfile.seek(0)
print "We seek before just fine"
print np.load(outfile)['data']
print "Can we seek after?"
outfile.seek(0) # Only needed here to simulate closing & reopening file
print "We could seek after as well"

produces on Debian squeeze

$> python /tmp/numpy-closingbug.py
Numpy version  1.4.1
/home/yoh/.tmp/tmpQltbIC.np
We seek before just fine
LOVELY LOAD
Can we seek after?
We could seek after as well

but on Debian sid

$> /tmp/numpy-closingbug.py
Numpy version  1.6.2
/home/yoh/.tmp/tmpdbfXKA.np
We seek before just fine
LOVELY LOAD
Can we seek after?
Traceback (most recent call last):
  File "/tmp/numpy-closingbug.py", line 20, in <module>
    outfile.seek(0) # Only needed here to simulate closing & reopening file
ValueError: I/O operation on closed file

and with current master:

$> PYTHONPATH=$HOME/proj/numpy /tmp/numpy-closingbug.py
Numpy version  1.8.0.dev-e15d0bd
/home/yoh/.tmp/tmpg1uQA4.np
We seek before just fine
LOVELY LOAD
Can we seek after?
Traceback (most recent call last):
  File "/tmp/numpy-closingbug.py", line 20, in <module>
    outfile.seek(0) # Only needed here to simulate closing & reopening file
ValueError: I/O operation on closed file

[numpy-gitbot]

atmention:yarikoptic wrote on 2012-07-02

a bit of the stack/trace for real-world example where in numpy.lib.npyio.NpzFile.close() (called from del) it would close the handle which it doesn't own... may be because somewhere it first assigned the same original FID to the temporary zipfile:

$> PYTHONPATH=$PWD:/home/yoh/proj/numpy nosetests -s -v test/unittests/test_files.py:test_NumpyBinaryFile
/usr/bin/nosetests:5: UserWarning: Module paste was already imported from None, but /usr/lib/python2.7/dist-packages is being added to sys.path
  from pkg_resources import load_entry_point
/usr/lib/python2.7/dist-packages/tables/__init__.py:59: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from tables.utilsExtension import getPyTablesVersion, getHDF5Version
/usr/lib/python2.7/dist-packages/tables/file.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from tables import hdf5Extension
/usr/lib/python2.7/dist-packages/tables/link.py:32: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from tables import linkExtension
/usr/lib/python2.7/dist-packages/tables/table.py:40: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from tables import tableExtension
/usr/lib/python2.7/dist-packages/tables/index.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from tables import indexesExtension
unittests.test_files.test_NumpyBinaryFile ... (/home/yoh/deb/gits/pkg-exppsy/pynn/test/unittests/test_files.py:107):  test_NumpyBinaryFile
107     metadata_ = nbf.get_metadata()
(Pydb) print nbf.fileobj
<open file 'tmp.npz', mode 'r' at 0x2c2c150>
(Pydb) b /usr/lib/python2.7/zipfile.py:1157
Breakpoint 1 set in file /usr/lib/python2.7/zipfile.py, line 1157.
(Pydb) c
(/usr/lib/python2.7/zipfile.py:1157):  close
1157         if self.fp is None:
(Pydb) print self.fp
<open file 'tmp.npz', mode 'r' at 0x2c2c150>
(Pydb) bt
-> 0 close(self=<zipfile.ZipFile instance at 0x2d6aa28>) called from file '/usr/lib/python2.7/zipfile.py' at line 1157
## 1 close(self=<numpy.lib.npyio.NpzFile object at 0x2dd3a90>) called from file '/home/yoh/proj/numpy/numpy/lib/npyio.py' at line 212
## 2 __del__(self=<numpy.lib.npyio.NpzFile object at 0x2dd3a90>) called from file '/home/yoh/proj/numpy/numpy/lib/npyio.py' at line 220
## 3 test_NumpyBinaryFile() called from file '/home/yoh/deb/gits/pkg-exppsy/pynn/test/unittests/test_files.py' at line 107
## 4 runTest(self=unittests.test_files.test_NumpyBinaryFile) called from file '/usr/share/pyshared/nose/case.py' at line 197
## 5 run(self=unittests.test_files.test_NumpyBinaryFile, result=<nose.result.TextTestResult run=1 errors=0 f...) called from file '/usr/lib/python2.7/unittest/case.py' at line 327
## 6 __call__(self=unittests.test_files.test_NumpyBinaryFile, *args=(<nose.result.TextTestResult run=1 errors=0 f...) called from file '/usr/lib/python2.7/unittest/case.py' at line 391
## 7 runTest(self=Test(unittests.test_files.test_NumpyBinaryFile), result=<nose.result.TextTestResult run=1 erro...) called from file '/usr/share/pyshared/nose/case.py' at line 151
## 8 run(self=Test(unittests.test_files.test_NumpyBinaryFile), result=<nose.result.TextTestResult run=1 erro...) called from file '/usr/share/pyshared/nose/case.py' at line 133
## 9 __call__(self=Test(unittests.test_files.test_NumpyBinaryFile), *arg=(<nose.result.TextTestResult run=1 error...) called from file '/usr/share/pyshared/nose/case.py' at line 45
## 10 run(self=<nose.suite.ContextSuite context=unittests.test_files>, result=<nose.result.TextTestResult run...) called from file '/usr/share/pyshared/nose/suite.py' at line 223
## 11 __call__(self=<nose.suite.ContextSuite context=unittests.test_files>, *arg=(<nose.result.TextTestResult run=...) called from file '/usr/share/pyshared/nose/suite.py' at line 176
## 12 run(self=<nose.suite.ContextSuite context=unittests.test_files>, result=<nose.result.TextTestResult run...) called from file '/usr/share/pyshared/nose/suite.py' at line 223
## 13 __call__(self=<nose.suite.ContextSuite context=unittests.test_files>, *arg=(<nose.result.TextTestResult run=...) called from file '/usr/share/pyshared/nose/suite.py' at line 176
## 14 run(self=<nose.core.TextTestRunner object at 0x2108fd0>, test=<nose.suite.ContextSuite context=unittest...) called from file '/usr/share/pyshared/nose/core.py' at line 61
## 15 runTests(self=<nose.core.TestProgram object at 0x1cf19d0>) called from file '/usr/share/pyshared/nose/core.py' at line 197
## 16 __init__(self=<nose.core.TestProgram object at 0x1cf19d0>, module=None, defaultTest='.', argv=['/usr/bin/nos...) called from file '/usr/lib/python2.7/unittest/main.py' at line 95
## 17 __init__(self=<nose.core.TestProgram object at 0x1cf19d0>, module=None, defaultTest='.', argv=None, testRunn...) called from file '/usr/share/pyshared/nose/core.py' at line 118
## 18 <module> file '/usr/bin/nosetests' at line 9
(Pydb) up
(/home/yoh/proj/numpy/numpy/lib/npyio.py:212):  close
212             self.zip.close()
(Pydb) 
(/home/yoh/proj/numpy/numpy/lib/npyio.py:220):  __del__
220         self.close()
(Pydb) 
(/home/yoh/deb/gits/pkg-exppsy/pynn/test/unittests/test_files.py:107):  test_NumpyBinaryFile
107     metadata_ = nbf.get_metadata()
(Pydb) down
(/home/yoh/proj/numpy/numpy/lib/npyio.py:220):  __del__
220         self.close()
(Pydb) 
(/home/yoh/proj/numpy/numpy/lib/npyio.py:212):  close
212             self.zip.close()
(Pydb) l
207             """
208             Close the file.
209     
210             """
211             if self.zip is not None:
212  ->             self.zip.close()
213                 self.zip = None
214             if self.fid is not None:
215                 self.fid.close()
216                 self.fid = None
(Pydb) print self.fid
<open file 'tmp.npz', mode 'r' at 0x2c2c150>
*(Pydb) print self.zip.fp == self.fid
True

[numpy-gitbot]

atmention:yarikoptic wrote on 2012-07-02

my "fix" submitted on github: https://github.com/numpy/numpy/pull/328