memmap issue

[rmg55]

Hello,

I am new the spectral python, but really am psyched about using it. I am having a problem with the open_memmap function. I ran the python -m spectral.tests.run script and got the following error. I am using the newest version of numpy (1.9.2)

Any help would be great!

------------------------------------------------------------------------
Running memmap tests.
------------------------------------------------------------------------
Testing memmaps with BIL image file.
Saving /media/sf_C_DRIVE/Courses/remote_sensing/project/testng_SPy/spectral_test_files/memmap_test_bil.img
Testing bil_memmap_read..................................... OK
Testing bil_memmap_write.................................... Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/spectral/tests/run.py", line 68, in <module>
test.run()
File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 192, in run
suite.run()
File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 186, in run
test.run()
File "/usr/local/lib/python2.7/dist-packages/spectral/tests/spytest.py", line 76, in run
method()
File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 133, in test_bil_memmap_write
mm[i, k, j] = 3 * self.value
TypeError: 'NoneType' object does not support item assignment

[donm]

What version of spectral python are you using? Did you install with pip, or are you using the master branch on github?

(Glad you're excited about SPy and that you're going to use it. @tboggs has done a great job.)

[tboggs]

Yes, please let us know what version of SPy you are using.

I'm wondering if the error could be due to a recent change in numpy. I forced a Travis CI restart (since it hasn't run in a few months) but all tests passed there (though it isn't clear what version of numpy the build systems are using). I'll try running the tests from a clean virtualenv with numpy 1.9.2 as soon as I can get on a system from which I can test.

[tboggs]

I am not able to reproduce your exception. Using python 2.7.6, I've run the unit tests against numpy 1.9.2 and 1.9.3 and all tests pass with spectral 0.16.2 as well as the current development source code.

Please try the following and provide your output:

import numpy as np
import spectral
img = spectral.open_image('92AV3C.lan')
(R, C, B) = img.shape
for mode in ['r', 'r+']:
    print 'mode =', mode
    mm = np.memmap(img.filename, dtype=img.dtype, mode=mode, offset=img.offset, shape=(R, B, C))
    print type(mm)

[tboggs]

Also, please try running the unit tests as follows to see if any others fail. If they do, please provide the output summary lines for the tests that fail.

python -m spectral.tests.run -c

[rmg55]

I installed spectral python with pip a couple weeks ago (I am running linux mint). The Spectral version is 0.16.2. Here are the code/results you requested I run.

[1]: import numpy as np

In [2]: import spectral

In [3]: img = spectral.open_image('92AV3C.lan')

In [4]: (R, C, B) = img.shape

In [5]: for mode in ['r', 'r+']:
   ...:         print 'mode =', mode
   ...:         mm = np.memmap(img.filename, dtype=img.dtype, mode=mode, offset=img.offset, shape=(R, B, C))
   ...:         print type(mm)
   ...:      
mode = r
<class 'numpy.core.memmap.memmap'>
mode = r+
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-5-a8bd01437e23> in <module>()
      1 for mode in ['r', 'r+']:
      2         print 'mode =', mode
----> 3         mm = np.memmap(img.filename, dtype=img.dtype, mode=mode, offset=img.offset, shape=(R, B, C))
      4         print type(mm)
      5 

/usr/local/lib/python2.7/dist-packages/numpy/core/memmap.pyc in __new__(subtype, filename, dtype, mode, offset, shape, order)
    255         bytes -= start
    256         offset -= start
--> 257         mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
    258 
    259         self = ndarray.__new__(subtype, shape, dtype=descr, buffer=mm,

error: [Errno 22] Invalid argument

And here are the results of python -m spectral.tests.run -c

------------------------------------------------------------------------
Running memmap tests.
------------------------------------------------------------------------
Testing memmaps with BIL image file.
Saving /media/sf_C_DRIVE/Courses/remote_sensing/project/testng_SPy/spectral_test_files/memmap_test_bil.img
Testing bil_memmap_read..................................... OK
Testing bil_memmap_write.................................... Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/spectral/tests/run.py", line 68, in <module>
    test.run()
  File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 192, in run
    suite.run()
  File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 186, in run
    test.run()
  File "/usr/local/lib/python2.7/dist-packages/spectral/tests/spytest.py", line 76, in run
    method()
  File "/usr/local/lib/python2.7/dist-packages/spectral/tests/memmap.py", line 133, in test_bil_memmap_write
    mm[i, k, j] = 3 * self.value
TypeError: 'NoneType' object does not support item assignment

Thoughts?

Thanks again!

[tboggs]

I ran my tests on Mint also. Did the other memmap unit tests (e.g., for BIP & BSQ) run successfully?

Please modify the code you ran above to print out the np.memmap arguments like so:

import numpy as np
import spectral
img = spectral.open_image('92AV3C.lan')
(R, C, B) = img.shape
for mode in ['r', 'r+']:
    print 'mode =', mode
    print 'file =', img.filename
    print 'dtype =', img.dtype
    print 'offset =', img.offset
    print 'shape =', (R, B, C)
    mm = np.memmap(img.filename, dtype=img.dtype, mode=mode, offset=img.offset, shape=(R, B, C))
    print type(mm)

And as a sanity check, perhaps you could try running the unit tests from your home directory to see if it makes a difference.

[tboggs]

Unable to reproduce this exception and haven't received any follow-up so this issue is being closed.

[matak07]

Hi, I am a little late for this conversation. I have recently started working with HSI images. I tried to execute the repo by @tboggs . I am getting the following error when it comes to running memmap tests. I would be thankful if anyone could help me.

---

Running memmap tests.

Testing memmaps with BIL image file. Saving D:\HSI_codes\classification\spectral-master\spectral-master\spectral_test_files\memmap_test_bil.img Header parameter names converted to lower case. Traceback (most recent call last): File "C:\Users\PILab_UTS\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "C:\Users\PILab_UTS\Anaconda3\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\run.py", line 68, in test.run() File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\memmap.py", line 192, in run suite.run() File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\memmap.py", line 186, in run test.run() File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\spytest.py", line 76, in run method() File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\memmap.py", line 108, in test_bil_memmap_read self.create_test_image_file() File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\tests\memmap.py", line 95, in create_test_image_file force=True) File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\io\envi.py", line 488, in save_image _write_image(hdr_file, data, metadata, **kwargs) File "D:\HSI_codes\classification\spectral-master\spectral-master\spectral\io\envi.py", line 726, in _write_image fout = builtins.open(img_file, 'wb', bufsize) OSError: [Errno 22] Invalid argument: 'D:\HSI_codes\classification\spectral-master\spectral-master\spectral_test_files\memmap_test_bil.img' Testing bil_memmap_read.....................................

This is where it gives me error.. while reading the memmap_test_bil.img file. I tried all the above solutions.

I also tried steps suggested my @tboggs . Following is the output.

mode = r <class 'numpy.core.memmap.memmap'> mode = r+ <class 'numpy.core.memmap.memmap'>

I am working on Windows10 with conda. Python 3.6 , numpy version 1.15.4 , spectral version 0.19, and pip version 18.1

I did not make any changes to the original code. But not sure why is the error. Can it be because of Python 3.6 version?

Thanks in advance!

[tboggs]

Do other unit tests pass for you? It's odd that you are getting an invalid argument error on the file name. I wonder if that could have something to do with backslashes in Windows file paths. Can you try running unit tests from the root of your D: drive and see if you get the same result?

[matak07]

Thank you @tboggs for your reply. I will try unit testing from D: drive. and will also check if the problem is because of backslashes. I will update soon.