Error on opening files from Cassini

putza commented 6 years ago

I am trying to execute imdata = PDS3Image.open('./Data/02NASA_Cassini/N1454725799_1.IMG') in my Jupyter notebook, and I see the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-7-ed1161877cab> in <module>()
----> 1 imdata = PDS3Image.open('./Data/02NASA_Cassini/N1454725799_1.IMG')

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in open(cls, filename)
     90         else:
     91             with open(filename, 'rb') as fp:
---> 92                 return cls(fp, filename)
     93 
     94     def __init__(self, stream_string_or_array, filename=None, compression=None):

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in __init__(self, stream_string_or_array, filename, compression)
    119 
    120             #: A numpy array representing the image
--> 121             self.data = self._load_data(stream_string_or_array)
    122 
    123     def __repr__(self):

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in _load_data(self, stream)
    198 
    199     def _load_data(self, stream):
--> 200         if self.data_filename is not None:
    201             return self._load_detached_data()
    202 

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/image.py in data_filename(self)
    172     def data_filename(self):
    173         """Return detached filename else None."""
--> 174         return self._data_filename
    175 
    176     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/pds3image.py in _data_filename(self)
    299     @property
    300     def _data_filename(self):
--> 301         return self._image_pointer.filename
    302 
    303     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/planetaryimage-0.4.1-py3.5.egg/planetaryimage/pds3image.py in _image_pointer(self)
    312     @property
    313     def _image_pointer(self):
--> 314         return Pointer.parse(self.label['^IMAGE'], self.record_bytes)
    315 
    316     @property

/home/andput/anaconda3/envs/py35/lib/python3.5/site-packages/pvl/_collections.py in __getitem__(self, key)
    121         if isinstance(key, INDEX_TYPES):
    122             return self.__items[key]
--> 123         return dict_getitem(self, key)[0]
    124 
    125     def __delitem__(self, key):

KeyError: '^IMAGE'

My default Python evironemnt is

Notebook Executed:   2017-09-12 21:39:34.026349
================================================================================
Python Version:
--------------------------------------------------------------------------------
3.5.3 |Anaconda custom (64-bit)| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
================================================================================
PlanetaryImaging Version:
--------------------------------------------------------------------------------
0.4.1 (also tried with 0.5.0 with the same error message)
================================================================================
PVL Version:
--------------------------------------------------------------------------------
0.2.0
================================================================================

I will try some debugging over night, but would really appreciate any help.

percurnicus commented 6 years ago

@putza This is related to #75

There is a detached label and in order to open the image you need to have the label too. Download the lablel, which can also be found here, to the same directory as the N1454725799_1.IMG file and then open the image with the label path.

I got it to work but let me know if you still have an issue with it

>>> from planetaryimage import PDS3Image
>>> imagedata = PDS3Image.open('Data/02NASA_Cassini/N1454725799_1.LBL') 
>>> imagedata.image.shape
(1042, 1024)
>>> imagedata.data_filename
'N1454725799_1.IMG'

putza commented 6 years ago

Thanks for getting back to me this fast. Loading the label file works. I do not get the error message anymore, but the image looks like pure noise. I will clean up my virtual environment, reinstall and try again.

thareUSGS commented 6 years ago

That image is pretty much noise. Perhaps try this image (from the same archive directory), which is a nice view - cropped below. https://pds-imaging.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2001/data/1454725799_1455008789/N1455008789_1.LBL https://pds-imaging.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2001/data/1454725799_1455008789/N1455008789_1.IMG n1455008789_1_crop

putza commented 6 years ago

I tried N1455008789_1.LBL and W1837766238_1. I got the LBL, IMG and JPEG file. The JPEG displays correctly, but the LBL has this striped structure. Something is going wrong during the binary import.

I have created a Jupyter Notebook to demonstrate:

W1837766238_1:

N1455008789_1:

muyajil commented 6 years ago

Does anybody have an update on this? I have the same issue when opening images from HiRISE. Is this a problem with how I am opening the file?

percurnicus commented 6 years ago

@muyajil @putza Sorry, I'm going to try to figure this out this week. I think it has to do with how the data is stored in the file and the label should have some information to extract the image correctly. But this will take some time reading through documentation. Hope to have a note book viewer example by the end of the weekend

tjslezak commented 6 years ago

Here is some information I previously collected in response to the initial issue noted by @putza. I've been meaning to post this - I don't want @pbvarga1 to waste his time with the work I've already done. The solution to the issue lies in the documentation and highlighted sections I've gathered below, and is summarized at the end of the post. At present I don't have time to actively develop an explicit solution (MS thesis crunch time!), but this should help provide the necessary "leads" to resolve the issue. Hope this helps!

Instrument Overview

https://pds-imaging.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2001/catalog/isswa_inst.cat

See from Section "Data Paths" to end.

Vicar data and detached PDS label SIS

https://pdsimg.jpl.nasa.gov/data/cassini/cassini_orbiter/coiss_2103/document/edrsis.pdf (updated)

See Table 6
really the whole document
Section 11 - Appendix D
Section 12 - Appendix E

Software available for Cassini image processing

(alternatives to the previously proprietary CISSCAL IDL routine)

http://pds-rings.seti.org/cassini/iss/software.html (Cassini-specific) https://www-mipl.jpl.nasa.gov/vicar_open.html https://www-mipl.jpl.nasa.gov/external/vicar.html https://github.com/nasa/VICAR/blob/master/vos/docsource/vicar/VICAR_build_2.0.pdf

Summary:

In general, issues seem to originate from Table conversion of bit depths for the imagery. 12 bit to 8 bit lossless conversion from 16 bit analog stored from sensor. Lossy or lossless compression may follow to 8 bit. Number of header lines consisting of [0,0,0,0......]'s in label defined by e.g, ("N12345678_1.IMG",4) where 4 is the line number in the data where the image object data begins. So image data starts at the fourth row down in this image. Three rows of [0,0,0,0......] preceed it.

putza commented 6 years ago

Thank you @tjslezak. This is very helpful. My new job left me with very little time to follow up on this, but there is a long weekend coming up. I will have an update soon.

Chandler commented 5 years ago

did anyone ever get to the bottom of this? seeing the same striped images as @putza with Cassini images.

godber commented 5 years ago

I haven't kept up with this issue nor with the progress in this project but I seem to recall that we didn't initially add support for these:

OBJECT = LINE_PREFIX_TABLE
      INTERCHANGE_FORMAT = BINARY
      ROWS = 1024
      COLUMNS = 7
      ROW_BYTES = 24
      ROW_SUFFIX_BYTES = 1024
      ^LINE_PREFIX_STRUCTURE = "PREFIX2.FMT"
END_OBJECT = LINE_PREFIX_TABLE
OBJECT = IMAGE
      LINES = 1024
      LINE_SAMPLES = 1024
      SAMPLE_BITS = 8
      SAMPLE_TYPE = SUN_INTEGER
      LINE_PREFIX_BYTES = 24
END_OBJECT = IMAGE

The progressively stripey appearance in your images is what the un-interpreted line suffix/prefix look like.

thareUSGS commented 5 years ago

perhaps this might be a good opportunity to see how planetaryimage and gdal (as optional) can be somehow used in concert. BTW, for gdal image reads, to grab the whole label for Python, we still heavily rely on the pvl library. Anyway, it seems impossible to get plaentaryimage or gdal to work an all PDS3 labels so perhaps we can use both to cover more use cases. Planetaryimage is still a good default, and if we can catch a failed read, then bounce to gdal to give it go...? gdal might also help with larger images, map projection capabilities, and other formats (there is now even some support for PDS4 labels). @putza - great notebook. I took it and tested gdal's "Open" on the labels and it somehow managed to support your examples. https://gist.github.com/thareUSGS/859b3b0d0764abdc3088b297fa039757

Chandler commented 5 years ago

wow I had no ideal gdal could handle these kind of images.

by extension, that means that the excellent 'rasterio' python library by @mapbox can also read them. I've just confirmed this.

ndarray = rasterio.open("N1455008789_1.LBL").read()

pedrohasselmann commented 3 years ago

Just to report, I'm getting a similar problem with DAWN VIR data cubes.

It seems that DAWN uses a different key (^QUBE rather than ^IMAGE) to point into their data cube from the .LBL file.

I think a good option would be to give users the chance to pass the Key as an argument.

Issue posted at PVL

planetarypy / planetaryimage