search

DOI-USGS / ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
https://isis.astrogeology.usgs.gov
Other
195 stars 166 forks source link

Handle Voyager IMG/LBL files in voy2isis #2252

Closed ascbot closed 5 years ago

ascbot commented 5 years ago

Author Name: Brian Burns (Brian Burns)

Original Assignee: Christopher Combs

Would it be possible for voy2isis to handle IMG/LBL files from the PDS? http://pds-rings.seti.org/voyager/iss/calib_images.html

$ voy2isis from=C1625003_CALIB.LBL to=foo.cub USER ERROR Input file [C1625003_CALIB.LBL] does not appear to be a Voyager EDR.

$ voy2isis from=C1625003_CALIB.IMG to=foo.cub USER ERROR Unable to set PDS file. Decompressed input file [C1625003_CALIB.IMG] does not appear to be a PDS product.

$ voy2isis from=C1625003.IMQ to=foo.cub (works)

It looks like the PDS split out the label information from the IMQ files to LBL files, and uncompressed the images to IMG files.

I've been using the IMG/LBL files in a project and just started using ISIS - if it's an involved fix then I'll see if I can switch to using the IMQ files, though the archives are organized differently.

Thank you

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2016-09-05T18:17:55Z

Hi Brian,

I am not familiar with the "PDS rings node" version of Voyager "ISS", and, I suspect that ISIS3 voy2isis isn't either: A _CALIB.LBL and .IMG filename sounds suspiciously like processed images (modified keyword labels) that ISIS3 voy2isis currently does not expect at ingestion. I will have to ask our internal PDS node staff here in Astro what this version is all about.

As you have already pointed out, the "PDS Imaging node" version (on volume vg_0017/jupiter/c1625003.imq) with voy2isis works fine; in fact the suite of applications including radiometric calibration seems to work ok:

voy2isis from=c1625003.imq to=c1625003.cub spiceinit from=c1625003.cub findrx from=c1625003.cub remrx from=c1625003.cub to=c1625003_remrx.cub (optional step to remove the reseaus) voycal from=c1625003_remrx.cub to=c1625003_cal.cub

I will network with our internal PDS staff as well as developer staff regarding voy2isis. This most likely will require a modification to the translation table that voy2isis refers to. This modification would have to be done very carefully in order to avoid causing a problem with the imaging node version (.imq).

When this gets addressed will depend on our developer resources. We will keep you posted through this ticket. In the meantime, I hope processing the pds imaging node version will be sufficient??!

ascbot commented 5 years ago

Original Redmine Comment Author Name: Brian Burns (Brian Burns) Original Date: 2016-09-05T19:16:49Z

Thanks Tammy,

Yes, I'll see if I can switch to using the IMQ files, unless it happened to be an easy fix - I guess no one has been using ISIS with the reorganized/processed Voyager datasets, as it looks like they were created in 2012. They include four versions of each image - RAW, CLEANED (reseau marks removed), CALIB (flatfield correction), and GEOMED (geometric distortion removed).

I would need to redownload the IMG/LBL archives anyway, as I hadn't kept the LBL files, so it wouldn't be too much trouble to get the IMQ archives - the main issue is that the organization is different, but I'll see if I can work around that. If I'd known about ISIS I would have started with the IMQ archives though!

For completeness I'll include the labels from an IMQ file and a corresponding LBL file - they do seem to have changed a bit... 


Q1625003.IMQ:

CSD3ZF0000100000001NJPL3IF0PDS200000001 = SFDU_LABEL

/*          FILE FORMAT AND LENGTH */
RECORD_TYPE                      = VARIABLE_LENGTH
RECORD_BYTES                     = 836
FILE_RECORDS                     = 861
LABEL_RECORDS                    = 55

/*          POINTERS TO STARTING RECORDS OF MAJOR OBJECTS IN FILE */
^IMAGE_HISTOGRAM                 = 56
^ENCODING_HISTOGRAM              = 58
^ENGINEERING_TABLE               = 61
^IMAGE                           = 62

/*          IMAGE DESCRIPTION  */
SPACECRAFT_NAME                  = VOYAGER_1
MISSION_PHASE_NAME               = JUPITER_ENCOUNTER
TARGET_NAME                      = JUPITER
IMAGE_ID                         = '0638J1-005'
IMAGE_NUMBER                     = 16250.03 /*FLIGHT DATA SUBSYSTEM (FDS)*/
IMAGE_TIME                       = 1979-02-28T20:35:47Z
EARTH_RECEIVED_TIME              = 1979-02-28T21:12:48Z
INSTRUMENT_NAME                  = NARROW_ANGLE_CAMERA
SCAN_MODE_ID                     = '1:1'
SHUTTER_MODE_ID                  = BOTALT
GAIN_MODE_ID                     = LOW
EDIT_MODE_ID                     = '1:1' /*FULL RESOLUTION*/
FILTER_NAME                      = VIOLET
FILTER_NUMBER                    = 1
EXPOSURE_DURATION                = 0.4800 <SECONDS>
NOTE                             = "NA EAST-WEST GLOBAL THERMAL MAP IN 2 COLORS
                                   (V,O) + WA"

/*          DESCRIPTION OF THE OBJECTS CONTAINED IN FILE */
OBJECT                           = IMAGE_HISTOGRAM
 ITEMS                           = 256
 ITEM_TYPE                       = VAX_INTEGER
 ITEM_BITS                       = 32
END_OBJECT

OBJECT                           = ENCODING_HISTOGRAM
 ITEMS                           = 511
 ITEM_TYPE                       = VAX_INTEGER
 ITEM_BITS                       = 32
END_OBJECT

OBJECT                           = ENGINEERING_TABLE
 BYTES                           = 242
 ^STRUCTURE                      = 'ENGTAB.LBL'
END_OBJECT

OBJECT                           = IMAGE
 ENCODING_TYPE                   = HUFFMAN_FIRST_DIFFERENCE
 LINES                           = 800
 LINE_SAMPLES                    = 800
 LINE_SUFFIX_BYTES               = 36
 SAMPLE_TYPE                     = UNSIGNED_INTEGER
 SAMPLE_BITS                     = 8
 SAMPLE_BIT_MASK                 = 2#11111111#
 ^LINE_SUFFIX_STRUCTURE          = 'LINESUFX.LBL'
END_OBJECT
END

----------------------------------------------------------------

C1625003_CALIB.LBL:

PDS_VERSION_ID                  = PDS3
RECORD_TYPE                     = FIXED_LENGTH
RECORD_BYTES                    = 1600
FILE_RECORDS                    = 802
^VICAR_HEADER                   = ("C1625003_CALIB.IMG", 1)
^IMAGE                          = ("C1625003_CALIB.IMG", 2)
^VICAR_EXTENSION_HEADER         = ("C1625003_CALIB.IMG", 802)

DATA_SET_ID                     = "VG1/VG2-J-ISS-2/3/4/6-PROCESSED-V1.0"
PRODUCT_ID                      = "C1625003_CALIB.IMG"
PRODUCT_CREATION_TIME           = 2012-06-06T16:00:00
SOURCE_PRODUCT_ID               = ("C1625003_CLEANED.IMG",
                                   "DC1_NA_11_X100.IMG",
                                   "FICOR77_VG1_NA_VIOLET.DAT",
                                   "VGRSCF.DAT")
PRODUCT_TYPE                    = CALIBRATED_IMAGE

/* Image Description  */

INSTRUMENT_HOST_NAME            = "VOYAGER 1"
INSTRUMENT_HOST_ID              = VG1
INSTRUMENT_NAME                 = "IMAGING SCIENCE SUBSYSTEM - NARROW ANGLE"
INSTRUMENT_ID                   = "ISSN"
MISSION_PHASE_NAME              = "JUPITER ENCOUNTER"
TARGET_NAME                     = "JUPITER"
IMAGE_ID                        = "0638J1-005"
IMAGE_NUMBER                    = "16250.03"
IMAGE_TIME                      = 1979-02-28T20:35:47.00
EARTH_RECEIVED_TIME             = 1979-02-28T21:12:48
SCAN_MODE_ID                    = "1:1"
SHUTTER_MODE_ID                 = "BOTALT"
GAIN_MODE_ID                    = "LOW"
EDIT_MODE_ID                    = "1:1"
FILTER_NAME                     = "VIOLET"
FILTER_NUMBER                   = "1"
EXPOSURE_DURATION               = 0.4800 <SECOND>
START_TIME                      = 1979-02-28T20:35:46.52
STOP_TIME                       = 1979-02-28T20:35:47.00
SPACECRAFT_CLOCK_START_COUNT    = "16250:02:792"
SPACECRAFT_CLOCK_STOP_COUNT     = "16250:03:001"

NOTE                            =
"NA EAST-WEST GLOBAL THERMAL MAP IN 2 COLORS (V,O) + WA."

DESCRIPTION                     = "This image is the result of calibrating the
corresponding CLEANED image (C1625003_CLEANED.IMG). It was created using the
VICAR software package on a Compaq Alpha running OpenVMS. Routine FICOR77
subtracted a background 'dark current' image (DC1_NA_11_X100.IMG) and
converted from dimensionless raw numbers to two-byte integers that are
proportional to I/F. The REFLECTANCE_SCALING_FACTOR value listed below defines
the conversion factor.

Note that this image contains geometric distortion, so the values for the
horizontal and vertical pixel fields of view listed below are only
approximate. See the corresponding GEOMED file (C1625003_GEOMED.IMG) for a
geometrically corrected version. See file DOCUMENT/PROCESSING.TXT for more
information about the image processing."

OBJECT                          = VICAR_HEADER
  HEADER_TYPE                   = VICAR
  BYTES                         = 1600
  RECORDS                       = 1
  INTERCHANGE_FORMAT            = ASCII
  DESCRIPTION                   = "VICAR format label for the image."
END_OBJECT                      = VICAR_HEADER

OBJECT                          = IMAGE
  LINES                         = 800
  LINE_SAMPLES                  = 800
  SAMPLE_TYPE                   = LSB_INTEGER
  SAMPLE_BITS                   = 16
  SAMPLE_DISPLAY_DIRECTION      = RIGHT
  LINE_DISPLAY_DIRECTION        = DOWN
  HORIZONTAL_PIXEL_FOV          = 5.2250E-04 <DEGREE> /* approximate */
  VERTICAL_PIXEL_FOV            = 5.2250E-04 <DEGREE> /* approximate */
  HORIZONTAL_FOV                = 0.4180     <DEGREE> /* approximate */
  VERTICAL_FOV                  = 0.4180     <DEGREE> /* approximate */
  REFLECTANCE_SCALING_FACTOR    = 1.0000E-04
END_OBJECT                      = IMAGE

OBJECT                          = VICAR_EXTENSION_HEADER
  HEADER_TYPE                   = VICAR
  BYTES                         = 1600
  RECORDS                       = 1
  INTERCHANGE_FORMAT            = ASCII
  DESCRIPTION                   = "Continuation of the VICAR header."
END_OBJECT                      = VICAR_EXTENSION_HEADER

END
ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2017-03-07T21:19:06Z

Unassigned Tammy to allow for developer assignment.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2017-06-19T23:32:38Z

Blocked until Adam returns

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2017-06-20T21:56:24Z

Adam has most of this work complete, please see him before starting work on this issue

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2017-06-28T23:40:57Z

/usgs/shareall/tbecker/VoyagerRingsNode/ = A collection of test data from the PDS rings node archive. Some files in this directory represent a number of problems we have encountered from the original PDS archive that are being addressed by a separate voy2isis ticket (#4421). We are interested in seeing if this processed archive has dealt with the corrupted or unavailable keywords we have encountered with the original set.

Bad Image Id: /pds_san/PDS_Archive/Voyager/vg_0036/saturn/c4362xxx/c4362638.imq Out-of-bounds Err: /pds_san/PDS_Archive/Voyager/vg_0037/s_rings/c4391xxx/c4391013.imq Decompression Err: /pds_san/PDS_Archive/Voyager/vg_0002/uranus/c2670xxx/c2670415.imq Cannot Read Data: /pds_san/PDS_Archive/Voyager/vg_0034/saturn/c4281xxx/c4281101.imq NAIF Err: /pds_san/PDS_Archive/Voyager/vg_0009/neptune/c0986xxx/c0986348.imq (not necessarily related, still a good test under this ticket).

Also, it will be extremely important to note the stage of processing these ringsnode-archive images have been through in order for ISIS to recognize..as an example, if a user has imported a C0261155_CALIB.IMG, this image version has been calibrated, so please don't apply the ISIS voycal, or the C0261155_GEOMED.IMG, geometrically corrected (distortion correction), don't apply the ISIS noproj application and treat this image as an "ideal camera". voycal and noproj might need to recognize a keyword label?

An example document archived with this ringsnode data describing the versions of images:
http://pds-rings.seti.org/volumes/VGISS_5xxx/VGISS_5213/DOCUMENT/PROCESSING.TXT

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2017-10-16T06:29:45Z

I have tested voy2isis on the Raw version of the 'rings node version' and it creates an output cub, but there is no output instrument group on the labels. It seems there is no translation at all. With the Raw version a full ingestion should be possible to proceed with spiceinit.

I have also tested the other versions (CALIB, GEOMED, etc) and the voy2isis failure with clear message that these versions are currently not supported is sufficient for now.

work area: /work/users/tbecker/IsisTesting/M04421_voy2isis/vg_0024/europa/C2062524_RAW.cub

The input is locate: /usgs/shareall/tbecker/VoyagerRingsNode/C2062524_RAW.LBL, IMG

ascbot commented 5 years ago

Original Redmine Comment Author Name: Christopher Combs (Christopher Combs) Original Date: 2017-12-22T02:46:02Z

I've updated the voyager.trn file to include the .LBL specific keys and values.

At the moment, this is using a local version of the translation file, but I will update the official ISIS3DATA version after testing.

Built on prog24 and ready for testing at /work/projects/isis/latest/m04345_m04421.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2018-01-10T05:29:07Z

I have tested the most recent voy2isis and it now functions as planned for now. Currently, ISIS3 will only ingest the _RAW version of the RingsNodeArchive. Upon the attempt to ingest the other processed versions from this archive, the user will be notified with a message explaining that ISIS3 does not currently support it.

The _CALIB, _CLEANED, _GEOMA, _GEOMB and _RESLOC will not be supported until resources are available to handle these processed versions appropriately within the ISIS3 environment.

This will be considered a new development and design that will need to be requested in a separate request/ticket with appropriate funding.