search

DOI-USGS / ISIS3

Integrated Software for Imagers and Spectrometers v3. ISIS3 is a digital image processing software package to manipulate imagery collected by current and past NASA and International planetary missions.
https://isis.astrogeology.usgs.gov
Other
200 stars 169 forks source link

hideal2pds export application for Ideal Camera HiRise images #1199

Closed ascbot closed 5 years ago

ascbot commented 5 years ago

Author Name: Sharmila Prasad (Sharmila Prasad)

Original Assignee: Jean Walldren

hideal2pds will allow both isis3 and non-isis3 users to associate each pixel on the PDS ideal image with the same position on the targeted body as the original Isis3 HiRise ideal cube within the ground resolution of half a pixel.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Sharmila Prasad (Sharmila Prasad) Original Date: 2012-02-03T22:52:29Z

The first draft of the application has been completed. Awaiting more information on whether to release as standard PDS product.

Note: This application generates a PDS MetaData file as extras and the isis3 cube in BSQ format.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Sharmila Prasad (Sharmila Prasad) Original Date: 2012-02-03T22:53:14Z

The first draft is ready. We will discuss more when we meet on monday.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Sharmila Prasad (Sharmila Prasad) Original Date: 2012-02-14T22:18:00Z

The application should export the binary spice blobs(table). New class ProcessExportBlob should handle the blob export.The PDS table description will be in the Label or a Format file if specified.

The PDS Metadata for now will be a PDS extra file. New PDS keywords will have to be created for missing ones after consultation with the HiRise team at UofA for complete PDS translation.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Stuart Sides (@scsides) Original Date: 2012-03-20T20:38:05Z

Debbie, do you have or can you get a required deadline for this. We are looking to establish a "Target Version". The current Triage Priority is set high because this is an active funded mission. We can change it to address the priority of the development once we get more information and re-assign too.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Debbie Cook (Debbie Cook) Original Date: 2012-04-16T22:46:23Z

Ken said it needs to be completed by the end of the fiscal year. The emails I get from the U of A folks indicate they would like it sooner if possible.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2012-04-20T20:18:59Z

Set Target Version to the August Public Release (last release before the end of FY12)

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2012-04-20T20:20:53Z

Reset the Triage Priority...was "blank" today for some reason

ascbot commented 5 years ago

Original Redmine Comment Author Name: Tammy Becker (Tammy Becker) Original Date: 2012-05-21T21:38:04Z

Stuart and Debbie, what is the status of priority on this one? Triage of 10 is pretty high...this means someone is 'dying' in emergancy-room speak...

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-09-13T18:03:43Z

================================================================================ The following is a log of email conversations for this project:

================================================================================ On 12/23/11 1:25 PM, Sharmila T Prasad wrote: Hello Rod,

  I was told to contact you regarding the "Isis3 Export and Import
  programs for Ideal Camera Images".

  Currently I am working on the Export program ideal2pds. I would like
  to know the following information:

  1. What is the information you will need from the isis3 cube
  regarding the ideal camera?

  2. What is the format of the output PDS text file for this
  application?

  3. I am assuming that the application ideal2pds has 2 parameters: a.
  Input Cube b. Exported output PDS text file. Is there any other
  parameters/options for this application?

  Thank you,

  Sharmila Prasad
  U. S. Geological Survey Astrogeology Science Center
  2255 N. Gemini Dr.
  Flagstaff, AZ 86001
  (928) 556-7308

================================================================================

================================================================================ On 12/23/11 1:53 PM, Rodney Heyd wrote: Hi Sharmila,

  I believe what we had more or less decided upon at our team meeting
  in August was a PDS raw image product with the SPICE/Ideal Camera
  Model data attached as a binary blob.

  Audrie, was that your impression?  Or is this still needed to
  (re)process EDR's with the jittery SPICE kernel?

  Rod

================================================================================

================================================================================ Date 12/29/2011 11:20 AM
That was the impression I got also. I don't think we need it to process the EDRs. I am currently able to process them using the jitter-corrected CK kernel without any Ideal Camera Model information.

  Audrie

================================================================================

================================================================================ The following emails after the task was reassigned to Jeannie Backer.......

================================================================================ Date: Tue, 14 Aug 2012 11:42:42 -0700 From: Rodney Heyd

Hi Everyone,

It has been a while since we have talked about the high precision products, but I think we're at the point where it is time to finalize these products and ensure that what we will be producing will be acceptable to the PDS (at least as extras if not official products).

In summary (and for Patty's benefit), I believe this is where we last left off:

The HiPrecision product will be a non-map projected (noproj) product in ideal camera space. This product will contain all HiRISE RED band CCD's stitched together into a mosaic. The exported file format will most likely be PDS RAW with binary SPICE blobs (I think this is where the PDS needs to be involved to ensure that the SPICE blob data is stored in an acceptable way).

Is there a time this week (say Thursday or Friday afternoon) where we could have a telecon to hash this out?

The agenda for the telecon would be as follows:

  1. Finalize the HiPrecision product file format

  2. Status updates on items needed to complete the HiPrecision pipeline a. Enable spiceinit to take smoothed ck b. Create tool to import/export products c. Add color processing to hijitter d. Add option to spiceinit to null out pixels outside of ck time coverage e. Get label update approved to distinguish HiJACK from HiNoProj products f. Update hijitter to maintain Frame kernel information in label

Thanks,

Rod

-- Rod Heyd HiRISE GDS Manager Office: (520) 626-0764 LPL, University of Arizona

================================================================================ Date: 23 Aug 2012 3:10 PM From: Jeff Anderson Subject: HiPrecision meeting notes (meeting took place August 23 at 2pm)

Something we forgot to address was the DEM file that was used in SPICEINIT and how that should be written into the PDS labels. We thought only the filename. The smoothed DEM should probably be archived too.

Thanks, Jeff

HiPrecision File Format PDS Product Non-map projected Embedded SPICE Blob Stuart: Blobs to be created as PDS Tables Rod: Should go into the extras directory Rod/Patty: Try to make them PDS compliant Rod/Patty: Use standard PDS keywords Patty: Unlikely to get new PDS3 keywords approved with PDS4 on the horizon Rod/Patty: Don't want to consider PDS4 for these products Jeannie: Programs support both attached and detached PDS tables Rod: Want's detached everything so UofA can make a JP2 Stuart: Do we want to use PDS qube, spectral cube, or image Rod: Use PDS Image object type Rod: We will convert PDS Images to JP2's Patty: Should there be a peer review? Rod: Product will be similar to the red nomap product Rod: As an extras product we don't need a formal peer review Patty: Will compare new product between red nomap product and determine if there is a need for peer review Patty: Lossy-compressed will prevent these products from ever being a official PDS product Rod: Don't want these to be lossy compressed Ken: How many people using the nomap / noproject products Rod: Don't have a good feel for how many. People using them for qualitative analysis because they are quick to view in hiview Patty: If we ever move these products up from extras we would need another SIS or an RDR SIS amendment Jeff: Can we amend the MRO dictionary if a keyword type doesn't exist? Rod: Yes that would be good Patty: Would be willing to try to get an informal approval for the PDS dictionary Stuart: If we can't find something in the PDS dictionary we will check with Patty about the MRO dictionary Jeff: What mission specific info so go in the labels Rod: Anything done by hirdrgen except map projection information Debbie: Some items many not apply Jeff: Debbie and Jeannie will discuss those items with Rod/Patty as they occur Sarah: Make sure we can read the PDS products back into ISIS for use in SOCET Jeff: Confirmed and will check with Annie Howington for any details Patty: USGS will create a proposed label for review that can be sent to Patty/Rod Jeannie: What byte order do you want? Rod: Whatever the existing format ... whatever hirdrgen does

Status for hiprecision pipeline Audrie: Hijitter doesnot maintain frame kernel in label Debbie: Fixed appjit ... spicefit was not maintaining kernel Debbie: Has to checkin spicefit which addresses acceptance of smoothed kernels and recording frame kernel Debbie: Must pad the top and bottom with spiceinit to ensure coverage. Only pad the file that was spicefit. Debbie used 10sec padding Debbie: Might make it a parameter in hijitter to allow user to specify padding? Audrie: To test to see if we need the padding options and let Debbie know. Audrie: Make label changes to identify the difference between hijittered product and no-hijittered product Debbie: USGS does not need to worry about item e All: We are in agreement that we do not need to bring the Jitter/noproj data type back into ISIS All: User will know the jitter/noproj type through the PDS label Audrie: Heard rumor that color processing will be added to hijitter Debbie: Yes ... will remove error checks that prevent running on color Audrie: Option to null out pixels at top/bottom of noproj Debbie: Option was added to hijitter Audrie: Doesn't solve the problem we need for noproj products too Audrie: Laz indicated that we could solve problem in spiceinit Jeff: We can't null pixels in spiceinit Jeff: USGS will need to figure out how to deal with this properly (Action item ... Jeff will put in a mantis ticket) Sarah: What about the agenda item "Create a tool to import/export" Debbie: Jeannie working on this as hideal2pds and pds2hideal

Other Stuart: What version of ISIS do we need to do this in Rod: Testing ISIS3.4.0 Rod: Not sure how close HiRISE is to moving to new ISIS version Rod: Will be moving to 3.4.0 in the next few months Stuart: What version would you like to see changes in? Jeff: Will 3.4.2 be okay which should be out in about 4 months Rod: Absolutely ... good goal to shoot for Stuart: Schedule sometime in November Jeff: Programmers meeting 8/24 to setup backward compatibility Jeff: New policy should be out soon Stuart: What type of ISIS build does HiRISE want Joe: Latest version of Debian Wheezy

-- Jeffery Anderson Supervisory Computer Scientist Astrogeology Science Center 2255 N Gemini Drive Flagstaff, Arizona 928-556-7167

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-09-13T18:10:21Z

================================================================================ More related emails

================================================================================ Date: 05 September, 2012 5:43 PM From: Debbie Cook

Stuart and Jeff,

When the NaifKeywords group was added to Isis, it was not folded into the noproj/IdealCamera software. For the most part the NaifKeywords group and related software in the Spice class are doing the same thing as the IdealCamera and its keywords in that they both are loading the camera specific keyword values into the Isis camera object. The Ideal camera still relies on the Naif kernel pool for loading the camera parameters. noproj loads the keywords into the instrument group in the Ideal cube labels, and the IdealCamera object reads the label and loads the keyword values into the Naif kernel pool to be exttracted by CameraFocalPlaneMap. Because the Ideal camera is using the Naif kernel pool, the Spice class requires all Naif kernels in the cube labels for Ideal camera objects. This is causing Jeannie problems when she tries to create a Spice-ready Ideal cube on import from PDS. We assumed she would not need the kernels. We might be able to fix the problem with several if statements in Spice, but I don't think that is the best way.

I think the better solution is to go back and fold in the NaifKeywords group change into noproj/IdealCamera. The label groups I pasted in below both came from an example Moc Ideal cube. Notice that the NaifKeywords group incorrectly contains information for the original instrument, not the IdealInstrument. Certainly the Trans keywords in the Instrument group could become the TRANS keywords in the NaifKeywords group. IdealCamera computes the 3 vector values from PixelPitch and whatever Trans keyword it has. I wonder if PixelPitch, FocalLength, and EphemerisTime should also be move to the NaifKeywords group. What do you think?

Thanks,
Debbie

Group = Instrument SpacecraftName = IdealSpacecraft InstrumentId = IdealCamera TargetName = Mars SampleDetectors = 1500 LineDetectors = 1 InstrumentType = LINESCAN FocalLength = 11.20537656 PixelPitch = 0.028 EphemerisTime = -69382819.360519 StartTime = 1997-10-20T10:58:37.46 StopTime = 1997-10-20T11:03:44.66 FocalPlaneXDependency = 2 TransX = 1.0 TransY = -1.0 ExposureDuration = 400.0 MatchedCube = /work/projects/progteam/dcook/ab102401.cub End_Group

Object = NaifKeywords BODY499_RADII = (3396.19, 3396.19, 3376.2) BODY_FRAME_CODE = 10014 INS-94032_FOCAL_LENGTH = 11.20537656 INS-94032_PIXEL_PITCH = 0.007 CLOCKET-94_561812335:32_COMPUTED = e62b718dca8a90c1 INS-94032_TRANSX = (0.0, 0.0, 0.007) INS-94032_TRANSY = (0.0, -0.007, 0.0) INS-94032_ITRANSS = (0.0, 0.0, -142.8571429) INS-94032_ITRANSL = (0.0, 142.8571429, 0.0) End_Object

================================================================================ Date: 06 September, 2012 5:44 PM From: Debbie Cook

Jeannie,

I asked Steven to help by taking care of the ideal camera changes so that you can focus on the import/export applications. Steven is most familiar with how the other cameras work in Isis, and the Ideal camera should resemble them as possible. If you made any changes to the Spice class, you might want to let him know what you did. You can work with Steven to make sure the NaifKeywords object in your imported cube labels match the NaifKeywords object in the noproj output cube labels after his changes.

Steven,

The Mantis issue number is 1094. Please let me know if you need any more information. You might want to also use the HiRise observation Jeannie is using for testing besides the Moc image.

Thanks,
Debbie

================================================================================

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-09-15T02:24:26Z

================================================================================ On 8/31/12 12:38 PM, Jeannie Walldren Backer wrote:

Hi Rod and Patty, I have attached a first draft of the proposed PDS label for the Isis3 program "hideal2pds". Let me know if you have comments or concerns. The label starts on the 3rd page of the attachment. The first 2 pages are a summary of the color coding in the label. Jeannie Backer USGS Astrogeology

================================================================================

On 09/07/2012 11:56 AM, Rodney Heyd wrote:

Hi Jeannie, Here's my thoughts on the keyword questions:

NOT_APPLICABLE_CONSTANT = -9998 This value is fine as-is and can be hard-coded.

PRODUCT_VERSION_ID = "HIRISE_IDEAL_CAMERA" The PRODUCT_VERSION_ID is a string value that we would supply as an argument to the program (Typically it will be an integer)

SOURCE_PRODUCT_ID = ??? If possible, the SOURCE_PRODUCT_ID, should be computed in the same way that hirdrgen does it (which is a list of all the EDR products that were used to create the final product). I'm not quite sure where the list is coming from (if we're inserting this into the label at some point during our pipeline processing, or if Isis is keeping track of it somehow). I do know that we aren't passing this list in directly to hirdrgen.

INSTRUMENT_NAME = "HIGH RESOLUTION IMAGING SCIENCE EXPERIMENT" INSTRUMENT_ID = "HIRISE_IDEAL_CAMERA" These are fine as-is, and can be hard-coded.

RATIONALE_DESC = "Possible MSL rover landing site - Nili Fossae" We'll want to pass the RATIONALE_DESC in as a command line parameter, like we do with hirdrgen.

SPACECRAFT_NAME = HIRISE_IDEAL_SPACECRAFT This sounds fine to me, and can be hard-coded.

OBJECT = IMAGE DESCRIPTION = "HiRISE mosaicked product, not projected" Let's set this to "HiRISE mosaicked product, not map projected"

We'll want to export the images as 16 bit data, and these keywords should be computed accordingly.

Finally a few questions and comments: How will the JITTER_CORRECTED keyword get set? Will we do that?

The form of the PRODUCT_ID should be something like "PSP_007556_2010_RED" or "PSP_007556_2010_COLOR" (We think we might like to eventually generate COLOR products in the future, but we're not there yet).

There's a similar issue with the table references, we just want to make sure the product_id root name is set consistently there.

Finally, (this is more for us than for you), we need to export and make available our smoothed mola cub somewhere in our PDS volume.

Sarah, and Audrie, please add anything that I missed....

Rod

-- Rod Heyd HiRISE GDS Manager Office: (520) 626-0764 LPL, University of Arizona

================================================================================

================================================================================ On 9/10/12 6:27 PM, Jeannie Walldren Backer wrote: Hi Rod,

I have a few more follow up questions/comments for your team:

For the image data, do you need 16 bit signed or 16 bit unsigned?

Do you want hideal2pds to mimic the hirdrgen application's handling of the rationale description? That is, give the user a choice of entering a value or preserving the original description in the Isis cube. If so, how would you like to handle the case in which the user chooses to preserve the rationale description but the input cube has no value for this keyword (RATIONALE_DESC = NULL)? Should the program throw an error and force the user to enter a description or just set the value to NULL as it is in the input cube?

There are two keywords in which we will need you to add to the mosaicked image in your pipeline (using editlab): "ProductId" and "SourceProductId". In the case of hirdrgen, the "SourceProductId" keyword is generated when himos is run. The handmos application does not generate a "SourceProductId" keyword, since it's a HiRise specific keyword. For "ProductId", there will be nothing in our labels to indicate whether a mosaicked image is RED or COLOR, so we will not be able to set this keyword in the hideal2pds program. The value should be reset in the input isis cube.

In hirdrgen, the units on BAND_WIDTH and CENTER_FILTER_WAVELENGTH are changed from NANOMETERS to NM or nm. Do you prefer hideal2pds to do the same? If so, do you prefer capitalized "NM" or lower-case "nm"?

Last, to answer your question, the JITTER_CORRECTED keyword will be set by the hijitter application and exported into the PDS label automatically. You will not need to do this.

Let me know when you have decided on values for DATA_SET_ID and DATA_SET_NAME.

Thanks for getting back to me on this. Let me know if there are any other questions from your team that I haven't addressed.

Jeannie

================================================================================

================================================================================ On 9/11/12 7:52 AM, Rodney Heyd wrote:

Do you want hideal2pds to mimic the hirdrgen application's handling of the rationale description? That is, give the user a choice of entering a value or preserving the original description in the Isis cube. If so, how would you like to handle the case in which the user chooses to preserve the rationale description but the input cube has no value for this keyword (RATIONALE_DESC = NULL)? Should the program throw an error and force the user to enter a description or just set the value to NULL as it is in the input cube?

Yes, let's mimic what hirdrgen does. As for the null problem, I don't think that should happen, and throwing an exception in a case where it is null seems reasonable to me. The RATIONALE_DESC shouldn't be null.

There are two keywords in which we will need you to add to the mosaicked image in your pipeline (using editlab): "ProductId" and "SourceProductId". In the case of hirdrgen, the "SourceProductId" keyword is generated when himos is run. The handmos application does not generate a "SourceProductId" keyword, since it's a HiRise specific keyword. For "ProductId", there will be nothing in our labels to indicate whether a mosaicked image is RED or COLOR, so we will not be able to set this keyword in the hideal2pds program. The value should be reset in the input isis cube.

Ok, we can do that without a problem.

In hirdrgen, the units on BAND_WIDTH and CENTER_FILTER_WAVELENGTH are changed from NANOMETERS to NM or nm. Do you prefer hideal2pds to do the same? If so, do you prefer capitalized "NM" or lower-case "nm"? Let's keep it consistent with the RDRs, so let's go with NM.

Last, to answer your question, the JITTER_CORRECTED keyword will be set by the hijitter application and exported into the PDS label automatically. You will not need to do this.

Perfect! Let me know when you have decided on values for DATA_SET_ID and DATA_SET_NAME.

Let's go with "MRO-M-HIRISE-3-CDR-V1.0" for the DATA_SET_ID and "MRO MARS HIGH RESOLUTION IMAGING SCIENCE EXPERIMENT CDR V1.0" for the DATA_SET_NAME. I think that should be good enough for now.

Rod

================================================================================

================================================================================ On 9/12/12 10:47 AM, Audrie Fennema wrote:

Will the hinoproj application be updated to set the JITTER_CORRECTED keyword to FALSE?

================================================================================

================================================================================ On 9/12/12 5:29 PM, Jeannie Walldren Backer wrote:

Hi Rod, You said you wanted the PDS image data to be 16 bit. Do you prefer signed or unsigned? Thanks, Jeannie

================================================================================

================================================================================ On 9/12/12 7:52 PM, Rodney Heyd wrote: Hi Jeannie,

Unsigned is what we want.

Thanks,

Rod

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-09-24T18:35:26Z

IMPACT New program hideal2pds will be added to Isis3. There is no expected impact on other Isis3 programs. A new class has been added to allow the export of Isis tables to PDS format.

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-09-24T18:46:10Z

Other Isis programs/classes modified for this ticket include::

classes:

ascbot commented 5 years ago

Original Redmine Comment Author Name: Jean Walldren (Jean Walldren) Original Date: 2012-11-21T23:09:58Z

From Rod Heyd at University of Arizona on November 21, 2012

Hi Jean,

Sarah and I have tested all the high points with hideal2pds and pds2hideal, and we're giving you a "go" to include it in the next release.

In testing, we uncovered several minor missing pieces in our processing of the noproj cube on our end, but I don't think there are any show stopping bugs in either the export or import programs that should prevent them from going out in the next release.

We ran into a couple of issues in attempting to create JP2 files from the exported img files, and they are our fault for not fully anticipating this problem. Our JP2 conversion software expects the input product to have an attached label, so a future update to create the image with an attached label would be a convenience for us (but shouldn't hold up the release of the program).

The other problem we ran into (which isn't an Isis problem) is that, once we re-attached the label to the img file, our jp2 conversion program got confused about the references to the table .dat files.

In short, we still have some issues to work out on our end, but I think we can generate usable products with the current programs.