search

MillionConcepts / pdr

[P]lanetary [D]ata [R]eader - A single function to read all Planetary Data System (PDS) data into Python
Other
60 stars 6 forks source link

Cannot reshape array error on reading Roseta/COSIMA product #28

Closed msbentley closed 2 years ago

msbentley commented 2 years ago

I am trying to open the below data product (FITS+detached label), however PDR is giving me an error:

/home/mbentley/Dropbox/software/pds/pdr/pdr/pdr.py:350: UserWarning: Unable to load COSISCOPE_FITS_IMAGE: cannot reshape array of size 263520 into shape (1024,256)
  warnings.warn(f"Unable to load {object_name}: {ex}")

The FITS file correctly loads into fv, and if I use the label and simply read in python 1024x256 16-bit signed integers from byte offset 8640 I also get the correct image.

http://archives.esac.esa.int/psa/ftp/INTERNATIONAL-ROSETTA-MISSION/COSIMA/RO-C-COSIMA-3-V6.0/DATA/SUB_3C3/CS_3C3_20160928T184637_IM_M.FIT http://archives.esac.esa.int/psa/ftp/INTERNATIONAL-ROSETTA-MISSION/COSIMA/RO-C-COSIMA-3-V6.0/DATA/SUB_3C3/CS_3C3_20160928T184637_IM_M.LBL

Many thanks for any hints as to if this is a pdr issue, or something unexpected in the label etc.

thareUSGS commented 2 years ago

@msbentley I'm not sure about all the allowable variants in PDS3 but seeing this raised a flag for me. ^COSISCOPE_FITS_IMAGE = ("CS_3C3_20160928T184637_IM_M.FIT",8641<BYTES>) instead of simply: ^IMAGE = ("CS_3C3_20160928T184637_IM_M.FIT",8641<BYTES>)

I guess I would consider "^IMAGE" (and a couple of others) reserved words for PDS3 images. The others I have seen are "^SPECTRAL_QUBE" or just "^QUBE" for multi-band images.

Anyway, if you change the above line above and the corresponding OBJECT section at the end, it should allow PDR and other applications to read it. If the labels are fixed, perhaps this is something PDR can check for? But really I am still unsure if that is a break from convention or not.

OBJECT                          = IMAGE
  LINES                         = 1024
  LINE_SAMPLES                  = 256
  ...
END_OBJECT                      = IMAGE
msbentley commented 2 years ago

Hi @thareUSGS I'm pretty sure that's valid, and you need it to disambiguate multiple IMAGE objects etc. e.g. from figure 4.1 in the PDS3 standards:

image

thareUSGS commented 2 years ago

okay (but ugh).

I did update ^COSISCOPE_FITS_IMAGE to ^IMAGE (and the object name) and it does fix the loading for PDR. But you are correct, PDR should be able to at least load it such that you could:

>>> import pdr
>>> data = pdr.read("CS_3C3_20160928T184637_IM_M.LBL")
>>> data['COSISCOPE_FITS_IMAGE']

This does error on the original (non-updated label), so it does look like something is amiss in PDR for reading that object as an array.

cmillion commented 2 years ago

Is this a product that used to work and doesn't now? We did a massive refactoring in the last six months, and there were certainly some regressions in supported data sets. (We also formalized what "support" and "data set" mean, to avoid regressions in the future.) Here is our living list of supported data. Rosetta is on our priority list to fully support, but we haven't gotten to it yet.

msbentley commented 2 years ago

I want to say yes, but I'm not 100% sure. I have a directory of PDS3 test products from PSA missions that I am running through pdr, and I believe I had raised tickets previously for those that failed. In any case, great that Rosetta is on your list (we have an intern starting in the spring who will hopefully use pdr to build some tutorials for Rosetta instruments)

cmillion commented 2 years ago

Ok. Well then I apologize for the regression. I promise that the end result will be better long term because of the refactor, though. And we'll let you know as soon as we have Rosetta support. We'll try to do that soon, but it probably won't happen before LPSC. If an old version of pdr did work, you should be able to use git to retrieve it.

msbentley commented 2 years ago

No problem - just to comment that I don't think this is a Rosetta-specific problem, but a PDS3 generic one - I'm pretty sure we'll hit this with MEX and VEX products also.

michaelaye commented 2 years ago

This is definitely PDS Generic because I had to come up with the pointer name grabbing code for my Index table reader as well because they are sometimes not ^TABLE but ^WHATEVER_TABLE

Get Outlook for iOShttps://aka.ms/o0ukef

From: Mark S. Bentley @.> Sent: Friday, February 25, 2022 10:03:19 AM To: MillionConcepts/pdr @.> Cc: Subscribed @.***> Subject: Re: [MillionConcepts/pdr] Cannot reshape array error on reading Roseta/COSIMA product (Issue #28)

No problem - just to comment that I don't think this is a Rosetta-specific problem, but a PDS3 generic one - I'm pretty sure we'll hit this with MEX and VEX products also.

— Reply to this email directly, view it on GitHubhttps://github.com/MillionConcepts/pdr/issues/28#issuecomment-1051029531, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAARBDRSJYYZ5GDTXRJP5GDU46Y5PANCNFSM5PJRQCXA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

m-stclair commented 2 years ago

I decided to do a quick shakedown on this -- we have handling implemented for all kinds of weird pointer names, so ^COSISCOPE_FITS_IMAGE shouldn't be a problem! The issue turned out to be FITS-specific and a very quick fix: we were not propagating pointer-filename mappings correctly through our FITS-handling routines. This should be fixed in bcdeb8cfbb94b95e3c31c27706de8e35c1756276