search

SETI / rms-pdsfile

pdsfile Python module
Apache License 2.0
0 stars 0 forks source link

Need PdsFile.primary_data_abspath to normalize primary filespecs for all types of data #2

Open rfrenchseti opened 9 months ago

rfrenchseti commented 9 months ago

From rms-webtools created by rfrenchseti: SETI/rms-webtools#4

On Nov 7, 2018, at 1:53 PM, Rob Frenchrfrench@seti.org wrote:

OK...so fundamentally there is a mismatch here. If I read the "primary file spec" from an index file, assuming it is in the proper format (ending in .LBL), then I have no way of using PdsFile.from_filespec() to look it up and get a viable ViewSet from it, since ViewSets explicitly don't work with the .LBL extension.

So either we need to change PdsFile to look up Viewables when the extension is .LBL, or we need to change the extension of the primary file spec before sending it to PdsFile for lookup. Or is there already some automated way to ask PdsFile for the "primary data product" which DOES have a ViewSet?

There's also the problem that Cassini ISS and Galileo SSI do NOT use .LBL as the extension in the index files. That means that, in OPUS, some observations have a primary file spec ending in .LBL and some end in .IMG. Do we want to make these consistent by having the import pipeline switch the extension to all be .LBL? Or do we want to keep the OPUS database consistent with what's actually in the PDS archives?

On 11/7/2018 12:28 PM, Mark Showalter wrote:

OK, I remember now why I did this and it has to do with making sensible Viewmaster pages.

We can solve this problem by having a PdsFile attribute "primary_data_abspath" that returns the absolute path to the primary data file. Then...

pdsf = pdsfile.PdsFile.from_path(filespec) viewset = pdsfile.PdsFile.from_abspath(pdsf.primary_data_abspath).viewset

...would do the trick. The problem is that the association between a random file and the primary data file is currently not easy to make unless you turn on "set_opus_lookups()", which is slow. I can fix that.

This will be the quantity that should be used as primary filespec, no matter what appears in the label. Also, PdsFile.from_abspath(primary_filespec).viewset will return a valid viewset.

On Nov 7, 2018, at 9:45 PM, Rob French rfrench@seti.org wrote:

OK, hopefully last question - do you really want abspath stored in the OPUS database? That exposes our internal filesystem structure. Wouldn't the logical_path, or better yet the logical_path with "volumes" stripped off, be more appropriate?

yes, logical path after "volumes/".

rfrenchseti commented 9 months ago

When this is fixed, change opus/import/populate_obs_general.py function populate_obs_general_preview_images.