Add index link to detail tab (Issue 981 & 323)

[juzen2003]

• Fixes #981. Fixes #323 .
• This needs to work with https://github.com/SETI/pds-webtools/pull/23
• Known problems & pending items:
  1. We need shelves/links and shelves/info for files like metadata/CORSS_8xxx/CORSS_8001/CORSS_8001_profile_index.lbl or metadata/EBROCC_xxxx/EBROCC_0001/ES1_profile_index.lbl. Because their internal_link_info is returning an empty tuple which will result in missing .lbl (only .tab exists) in the return of opus_products. It will also cause error in import when calling opus_products. And the missing info folder will raise error in import when calling file.size_bytes. For now import_for_tests will not import the following volsets due to the error mentioned above:
     • EBROCC_xxxx
     • CORSS_8xxx
     • COUVIS_8xxx
     • COVIMS_8xxx
  2. The following opus types need proper tooltips in internal_def_product_types.json:
     • hstfiles_index
     • raw_image_index
     • profile_index
     • supplemental_index
  3. Add profiling to import
  4. Fix OPUS tests

[juzen2003]

• Fixes #981 and #323
• Were any Django, import pipeline, table_schema, or dictionary files modified? Y
  • If YES:
  • Database used: New database
  • All Django tests pass: Y
  • FLAKE8 run on modified code: Y
• Were any JavaScript or CSS files modified? N
  • If YES:
  • JSHINT run on all affected files: NA
  • Tested on Chrome, Firefox, Safari, Edge, and iOS: NA (Remember to test all browser sizes)
  • Tested for race conditions (with delayed API calls if applicable): NA

Description of changes:

1. Add link to index file in detail tab, it will only show up if the row entry exists in the index file.
2. Make index files downloadable in cart tab. By default they won't be selected.

Known problems: None of index files in EBROCC_xxxx will be imported because it will have key error when calling shelf_lookup('links').

In [12]: e = pdsfile.PdsFile.from_abspath('/Users/yjchang/Dropbox/Shared-OPUS/pdsdata/holdings/metadata/EBROCC_xxxx
    ...: /EBROCC_0001/EBROCC_0001_index.tab')                                                                      

In [13]: e.shelf_lookup('links')                                                                                   
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-13-0ea328f82d78> in <module>
----> 1 e.shelf_lookup('links')

~/Desktop/seti/workspace/pds-webtools/pdsfile.py in shelf_lookup(self, id, volname)
   4082             return None
   4083 
-> 4084         return shelf[key]
   4085 
   4086     def shelf_null_key_value(self, id='info', volname=''):

KeyError: 'EBROCC_0001_index.tab'

In [14]:

[rfrenchseti]

I ran the test import on the real server with real data and got the following issues (only one example given for each):

2020-08-28 11:20:49 | pds.opus_import.main |--| WARNING | [COISS_2002 index row 1] Empty opus_product key for files: /Volumes/pdsdata-raid45
/holdings/volumes/COISS_2xxx/COISS_2002/label/vicar2.txt  /Volumes/pdsdata-raid45/holdings/volumes/COISS_2xxx/COISS_2002/data/1460960653_146
1048959/N1460960653_1.LBL  /Volumes/pdsdata-raid45/holdings/volumes/COISS_2xxx/COISS_2002/label/prefix2.fmt  /Volumes/pdsdata-raid45/holding
s/volumes/COISS_2xxx/COISS_2002/label/tlmtab.fmt

2020-08-28 11:32:37 | pds.opus_import.main |--| WARNING | [COCIRS_5408 index row 10] Empty opus_product key for files: /Volumes/pdsdata-raid
45/holdings/volumes/COCIRS_5xxx/COCIRS_5408/DOCUMENT/DATASIS.PDF  /Volumes/pdsdata-raid45/holdings/volumes/COCIRS_5xxx/COCIRS_5408/DATA/APOD
SPEC/SPEC0408020122_FP1.LBL

2020-08-28 11:32:53 | pds.opus_import.main |--| WARNING | [GO_0017 index row 31] Empty opus_product key for files: /Volumes/pdsdata-raid45/h
oldings/volumes/GO_0xxx/GO_0017/DOCUMENT/VICAR2.TXT  /Volumes/pdsdata-raid45/holdings/volumes/GO_0xxx/GO_0017/G1/IO/C0349542165R.LBL  /Volum
es/pdsdata-raid45/holdings/volumes/GO_0xxx/GO_0017/LABEL/RLINEPRX.FMT  /Volumes/pdsdata-raid45/holdings/volumes/GO_0xxx/GO_0017/LABEL/RTLMTA
B.FMT

2020-08-28 11:33:17 | pds.opus_import.main |--| WARNING | [VGISS_6210 index row 67] Empty opus_product key for files: /Volumes/pdsdata-raid4
5/holdings/volumes/VGISS_6xxx/VGISS_6210/DOCUMENT/PROCESSING.TXT  /Volumes/pdsdata-raid45/holdings/volumes/VGISS_6xxx/VGISS_6210/DATA/C43608
XX/C4360845_CALIB.LBL

2020-08-28 11:35:31 | pds.opus_import.main |--| FATAL | Import failed with exception:
Traceback (most recent call last):
  File "main_opus_import.py", line 467, in <module>
    do_import.do_import_steps()
  File "/home/rfrench/src/dave-pds-opus/opus/import/do_import.py", line 1881, in do_import_steps
    if not import_one_volume(volume_id):
  File "/home/rfrench/src/dave-pds-opus/opus/import/do_import.py", line 691, in import_one_volume
    volume_label_path)
  File "/home/rfrench/src/dave-pds-opus/opus/import/do_import.py", line 1210, in import_one_index
    obs_general_row['instrument_id'])
  File "/home/rfrench/src/dave-pds-opus/opus/import/do_import.py", line 1675, in get_pdsfile_rows_for_filespec
    products = pdsf.opus_products()
  File "/home/rfrench/src/dave-pds-webtools/pdsfile.py", line 3710, in opus_products
    self.opus_prioritizer(opus_pdsfiles)
  File "/home/rfrench/src/dave-pds-webtools/rules/NHxxxx_xxxx.py", line 367, in opus_prioritizer
    if header[0] != 'New Horizons': continue
IndexError: string index out of range

2020-08-28 11:35:34 | pds.opus_import.main |--| WARNING | [EBROCC_0001 index row 1] Empty opus_product key for files: /Volumes/pdsdata-raid4
5/holdings/volumes/EBROCC_xxxx/EBROCC_0001/DOCUMENT/MODEL.TXT  /Volumes/pdsdata-raid45/holdings/volumes/EBROCC_xxxx/EBROCC_0001/DATA/ESO1M/E
S1_EPD.LBL

2020-08-28 11:36:42 | pds.opus_import.main |--| WARNING | [COVIMS_8001 index row 242] Empty opus_product key for files: /Volumes/pdsdata-rai
d45/holdings/volumes/COVIMS_8xxx/COVIMS_8001/document/VIMS_ring_occultations_summary.pdf  /Volumes/pdsdata-raid45/holdings/volumes/COVIMS_8x
xx/COVIMS_8001/data/VIMS_2017_178_ALPCMA_E_TAU_01KM.LBL

[juzen2003]

• Add a parameter --import-report-empty-products to report empty products during import.
• Hide empty products warning messages in import by default.