Major import pipeline refactorization, bug fixes, new NH/GOSSI volume support

[rfrenchseti]

This pull includes major changes to the import pipeline, miscellaneous bug fixes, cosmetic fixes, and support for the recently added NHKC and earlier GOSSI volumes.

Change list:

• Fixed #1193 (Refactor import pipeline)
• Fixed #1208 (Bugs with Cassini Original Target Name): Fixed by merging the two types of N/A results and mapping "ATLAS:" to "ATLAS"
• Fixed #1188 (get rid of profile indexes)
• Fixed #540 (index file fallback)
• Fixed #1194 (Add east longitude)
• Fixed #1198 (Add new GOSSI volumes)
• Fixed #1197 (MVIC previews)

Major changes:

• Split config_data.py into config_data.py, config_targets.py, and config_volume_info.py.
• Refactored all populate_XXX files. The entire system is now class-based. The top-level class is ObsBase, and then there are subclasses for each of the permanent tables ObsGeneral, ObsPds, ObsWavelength, ObsTypeImage, ObsProfile, ObsRingGeometry, and various versions of ObsSurfaceGeometry. These are lumped together into a single ObsCommon, which is the parent of each mission subclass, and then each instrument subclass inherits from the appropriate mission class. The overall idea is that the higher-level classes supply intelligent defaults, and subclasses only override functions as necessary, thus saving many lines of code.
• The config_targets.py file contains a list of regular expressions for volume IDs along with details on how to import them:
  • The name of the primary index file.
  • The name of the instrument subclass.
  • A flag indicating whether each row should be validated (not all volumes require this, and it speeds up the import to only do it when necessary).
• The table schemas have been changed so that all class methods have the same names as the database fields, and all database fields are computed (there are no direct references to index files or columns). This makes overriding ring and surface geo much easier.
• For many volumes, each row in the index file is "validated" before being allowed into the database. Validation is performed by taking the filespec contained in the row, converting it to an OPUS ID, and then converting the OPUS ID back to a filespec. If the original and new filespecs match, the row is considered valid. This serves several purposes:
  • For various volumes including Cassini occultations and GOSSI, there are extraneous rows in the index file for all sorts of support files or other resolutions that are not the primary filespec we want to use for importing. This validation process eliminates consideration of all of these rows. This means there is no longer need for a separate profile_index table, which accomplished essentially the same thing by hand.
  • A similar problem exists for NH, but in that case the consideration is which version of the file should be considered primary (e.g. lossy vs. lossless vs. uncompressed). Previously we simply used whatever observation occurred last in the file, but it turns out this gave preference to lower-priority observations in some cases (e.g. lossy over lossless). Since PdsFile's conversion of OPUS ID to filespec automatically returns the highest-priority product, row validation solves this problem. This has changed the results of some tests as we are now putting the correct version in the database.
  • Something similar happens with GO_0016, where there are multiple SL9 lines for a single opus id. In this case we return NULL for opus id to eliminate those lines entirely and then go read the new GO_0016_sl9_index.tab file, which includes a summary of the grouped files with min and max fields that we then put in the database.
• Under Surface Geometry Constraints, renamed the existing "Longitude" subcategory to "West Longitude" and added a new "East Longitude" subcategory with the same fields but east versions. Renamed the west fields to have "West" in their names. The east versions are computed automatically (east = 360-west, or -west for WRT). Tests were updated.

Cosmetic changes:

• Unified styling by changing primary_file_spec to primary_filespec and file_spec to filespec throughout all of OPUS.
• Unified styling by renaming database fields solar_ring_elev1/2 to solar_ring_elevation1/2 and renaming slugs RINGGEOsolarringelev1/2 to RINGGEOsolarringelevation1/2 to match RINGGEOobserverringelevation1. Removed pre-OPUS3 old_slug and replaced it with RINGGEOsolarringelev1/2 for more contemporary backwards compatibility.
• mission_abbrev has been changed to mission_id, instrument_abbrev to instrument_id, and inst_host_abbrev to inst_host_id.
• Removed old table_schemas/proto directories, which are not used.

Bug fixes:

• Fix backend bug where the images.json API call was returning URLs like https://opus.pds-rings.seti.org//holdings... with an extraneous slash.
• Fixed bugs in import of COCIRS_[01]xxx volumes:
  • Remove erroneous ring_obs_id and set to NULL.
  • Populated missing Wavenumber Resolution fields with contents of BAND_BIN_WIDTH.
  • Changed Cassini OBSERVATION_ID to be drawn from the supplemental index instead of the primary index (they are different!). This causes fields such as Activity Name, Cassini Target Name, and revno to be set correctly.
• Fixed bug with COUVIS EUV/FUV observations where the maximum wavelength was being set incorrectly (which propagated into the wave number and wave number resolution).
• Removed code that was preventing COUVIS_8001 preview diagrams from being included in the database for reasons that were lost in the mists of time but no longer appear to be valid.
• Fixed COUVIS_8001 to get its Data Set ID from the index label rather than the profile_index label, which is now out of date.
• Removed presence of Image Constraints Intensity Levels value for COVIMS when the observation is not actually an image.
• We no longer import the 2xxx versions of NH volumes. This was a holdover from when not all 1001 observations had a corresponding 2001 version, but now they are identical and PdsFile gives both 1001 and 2001 products for download. This means we no longer need to check for duplicate OPUS IDs when importing NH volumes.

Other test case changes:

• Added test case for co-uvis-occ-2009-062-thehya-e which used to have bad filenames in the holdings.
• Added new GOSSI test cases for GO_0016 normal and SL9 observations.
• Updated NHMVIC images results for sizes of new preview images.
• Some NH test results have changed because we are now using the highest-priority version (e.g. 0x630 lossless instead of 0x632 lossy).
• GOSSI image sizes have changed.

Support for new volumes:

• Added target names and mission phase names for new NHKC and GOSSI volumes.
• Made filespec matching between index and supplemental index case-insensitive.
• Needed changes in new volumes: The NH index files have FILE_SPECIFICATION_NAME entries in uppercase while the supplemental index files have them in lowercase. Import has been changed to ignore case.

[juzen2003]

• Database used: re-import
• All Django tests pass: Y
• JSHINT run on all affected files: NA
• FLAKE8 run on all affected files: Y
• Tested on Chrome, Firefox, Safari, and iOS: NA (Remember to test all browser sizes)
• Tested for race conditions (with delayed API calls if applicable): NA