sandy-utils

processing bits attached to the sandy (and other) data processing streams.

This toolset is intended to provide a (fairly) uniform interface for processing GINA's NRT data.

basic structure (ruby gem style):

• sandy-utils
  • bin
  • cris_awips.rb
    • SNPP CRIS L1 -> AWIPS
  • rtstps.rb
    • Does Raw -> L0/RDR processing for SNPP, NOAA20 TERRA, and AQUA.
  • snpp_edr.rb
    • SNPP sdr -> edr
  • snpp_sdr.rb
    • SNPP and NOAA20 rdr -> sdr
  • viirs_awips.rb
    • SNPP VIIRS L1 -> AWIPS
  • awips_scmi.rb
    • generates scmi tiles for awips
  • nucaps_l2.rb
    • nucaps l2 generation
  • nucaps_sdr.rb
    • generates CRIS and ATMS sdrs bundled together for nucaps.
  • p2g_geotif.rb
    • generates geotifs slices for each sensor (MODIS, VIIRS, etc)
  • feeder_geotif.rb
    • takes geotifs slices from p2g_geotif.rb and generates nice geotifs for feeder or gis clients
  • lib
  • processingframework.rb (includes everything)_
  • processing_framework/
    • version.rb
    • other shared stuff
  • config Each tool in bin has a config file
  • cris_awips.yml
  • rtstps.yml
  • snpp_edr.yml
  • snpp_sdr.yml
  • viirs_awips.yml
  • awips_scmi.yml

Notes

Each tool should provide a "--help" option showing a brief description of what it does, and how to use it.

[jcable@spam sandy-utils]$ ./bin/rtstps.rb --help
Usage:
    rtstps.rb [OPTIONS]

  This tool does CCSDS unpacking using Rtstps for SNPP, AQUA, and TERRA.

Options:
    -c, --config config           The config file. Using ./bin/../config/rtstps.yml as the default. (default: "./bin/../config/rtstps.yml")
    -i, --input input             The input file.
    -h, --help                    print help
    -b, --basename basename       The basename of the data to be processed. For example npp.14300.1843 .  Appended to --outputdir if included.
    -o, --outdir outdir           The output directory. If --basename is included, it is appended to this.
    -t, --tempdir tempdir         The temp directory, used for working space. A sub directory will made inside this named after the basename. (default: $PROCESSING_TEMPDIR)

Common arguments

• tempdir
  • working or temp directory to use. Also controlled via $PROCESSING_TEMPDIR .
• outdir
  • output directory
• config
  • overrides the use of the config/util.yml config file, where util is the name of the util
• processors
  • The number of processors to use. Also controlled via $PROCESSING_NUMBER_OF_CPUS .

Deploy updates

1. After merging pull request into master bump version

   rake -T version # to see available commands for bumping version
   rake version:bump:minor
   git push && git push --tags

2. Build new habitat package

   hab studio enter
   build
   exit # if successfull

3. Upload package to s3

   rake s3

4. Update test and/or prod environment with pkg_artifact and pkg_sha256sum values output from the s3 upload command

   vim PATH_TO_GINA_CHEF/environments/near-real-time-test.json
   # update the utils section like so
   # "utils": {
   #        "package": "uafgina-sandy-utils-1.2.8-20170312231424-x86_64-linux.hart",
   #        "checksum": "588fc6d85ccc749785054b8a58cb985f48cb0813e8dad5a743ace2729b6bddf5"
   #      }

5. Upload environment to chef server

6. Commit environment to git

7. DANGER Update test environment with latest changes

   DON'T DO THIS IS IN PRODUCTION

   # knife ssh 'chef_environment:near-real-time-test' -x USERNAME 'sudo chef-client' -C 1