Problems with 19.9 release

[weaverba137]

Really only one serious problem, and a few questions.

• [X] dc17a-truth.ipynb: No problems.
• [x] FiberAssign.ipynb: Command-line fiberassign program raised an exception, see below.
• [x] FiberAssignDECaLS.ipynb: Probably has similar problems.
• [x] FiberAssignMocks.ipynb: Probably has similar problems.
• [X] Intro_to_DESI_Spectra.ipynb: No problems.
• [ ] redshift-database.ipynb: Redshift database tutorial does not incorporate 'bright' files: targets-bright.fits, truth-bright.fits.
• [X] simulating-desi-spectra.ipynb: No problems.
• [X] survey-simulations.ipynb: No problems.
• [X] redrock/RedrockOutputs.ipynb: No problems.
• [ ] redrock/RedrockBOSSDemo.ipynb: Is this relevant?

Other notes:

• desitarget tag 0.33.2 has desitarget.__version__ = '0.33.2.dev3577'.
• We are not running on a tagged version of speclite, speclite.__version__ = '0.9dev514'.
• The notebook server(s) do not appear to have $SCRATCH defined. Had to switch to $CSCRATCH. Reported this to NERSC Consulting.

Exception reported in FiberAssign.ipynb

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/1.2.0/lib/python3.6/site-packages/fiberassign-1.2.0-py3.6-linux-x86_64.egg/fiberassign/assign.py", line 1131, in merge_results_tile
    skymon[field] = outdata[field][sky_rows]
ValueError: no field of name BRICKID
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/1.2.0/bin/fiberassign", line 4, in <module>
    __import__('pkg_resources').run_script('fiberassign==1.2.0', 'fiberassign')
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1453, in run_script
    exec(code, namespace, namespace)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/1.2.0/lib/python3.6/site-packages/fiberassign-1.2.0-py3.6-linux-x86_64.egg/EGG-INFO/scripts/fiberassign", line 168, in <module>
    run_merge(merge_args)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/1.2.0/lib/python3.6/site-packages/fiberassign-1.2.0-py3.6-linux-x86_64.egg/fiberassign/scripts/merge.py", line 153, in run_merge
    copy_fba=(not args.skip_raw))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/1.2.0/lib/python3.6/site-packages/fiberassign-1.2.0-py3.6-linux-x86_64.egg/fiberassign/assign.py", line 1341, in merge_results
    results = pool.map(merge_tile, tile_map_list)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ValueError: no field of name BRICKID

[weaverba137]

@sbailey, I think the fiberassign problem may simply be a change in command-line options. Can you take a look?

[forero]

@weaverba137 I fixed FiberAssignDECaLS.ipynb. I will check the other two FiberAssign notebooks.

[weaverba137]

@sbailey, did #34 fix all the open problems in this ticket? Should we also create a 19.10 tag?

[weaverba137]

@sbailey, as I mentioned on email, this is one of the issues in project 20.3 that is just a question that needs to be answered. Could you please take a look at this?

[sbailey]

PR #34 fixed the fiberassign problems (which I have now checked off), but did not do anything with redshift-database.ipynb or the redrock tutorials. OK with me to make a 19.10 tag as-is, close this ticket, and re-review the current state of the tutorials in prep for a 20.x release.

[sbailey]

Regarding redrock/RedrockBOSSDemo.ipynb : if it still works, great. If it doesn't but it is easy to fix (e.g. a change of command line options or similar), let's keep it alive. If it is broken for some complicated reason we can deprecate it.