search

desihub / tutorials

DESI tutorials
BSD 3-Clause "New" or "Revised" License
43 stars 17 forks source link

fiberassign exception in FiberAssignDECaLS notebook #37

Closed weaverba137 closed 1 year ago

weaverba137 commented 4 years ago

The FiberAssignDECaLS notebook attempts to run this fiberassign command:

fiberassign --overwrite --mtl mtl.fits --stdstar std.fits --sky sky.fits --footprint ./tiles.fits --outdir ./output/

The output is:

INFO: Keeping 3995 standards that aren't already in MTL
INFO: Loaded focalplane for time stamp 2020-01-30 20:45:10.723155
INFO: Focalplane has 0 fibers that are stuck / broken
INFO: Target file mtl.fits has 297129 rows.  Reading in chunks of 1000000
INFO: Read target file mtl.fits:  0.61 seconds (1 calls)
INFO: Target file /tmp/tmppyrs8vew/standards.fits has 3995 rows.  Reading in chunks of 1000000
INFO: Read target file /tmp/tmppyrs8vew/standards.fits:  0.02 seconds (1 calls)
INFO: Target file sky.fits has 165849 rows.  Reading in chunks of 1000000
INFO: Read target file sky.fits:  0.21 seconds (1 calls)
INFO: Building target tree:  0.11 seconds (1 calls)
INFO: Computing targets available to all tile / locations:  1.01 seconds (1 calls)
INFO: Computing tile / locations available to all objects:  0.45 seconds (1 calls)
INFO: Assignment constructor project targets:  0.71 seconds (1 calls)
INFO: assign unused science:  considering 5000 locations of positioner type "POS"
INFO: assign unused science:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign science targets to unused locations:  2.42 seconds (1 calls)
INFO: redist:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Redistribute science targets:  0.46 seconds (1 calls)
INFO: assign unused standard:  considering 5000 locations of positioner type "POS"
INFO: assign unused standard:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign up to 10 standard targets to unused locations:  0.11 seconds (1 calls)
INFO: assign unused sky:  considering 5000 locations of positioner type "POS"
INFO: assign unused sky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign up to 40 sky targets to unused locations:  0.24 seconds (1 calls)
INFO: assign unused suppsky:  considering 5000 locations of positioner type "POS"
INFO: assign unused suppsky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign up to 40 suppsky targets to unused locations:  0.09 seconds (1 calls)
INFO: assign force standard:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Force assignment of 10 standard targets:  0.29 seconds (1 calls)
INFO: assign force sky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Force assignment of 40 sky targets:  0.85 seconds (1 calls)
INFO: assign force suppsky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Force assignment of 40 suppsky targets:  0.00 seconds (1 calls)
INFO: assign unused science:  considering 5000 locations of positioner type "POS"
INFO: assign unused science:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign science targets to unused locations:  0.11 seconds (1 calls)
INFO: assign unused standard:  considering 5000 locations of positioner type "POS"
INFO: assign unused standard:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign standard targets to unused locations:  0.10 seconds (1 calls)
INFO: assign unused sky:  considering 5000 locations of positioner type "POS"
INFO: assign unused sky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign sky targets to unused locations:  0.13 seconds (1 calls)
INFO: assign unused suppsky:  considering 5000 locations of positioner type "POS"
INFO: assign unused suppsky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign suppsky targets to unused locations:  0.09 seconds (1 calls)
INFO: assign unused safe:  considering 5000 locations of positioner type "POS"
INFO: assign unused safe:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign safe targets to unused locations:  0.09 seconds (1 calls)
INFO: assign unused sky:  considering 20 locations of positioner type "ETC"
INFO: assign unused sky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign sky targets to unused locations:  0.01 seconds (1 calls)
INFO: assign unused suppsky:  considering 20 locations of positioner type "ETC"
INFO: assign unused suppsky:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign suppsky targets to unused locations:  0.00 seconds (1 calls)
INFO: assign unused safe:  considering 20 locations of positioner type "ETC"
INFO: assign unused safe:  working on tiles -1 (index 0) to -1 (index 14)
INFO: Assign safe targets to unused locations:  0.00 seconds (1 calls)
WARNING: Overwriting ./output/fba-000543.fits
INFO: Writing tile 543
WARNING: Overwriting ./output/fba-000544.fits
INFO: Writing tile 544
WARNING: Overwriting ./output/fba-001587.fits
INFO: Writing tile 1587
WARNING: Overwriting ./output/fba-006305.fits
INFO: Writing tile 6305
WARNING: Overwriting ./output/fba-006306.fits
INFO: Writing tile 6306
WARNING: Overwriting ./output/fba-007349.fits
INFO: Writing tile 7349
WARNING: Overwriting ./output/fba-012065.fits
INFO: Writing tile 12065
WARNING: Overwriting ./output/fba-012067.fits
INFO: Writing tile 12067
WARNING: Overwriting ./output/fba-013111.fits
INFO: Writing tile 13111
WARNING: Overwriting ./output/fba-017827.fits
INFO: Writing tile 17827
WARNING: Overwriting ./output/fba-017829.fits
INFO: Writing tile 17829
WARNING: Overwriting ./output/fba-018873.fits
INFO: Writing tile 18873
WARNING: Overwriting ./output/fba-023589.fits
INFO: Writing tile 23589
WARNING: Overwriting ./output/fba-024633.fits
INFO: Writing tile 24633
WARNING: Overwriting ./output/fba-024635.fits
INFO: Writing tile 24635
INFO: Write output files:  2.40 seconds (1 calls)
INFO: Global timer: Assignment ctor: project targets:  0.71 seconds (1 calls)
INFO: Global timer: Assignment ctor: total:  0.71 seconds (1 calls)
INFO: Global timer: force sky: per petal possible objects:  0.18 seconds (150 calls)
INFO: Global timer: force sky: total:  0.85 seconds (1 calls)
INFO: Global timer: force standard: per petal possible objects:  0.17 seconds (150 calls)
INFO: Global timer: force standard: total:  0.29 seconds (1 calls)
INFO: Global timer: force suppsky: total:  0.00 seconds (1 calls)
INFO: Global timer: redistribute science: total:  0.46 seconds (1 calls)
INFO: Global timer: run_assign_full calculation:  10.51 seconds (1 calls)
INFO: Global timer: run_assign_full write output:  2.40 seconds (1 calls)
INFO: Global timer: unused safe: assign locations:  0.00 seconds (30 calls)
INFO: Global timer: unused safe: location order:  0.09 seconds (30 calls)
INFO: Global timer: unused safe: total:  0.09 seconds (2 calls)
INFO: Global timer: unused science: assign locations:  2.39 seconds (30 calls)
INFO: Global timer: unused science: location order:  0.14 seconds (30 calls)
INFO: Global timer: unused science: total:  2.53 seconds (2 calls)
INFO: Global timer: unused sky: assign locations:  0.18 seconds (45 calls)
INFO: Global timer: unused sky: location order:  0.19 seconds (45 calls)
INFO: Global timer: unused sky: total:  0.38 seconds (3 calls)
INFO: Global timer: unused standard: assign locations:  0.02 seconds (30 calls)
INFO: Global timer: unused standard: location order:  0.18 seconds (30 calls)
INFO: Global timer: unused standard: total:  0.21 seconds (2 calls)
INFO: Global timer: unused suppsky: assign locations:  0.00 seconds (45 calls)
INFO: Global timer: unused suppsky: location order:  0.17 seconds (45 calls)
INFO: Global timer: unused suppsky: total:  0.17 seconds (3 calls)
INFO: Read mtl.fits into shared memory:  1.04 seconds (1 calls)
INFO: Read sky.fits into shared memory:  0.57 seconds (1 calls)
INFO: Read /tmp/tmppyrs8vew/standards.fits into shared memory:  0.02 seconds (1 calls)
INFO: Read sky.fits into shared memory:  0.72 seconds (1 calls)
INFO: Reading raw tile data ./output/fba-012065.fits
INFO: Reading raw tile data ./output/fba-000544.fits
INFO: Reading raw tile data ./output/fba-007349.fits
INFO: Reading raw tile data ./output/fba-013111.fits
INFO: Reading raw tile data ./output/fba-023589.fits
INFO: Reading raw tile data ./output/fba-018873.fits
INFO: Reading raw tile data ./output/fba-012067.fits
INFO: Reading raw tile data ./output/fba-017829.fits
INFO: Reading raw tile data ./output/fba-006305.fits
INFO: Reading raw tile data ./output/fba-000543.fits
INFO: Reading raw tile data ./output/fba-017827.fits
INFO: Reading raw tile data ./output/fba-024635.fits
INFO: Reading raw tile data ./output/fba-001587.fits
INFO: Reading raw tile data ./output/fba-006306.fits
INFO: Reading raw tile data ./output/fba-024633.fits
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/master/py/fiberassign/assign.py", line 981, in merge_results_tile
    tile_targets[nm][orw] = tgview[c][irw]
ValueError: setting an array element with a sequence.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/master/bin/fiberassign", line 168, in <module>
    run_merge(merge_args)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/master/py/fiberassign/scripts/merge.py", line 153, in run_merge
    copy_fba=(not args.skip_raw))
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/code/fiberassign/master/py/fiberassign/assign.py", line 1359, in merge_results
    results = pool.map(merge_tile, tile_map_list)
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/global/common/software/desi/cori/desiconda/20190804-1.3.0-spec/conda/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
ValueError: setting an array element with a sequence.

The "base" FiberAssign notebook runs similar commands without these errors. Both notebooks were run on the 'master' version of the DESI stack at NERSC.

Also note that the mtl file was created with this command:

mtl = desitarget.mtl.make_mtl(targetdata, 'DARK|GRAY')

which is a change I made myself because the second argument to make_mtl() is now required. The value is just a guess.

tskisner commented 4 years ago

This looks like the input science target file has a binary table column containing vector elements. I thought I had fixed this case in the past, but apparently there is something different about his MTL. I'll look into it.

weaverba137 commented 4 years ago

Another data point. This is the output of the notebook cell where make_mtl() is called. Maybe the exception is caused by the absence of MWS & BGS?

computing mtl
INFO:mtl.py:161:make_mtl: 79796 of 484445 targets have priority zero, setting N_obs=0.
INFO:mtl.py:197:make_mtl: Done...t=3.3s
MWS_TARGETS: 0
BGS_TARGETS: 0
DESI_TARGETS: 297129
finished computing mtl
weaverba137 commented 1 year ago

I believe FiberAssignDECaLS is working, at least as of recently. Open a separate ticket if other problems with that notebook are found.