Crash when ending with a Pipeline object

[jeffkinnison]

Within a pipeline main loop, ending on a pipeline will crash the program if the output of the last operation is not the proper input for florin.join.

The following code should reproduce the bug on any system:

import argparse

import florin
import florin.conncomp as conncomp
import florin.morphology as morphology
import florin.thresholding as thresholding

def parse_args(args=None):
    p = argparse.ArgumentParser()

    p.add_argument('--input', type=str, help='path to input dataset')
    p.add_argument('--output', type=str, help='path to output segmentation')
    p.add_argument('--ndnt-shape', type=int, nargs='*', help='shape of tiles')
    p.add_argument('--threshold', type=float, help='NDNT threshold value')
    p.add_argument('--tile-shape', type=int, nargs='*', help='shape of tiles')
    p.add_argument('--tile-stride', type=int, nargs='*', help='stride between tiles')
    p.add_argument('--offset', type=int, nargs='*', help='offset from (0, 0, 0)')
    p.add_argument('--offset-shape', type=int, nargs='*',
                 help='shape of the offset block')

    return p.parse_args(args)

def main():
    args = parse_args()

    pipeline = florin.Serial(
        florin.load(key='merged'),
        florin.tile(shape=args.tile_shape, stride=args.tile_stride),
        florin.MPI(
            thresholding.ndnt(shape=args.ndnt_shape, threshold=args.threshold),
            morphology.binary_opening(),
            morphology.remove_small_holes(area_threshold=20),
        ),
        florin.MPI(
            conncomp.label(),
            conncomp.regionprops(),
            florin.Multithread(
                florin.classify(
                    florin.bounds_classifier(
                        'cell',
                        area=(100, 300),
                        width=(10, 20),
                        height=(10, 20),
                        depth=(10, 20)
                    ),
                    florin.bounds_classifier('vasculature')
                )
            ),
            florin.reconstruct(),
            florin.save(args.output)
        )
    )

    pipeline.run(args.input)

main()

This generates the traceback

Traceback (most recent call last):
  File "/opt/intel/python/2017.0.035/intelpython35/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/intel/python/2017.0.035/intelpython35/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/soft/datascience/mpi4py/lib/python3.5/site-packages/mpi4py/futures/__main__.py", line 72, in <module>
    main()
  File "/soft/datascience/mpi4py/lib/python3.5/site-packages/mpi4py/futures/__main__.py", line 60, in main
    run_command_line()
  File "/soft/datascience/mpi4py/lib/python3.5/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/opt/intel/python/2017.0.035/intelpython35/lib/python3.5/runpy.py", line 254, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/opt/intel/python/2017.0.035/intelpython35/lib/python3.5/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/opt/intel/python/2017.0.035/intelpython35/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "srb_pipeline.py", line 70, in <module>
    main()
  File "srb_pipeline.py", line 66, in main
    pipeline(args.input)
  File "/projects/connectomics_aesp/kinnison/florin/theta-target/florin/pipelines/pipeline.py", line 64, in __call__
    return self.run(data)
  File "/projects/connectomics_aesp/kinnison/florin/theta-target/florin/pipelines/serial.py", line 27, in run
    else next(result)
  File "/projects/connectomics_aesp/kinnison/florin/theta-target/florin/compose.py", line 32, in execute
    return g(f(x))
  File "/projects/connectomics_aesp/kinnison/florin/theta-target/florin/closure.py", line 70, in delayed
    return func(*innerargs, **kwargs)
  File "/projects/connectomics_aesp/kinnison/florin/theta-target/florin/tiling.py", line 132, in join_tiles
    for tile, metadata in tiles:
ValueError: too many values to unpack (expected 2)

This should be handled by the improvements suggested in #6, which should handle the whole reason florin.join was designed the way it is and auto-added to certain parts of the pipeline in the first place.

[jeffkinnison]

Note: The pipeline will in fact complete when this happens. The crash happens after the last operation in the pipeline, which means that this is sort of a non-bug bug that just sets an error return code when Python exits.

Also, it will happen with any combination of different Pipeline subclasses given that the hierarchy looks like the code sample.

[jeffkinnison]

Fixed as of #9