map_async causing issue when doing CNMF using ipyparallel

[ethanbb]

Your setup:

1. Operating System (Linux, MacOS, Windows): Linux
2. Hardware type (x86, ARM..) and RAM: x86, 128 GB
3. Python Version (e.g. 3.9): 3.11.8
4. Caiman version (e.g. 1.9.12): 1.11.0
5. Which demo exhibits the problem (if applicable): N/A
6. How you installed Caiman (pure conda, conda + compile, colab, ..): conda + compile
7. Details:

I'm getting the following error when running CNMF with ipyparallel:

Traceback (most recent call last):                                                                     
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 712, in __getattr__                                                                           
    return self.__getitem__(key)                                                                       
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                               
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in __getitem__                                                                           
    values = [md[key] for md in self._metadata]                                                                                                                                                                
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 699, in <listcomp>                                                                            
    values = [md[key] for md in self._metadata]                                                                                                                                                                
              ~~^^^^^                                                                                  
KeyError: 'sort'                                   

During handling of the above exception, another exception occurred:                                                                                                                                            

Traceback (most recent call last):                                                                     
  File "<stdin>", line 1, in <module>                                                                  
  File "/mnt/bigdata/eblackwood/2p_analysis/cmcode/caiman_analysis.py", line 559, in do_cnmf                                                                                                                   
    self.cnmf_fit2 = self.cnmf_fit1.refit(images, dview=cluster_info['dview'])                                                                                                                                 
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 405, in refit                                              
    return cnm.fit(images)                                                                             
           ^^^^^^^^^^^^^^^                                                                             
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 522, in fit                                                
    self.update_spatial(Yr, use_init=True)                                                             
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/cnmf.py", line 914, in update_spatial                                     
    update_spatial_components(Y, C=self.estimates.C, f=self.estimates.f, A_in=self.estimates.A,                                                                                                                
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 238, in update_spatial_components                       
    A_ = threshold_components(A_, dims, dview=dview, medw=medw, thr_method=thr_method,                                                                                                                         
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         
  File "/synology/code/CalciumImaging/CaImAn/build/__editable__.caiman-1.10.5-cp311-cp311-linux_x86_64/caiman/source_extraction/cnmf/spatial.py", line 511, in threshold_components                            
    res.sort(key=lambda x: x[1])                                                                       
    ^^^^^^^^                                       
  File "/home/ethan/mambaforge/envs/caiman/lib/python3.11/site-packages/ipyparallel/client/asyncresult.py", line 714, in __getattr__                                                                           
    raise AttributeError(                          
AttributeError: 'AsyncMapResult' object has no attribute 'sort'. Did you mean: 'abort'?  

This seems to be because of the following code in cnmf/spatial.py, which calls map_async on the dview and then tries to sort the return value:

    if dview is not None:
        if 'multiprocessing' in str(type(dview)):
            res = dview.map_async(
                threshold_components_parallel, pars).get(4294967)
        else:
            res = dview.map_async(threshold_components_parallel, pars)
    else:
        res = list(map(threshold_components_parallel, pars))

    res.sort(key=lambda x: x[1])

I think most of the other code uses map_sync for the ipyparallel case - is that what was intended here? Oddly, this code doesn't seem to have changed recently; seems like someone would have run into this already.

[pgunn]

Can you tell me what version of ipyparallel you have? I'm going to look over changes to the APIs to see if they may have changed it.

[ethanbb]

This was on version 8.7.0.

[pgunn]

Given that we're sorting it right after, I don't think there's a good reason for that not to be map_sync; it's unfortunate that the async object doesn't implement the rest of a map api by immediately loading everything if a call requires everything. Maybe it did that in the past, but the level of inheritance used in ipyparallel makes looking over various versions of the package a lot of work.

The return value for map_sync should be sortable. If you'd like to turn this into a PR after verifying it fixes the issue, that'd be great, but if you'd rather me do it, that's also fine.

[ethanbb]

OK thanks, let me verify and let you know. I know what you mean about looking over the ipyparallel code...it's not very readable.

[ethanbb]

All good, submitted the PR.