ggirelli / gpseqc

Nuclear centrality estimation from GPSeq experiment.
MIT License
2 stars 2 forks source link

problem with running the microscopy pipeline #10

Closed tomkal closed 6 years ago

tomkal commented 6 years ago

Before submitting an issue, please be sure to

This issue affects

- [ ] Centrality estimation (gpseqc_estimate)

What did you do (e.g., steps to reproduce)

I tried to run GPSeq microscopy pipeline.

What did you expect to happen?

To get beautiful YFISH plots as an output :)

What happened instead?

The pipeline failed by the segmentation step.

* Looking for nuclei *

 Current condition: "iTK226_070618_001"...
Traceback (most recent call last):
  File "/usr/local/bin/gpseq_anim", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/bicro/Desktop/ggcode/pygpseq/bin/gpseq_anim", line 388, in <module>
    gpi = gpi.run()
  File "/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/main.py", line 716, in run
    self.run_segmentation(**kwargs)
  File "/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/main.py", line 914, in run_segmentation
    [c.find_nuclei(**kwargs) for c in self.conds]
  File "/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/condition.py", line 505, in find_nuclei
    for i in range(len(self.series)))
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 789, in __call__
    self.retrieve()
  File "/usr/local/lib/python2.7/dist-packages/joblib/parallel.py", line 740, in retrieve
    raise exception
joblib.my_exceptions.JoblibTypeError: JoblibTypeError
___________________________________________________________________________
Multiprocessing exception:
...........................................................................
/usr/local/bin/gpseq_anim in <module>()
      1 #!/usr/bin/env python3
      2 # -*- coding: utf-8 -*-
      3 
      4 # ------------------------------------------------------------------------------
      5 # 
----> 6 # MIT License
      7 # 
      8 # Copyright (c) 2017 Gabriele Girelli
      9 # 
     10 # Permission is hereby granted, free of charge, to any person obtaining a copy

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/bin/gpseq_anim in <module>()
    383     OH.write("@%s\n\n" % datetime.datetime.now())
    384     OH.write("%s\n" % sys.version)
    385     OH.write(settings_string)
    386 
    387 # Start the analysis
--> 388 gpi = gpi.run()
    389 
    390 # End --------------------------------------------------------------------------
    391 
    392 ################################################################################

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/main.py in run(self=<pygpseq.anim.main.Main object>, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'correctCA': False, 'debugging': False, 'dfield': 'lamin_dnorm', ...})
    711                 self.printout('Unskipping segmentation...', 0)
    712                 self.unskip(2)
    713         
    714         if not self.is_skipped(2):
    715             # Run segmentation if not skipped
--> 716             self.run_segmentation(**kwargs)
        self.run_segmentation = <bound method Main.run_segmentation of <pygpseq.anim.main.Main object>>
        kwargs = {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'correctCA': False, 'debugging': False, 'dfield': 'lamin_dnorm', ...}
    717 
    718             # Dump
    719             fname = self.outdir + 'gpi.seg'+ kwargs['suffix'] +'.cpickle'
    720             f = open(fname, 'wb')

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/main.py in run_segmentation(self=<pygpseq.anim.main.Main object>, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'correctCA': False, 'debugging': False, 'dfield': 'lamin_dnorm', ...})
    909         kwargs['an_type'] = self.an_type
    910 
    911         # Identify nuclei
    912         self.printout('* Looking for nuclei *', 0)
    913         self.printout('', 0)
--> 914         [c.find_nuclei(**kwargs) for c in self.conds]
        c.find_nuclei = <bound method Condition.find_nuclei of <pygpseq.anim.condition.Condition object>>
        kwargs = {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'correctCA': False, 'debugging': False, 'dfield': 'lamin_dnorm', ...}
        c = <pygpseq.anim.condition.Condition object>
        self.conds = [<pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>, <pygpseq.anim.condition.Condition object>]
    915         self.printout('', 0)
    916 
    917     def unskip(self, step):
    918         """Unskips a run step that was supposed to be skipped. """

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/condition.py in find_nuclei(self=<pygpseq.anim.condition.Condition object>, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...})
    500 
    501         # Segment every series in the condition
    502         self.printout('Current condition: "' + self.name + '"...', 0)
    503         self.series = Parallel(n_jobs = ncores)(
    504             delayed(find_series_nuclei)(self, i, **kwargs)
--> 505             for i in range(len(self.series)))
        self.series = [<pygpseq.anim.series.Series object>, <pygpseq.anim.series.Series object>, <pygpseq.anim.series.Series object>]
    506 
    507     def get_nuclei(self):
    508         """Return a list of the nuclei in the condition. """
    509         nuclei = []

...........................................................................
/usr/local/lib/python2.7/dist-packages/joblib/parallel.py in __call__(self=Parallel(n_jobs=36), iterable=<generator object <genexpr>>)
    784             if pre_dispatch == "all" or n_jobs == 1:
    785                 # The iterable was consumed all at once by the above for loop.
    786                 # No need to wait for async callbacks to trigger to
    787                 # consumption.
    788                 self._iterating = False
--> 789             self.retrieve()
        self.retrieve = <bound method Parallel.retrieve of Parallel(n_jobs=36)>
    790             # Make sure that we get a last message telling us we are done
    791             elapsed_time = time.time() - self._start_time
    792             self._print('Done %3i out of %3i | elapsed: %s finished',
    793                         (len(self._output), len(self._output),

---------------------------------------------------------------------------
Sub-process traceback:
---------------------------------------------------------------------------
TypeError                                          Mon Jun 18 20:18:22 2018
PID: 223234                               Python 2.7.15rc1: /usr/bin/python
...........................................................................
/usr/local/lib/python2.7/dist-packages/joblib/parallel.py in __call__(self=<joblib.parallel.BatchedCalls object>)
    126     def __init__(self, iterator_slice):
    127         self.items = list(iterator_slice)
    128         self._size = len(self.items)
    129 
    130     def __call__(self):
--> 131         return [func(*args, **kwargs) for func, args, kwargs in self.items]
        func = <function find_series_nuclei>
        args = (<pygpseq.anim.condition.Condition object>, 0)
        kwargs = {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...}
        self.items = [(<function find_series_nuclei>, (<pygpseq.anim.condition.Condition object>, 0), {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...})]
    132 
    133     def __len__(self):
    134         return self._size
    135 

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/condition.py in find_series_nuclei(self=<pygpseq.anim.condition.Condition object>, i=0, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...})
    743 
    744     # Get starting time
    745     start_time = time.time()
    746 
    747     # Find nuclei
--> 748     self.series[i], log = self.series[i].find_nuclei(**kwargs)
        self.series = [<pygpseq.anim.series.Series object>, <pygpseq.anim.series.Series object>, <pygpseq.anim.series.Series object>]
        i = 0
        log = undefined
        i.find_nuclei = undefined
        kwargs = {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...ne/IMR90_iTK226-241_080618/Data_input_iTK226_233/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...}
    749 
    750     # Print log all at once
    751     time_msg = 'Took %s s.\n' % (round(time.time() - start_time, 3))
    752     if not 1 == kwargs['ncores']:

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/anim/series.py in find_nuclei(self=<pygpseq.anim.series.Series object>, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...1_080618/Data_input_iTK226_233/iTK226_070618_001/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...})
    229 
    230         # Make new channel copy
    231         i = dna_ch.copy()
    232 
    233         # Produce a mask
--> 234         Segmenter = Binarize(path = kwargs['logpath'], append = True, **kwargs)
        Segmenter = undefined
        kwargs = {'adaptive_neighbourhood': 101, 'an_type': 3, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...1_080618/Data_input_iTK226_233/iTK226_070618_001/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, 'debugging': False, ...}
    235         Segmenter.verbose = self.verbose
    236         
    237         # Check if already segmented
    238         already_segmented = False

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/tools/binarize.py in __init__(self=<pygpseq.tools.binarize.Binarize object>, **kwargs={'adaptive_neighbourhood': 101, 'an_type': 3, 'append': True, 'aspect': (300.0, 216.6, 216.6), 'basedir': '/home/bicro/Desktop/user_folders/Tomek/microscop...1_080618/Data_input_iTK226_233/iTK226_070618_001/', 'calc_n_surface': False, 'cdescr': {}, 'compressed': False, 'cond_name': 'iTK226_070618_001', 'correctCA': False, ...})
     60         """Initialize binarization settings all at once with kwargs.
     61 
     62         Args:
     63           **kwargs: arbitrary keyword arguments stored in the class.
     64         """
---> 65         super(Binarize, self).__init__()
        self.__init__ = <bound method Binarize.__init__ of <pygpseq.tools.binarize.Binarize object>>
     66 
     67         # Store provided kwargs in the current instance.
     68         excluded = ['logpath']
     69         for k in kwargs.keys():

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/tools/io.py in __init__(self=<pygpseq.tools.binarize.Binarize object>, **kwargs={})
     64                 fname, fext = os.path.splitext(curpath)
     65                 fname = fname + ' (' + str(c) + ')'
     66                 curpath = fname + fext
     67                 c += 1
     68 
---> 69         self.logpath = curpath
        self.logpath = ''
        curpath = '/tmp/pyGPSeq/log/2018-06-18_20:18:22.log'
     70     
     71     def check_log_dir(self, path = None):
     72         """Create log file directory if missing.
     73 

...........................................................................
/home/bicro/Desktop/ggcode/pygpseq/pygpseq/tools/binarize.py in __setattr__(self=<pygpseq.tools.binarize.Binarize object>, name='logpath', value='/tmp/pyGPSeq/log/2018-06-18_20:18:22.log')
     74         if key in dir(self): return(getattr(self, key))
     75         else: return(None)
     76 
     77     def __setattr__(self, name, value):
     78         """ Check the attribute and set it. """
---> 79         Binarize.check_attr(name, value)
        name = 'logpath'
        value = '/tmp/pyGPSeq/log/2018-06-18_20:18:22.log'
     80         return(super(Binarize, self).__setattr__(name, value))
     81 
     82     def __setitem__(self, key, value):
     83         """ Allow set item. """

TypeError: unbound method check_attr() must be called with Binarize instance as first argument (got str instance instead)

Additional information

My first action was to see if I am using the latest version of the pipeline. It happened to be already updated. Then, I took my previous data sets that I ran in the past and they failed at the same step. So the issue was not related to the images. I thought that maybe there was something fishy about them. Therefore, dear GG, maybe there is something silly from my side that I am doing wrong but at this point I do not know what could be the problem. Thanks! :)

ggirelli commented 6 years ago

Hi there!

Thank you for the bug report, although this is the wrong place to report such bugs. This is the repository for the centrality estimation pipeline. Please, report bugs on the image analysis pipeline in the pygpseq repository.

The bug in question was reported yesterday by another user and is already fixed in version 3.0.2. It was caused by a wrong dependency setup (which can happen during the installation process) and is now fixed.

I would also suggest to use the -m and -M parameters (added from v3.0.0) when running the pipeline, as they should speed up the segmentation process. Also, using the --compressed flag will decrease the space occupied by the generated masks.

Check out the wiki for more details on how to run the latest versions of the pipeline.

Cheers