search

scverse / scanpy

Single-cell analysis in Python. Scales to >1M cells.
https://scanpy.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.92k stars 601 forks source link

scanpy sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True) #1341

Closed taopeng1100 closed 4 years ago

taopeng1100 commented 4 years ago

After I use function

adata = sc.read_visium(
    './', count_file='V1_Human_Lymph_Node_filtered_feature_bc_matrix.h5',
    genome=None, library_id=None, load_images=True,
)

I use sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True) and got an error message:

RuntimeError                              Traceback (most recent call last)
~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\errors.py in new_error_context(fmt_, *args, **kwargs)
    744     try:
--> 745         yield
    746     except NumbaError as e:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower_block(self, block)
    272                                    loc=self.loc, errcls_=defaulterrcls):
--> 273                 self.lower_inst(inst)
    274         self.post_block(block)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower_inst(self, inst)
    485                 if isinstance(inst, _class):
--> 486                     func(self, inst)
    487                     return

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\parfors\parfor_lowering.py in _lower_parfor_parallel(lowerer, parfor)
    239             lowerer, parfor, typemap, typingctx, targetctx, flags, {},
--> 240             bool(alias_map), index_var_typ, parfor.races)
    241     finally:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\parfors\parfor_lowering.py in _create_gufunc_for_parfor_body(lowerer, parfor, typemap, typingctx, targetctx, flags, locals, has_aliases, index_var_typ, races)
   1326         flags,
-> 1327         locals)
   1328 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in compile_ir(typingctx, targetctx, func_ir, args, return_type, flags, locals, lifted, lifted_from, is_lifted_loop, library, pipeline_class)
    666         return pipeline.compile_ir(func_ir=func_ir, lifted=lifted,
--> 667                                    lifted_from=lifted_from)
    668 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in compile_ir(self, func_ir, lifted, lifted_from)
    348         FixupArgs().run_pass(self.state)
--> 349         return self._compile_ir()
    350 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_ir(self)
    407         assert self.state.func_ir is not None
--> 408         return self._compile_core()
    409 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_core(self)
    380                 if is_final_pipeline:
--> 381                     raise e
    382         else:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_core(self)
    371             try:
--> 372                 pm.run(self.state)
    373                 if self.state.cr is not None:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in run(self, state)
    340                 patched_exception = self._patch_error(msg, e)
--> 341                 raise patched_exception
    342 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in run(self, state)
    331                 if isinstance(pass_inst, CompilerPass):
--> 332                     self._runPass(idx, pass_inst, state)
    333                 else:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in _runPass(self, index, pss, internal_state)
    290         with SimpleTimer() as pass_time:
--> 291             mutated |= check(pss.run_pass, internal_state)
    292         with SimpleTimer() as finalize_time:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in check(func, compiler_state)
    263         def check(func, compiler_state):
--> 264             mangled = func(compiler_state)
    265             if mangled not in (True, False):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\typed_passes.py in run_pass(self, state)
    441         # TODO: Pull this out into the pipeline
--> 442         NativeLowering().run_pass(state)
    443         lowered = state['cr']

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\typed_passes.py in run_pass(self, state)
    369                                        metadata=metadata)
--> 370                 lower.lower()
    371                 if not flags.no_cpython_wrapper:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower(self)
    216         # Materialize LLVM Module
--> 217         self.library.add_ir_module(self.module)
    218 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\codegen.py in add_ir_module(self, ir_module)
    205         ir = cgutils.normalize_ir_text(str(ir_module))
--> 206         ll_module = ll.parse_assembly(ir)
    207         ll_module.name = ir_module.name

~\AppData\Local\Continuum\anaconda3\lib\site-packages\llvmlite\binding\module.py in parse_assembly(llvmir, context)
     24             mod.close()
---> 25             raise RuntimeError("LLVM IR parsing error\n{0}".format(errmsg))
     26     return mod

RuntimeError: Failed in nopython mode pipeline (step: nopython mode backend)
LLVM IR parsing error
<string>:4079:36: error: '%.2747' defined with type 'i64' but expected 'i32'
  %".2748" = icmp eq i32 %".2746", %".2747"
                                   ^

During handling of the above exception, another exception occurred:

LoweringError                             Traceback (most recent call last)
<ipython-input-10-a83dc5279093> in <module>
----> 1 sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py in calculate_qc_metrics(adata, expr_type, var_type, qc_vars, percent_top, layer, use_raw, inplace, log1p, parallel)
    294         inplace=inplace,
    295         X=X,
--> 296         log1p=log1p,
    297     )
    298     var_metrics = describe_var(

~\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py in describe_obs(adata, expr_type, var_type, qc_vars, percent_top, layer, use_raw, log1p, inplace, X, parallel)
    112     if percent_top:
    113         percent_top = sorted(percent_top)
--> 114         proportions = top_segment_proportions(X, percent_top)
    115         for i, n in enumerate(percent_top):
    116             obs_metrics[f"pct_{expr_type}_in_top_{n}_{var_type}"] = (

~\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py in top_segment_proportions(mtx, ns)
    377             mtx = csr_matrix(mtx)
    378         return top_segment_proportions_sparse_csr(
--> 379             mtx.data, mtx.indptr, np.array(ns, dtype=np.int)
    380         )
    381     else:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in _compile_for_args(self, *args, **kws)
    432                     e.patch_message('\n'.join((str(e).rstrip(), help_msg)))
    433             # ignore the FULL_TRACEBACKS config, this needs reporting!
--> 434             raise e
    435 
    436     def inspect_llvm(self, signature=None):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in _compile_for_args(self, *args, **kws)
    365                 argtypes.append(self.typeof_pyval(a))
    366         try:
--> 367             return self.compile(tuple(argtypes))
    368         except errors.ForceLiteralArg as e:
    369             # Received request for compiler re-entry with the list of arguments

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in compile(self, sig)
    806             self._cache_misses[sig] += 1
    807             try:
--> 808                 cres = self._compiler.compile(args, return_type)
    809             except errors.ForceLiteralArg as e:
    810                 def folded(args, kws):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in compile(self, args, return_type)
     76 
     77     def compile(self, args, return_type):
---> 78         status, retval = self._compile_cached(args, return_type)
     79         if status:
     80             return retval

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in _compile_cached(self, args, return_type)
     90 
     91         try:
---> 92             retval = self._compile_core(args, return_type)
     93         except errors.TypingError as e:
     94             self._failed_cache[key] = e

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\dispatcher.py in _compile_core(self, args, return_type)
    108                                       args=args, return_type=return_type,
    109                                       flags=flags, locals=self.locals,
--> 110                                       pipeline_class=self.pipeline_class)
    111         # Check typing error if object mode is used
    112         if cres.typing_error is not None and not flags.enable_pyobject:

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    601     pipeline = pipeline_class(typingctx, targetctx, library,
    602                               args, return_type, flags, locals)
--> 603     return pipeline.compile_extra(func)
    604 
    605 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in compile_extra(self, func)
    337         self.state.lifted = ()
    338         self.state.lifted_from = None
--> 339         return self._compile_bytecode()
    340 
    341     def compile_ir(self, func_ir, lifted=(), lifted_from=None):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_bytecode(self)
    399         """
    400         assert self.state.func_ir is None
--> 401         return self._compile_core()
    402 
    403     def _compile_ir(self):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_core(self)
    379                 self.state.status.fail_reason = e
    380                 if is_final_pipeline:
--> 381                     raise e
    382         else:
    383             raise CompilerError("All available pipelines exhausted")

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler.py in _compile_core(self)
    370             res = None
    371             try:
--> 372                 pm.run(self.state)
    373                 if self.state.cr is not None:
    374                     break

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in run(self, state)
    339                     (self.pipeline_name, pass_desc)
    340                 patched_exception = self._patch_error(msg, e)
--> 341                 raise patched_exception
    342 
    343     def dependency_analysis(self):

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in run(self, state)
    330                 pass_inst = _pass_registry.get(pss).pass_inst
    331                 if isinstance(pass_inst, CompilerPass):
--> 332                     self._runPass(idx, pass_inst, state)
    333                 else:
    334                     raise BaseException("Legacy pass in use")

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in _runPass(self, index, pss, internal_state)
    289             mutated |= check(pss.run_initialization, internal_state)
    290         with SimpleTimer() as pass_time:
--> 291             mutated |= check(pss.run_pass, internal_state)
    292         with SimpleTimer() as finalize_time:
    293             mutated |= check(pss.run_finalizer, internal_state)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\compiler_machinery.py in check(func, compiler_state)
    262 
    263         def check(func, compiler_state):
--> 264             mangled = func(compiler_state)
    265             if mangled not in (True, False):
    266                 msg = ("CompilerPass implementations should return True/False. "

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\typed_passes.py in run_pass(self, state)
    440 
    441         # TODO: Pull this out into the pipeline
--> 442         NativeLowering().run_pass(state)
    443         lowered = state['cr']
    444         signature = typing.signature(state.return_type, *state.args)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\typed_passes.py in run_pass(self, state)
    368                 lower = lowering.Lower(targetctx, library, fndesc, interp,
    369                                        metadata=metadata)
--> 370                 lower.lower()
    371                 if not flags.no_cpython_wrapper:
    372                     lower.create_cpython_wrapper(flags.release_gil)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower(self)
    177         if self.generator_info is None:
    178             self.genlower = None
--> 179             self.lower_normal_function(self.fndesc)
    180         else:
    181             self.genlower = self.GeneratorLower(self)

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower_normal_function(self, fndesc)
    231         # Init argument values
    232         self.extract_function_arguments()
--> 233         entry_block_tail = self.lower_function_body()
    234 
    235         # Close tail of entry block

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower_function_body(self)
    257             bb = self.blkmap[offset]
    258             self.builder.position_at_end(bb)
--> 259             self.lower_block(block)
    260         self.post_lower()
    261         return entry_block_tail

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\lowering.py in lower_block(self, block)
    271             with new_error_context('lowering "{inst}" at {loc}', inst=inst,
    272                                    loc=self.loc, errcls_=defaulterrcls):
--> 273                 self.lower_inst(inst)
    274         self.post_block(block)
    275 

~\AppData\Local\Continuum\anaconda3\lib\contextlib.py in __exit__(self, type, value, traceback)
    128                 value = type()
    129             try:
--> 130                 self.gen.throw(type, value, traceback)
    131             except StopIteration as exc:
    132                 # Suppress StopIteration *unless* it's the same exception that

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\errors.py in new_error_context(fmt_, *args, **kwargs)
    750         newerr = errcls(e).add_context(_format_msg(fmt_, args, kwargs))
    751         tb = sys.exc_info()[2] if numba.core.config.FULL_TRACEBACKS else None
--> 752         reraise(type(newerr), newerr, tb)
    753 
    754 

~\AppData\Local\Continuum\anaconda3\lib\site-packages\numba\core\utils.py in reraise(tp, value, tb)
     79     if value.__traceback__ is not tb:
     80         raise value.with_traceback(tb)
---> 81     raise value
     82 
     83 

LoweringError: Failed in nopython mode pipeline (step: nopython mode backend)
Failed in nopython mode pipeline (step: nopython mode backend)
LLVM IR parsing error
<string>:4079:36: error: '%.2747' defined with type 'i64' but expected 'i32'
  %".2748" = icmp eq i32 %".2746", %".2747"
                                   ^

File "..\..\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py", line 412:
def top_segment_proportions_sparse_csr(data, indptr, ns):
    <source elided>
    partitioned = np.zeros((indptr.size - 1, maxidx), dtype=data.dtype)
    for i in numba.prange(indptr.size - 1):
    ^

During: lowering "id=13[LoopNest(index_variable = parfor_index.264, range = (0, $122binary_subtract.5, 1))]{130: <ir.Block at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (412)>, 400: <ir.Block at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (418)>, 402: <ir.Block at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (419)>, 276: <ir.Block at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (416)>, 318: <ir.Block at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (417)>}Var(parfor_index.264, _qc.py:412)" at C:\Users\tpeng\AppData\Local\Continuum\anaconda3\lib\site-packages\scanpy\preprocessing\_qc.py (412)

The package version is as follows:

scanpy==1.5.1 anndata==0.7.4 umap==0.4.6 numpy==1.18.5 scipy==1.5.0 pandas==1.0.5 scikit-learn==0.23.1 statsmodels==0.11.1 python-igraph==0.7.1+5.3b99dbf6 leidenalg==0.7.0

flying-sheep commented 4 years ago

@taopeng1100: This is a numba bug. Please report this there, but only if you use the newest numba version (otherwise install it and try to reproduce this with the newest version). Give them this link so they see the code that triggers their bug:

https://github.com/theislab/scanpy/blob/2f160ea403d124d237fc2138c0aa0d175fbad22a/scanpy/preprocessing/_qc.py#L402-L428

@team: We should include numba in the package versions list.

taopeng1100 commented 4 years ago

Can you let me know the version of numba that worked for you?

Get Outlook for Androidhttps://aka.ms/ghei36

From: Philipp A. notifications@github.com Sent: Thursday, July 30, 2020 2:06:10 AM To: theislab/scanpy scanpy@noreply.github.com Cc: Peng, Tao tpeng@fredhutch.org; Mention mention@noreply.github.com Subject: Re: [theislab/scanpy] scanpy sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True) (#1341)

@taopeng1100https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_taopeng1100&d=DwMCaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=j6EgtEBZ-6pbDONgnwVzuTHJ6L-gWcikckOhZCwVjPc&m=j6WdQxFMfGL2BOrFUsLxuCIlMEWSPOGMoQ3sfPhU4hw&s=mZvFxBCIH_-96BcNkwcrYuLqIFdGUQVv9-NgmpF6Kyc&e=: This is a numba bug. Please report this there, but only if you use the newest numba version. Give them this link so they see the code that triggers their bug:

https://github.com/theislab/scanpy/blob/2f160ea403d124d237fc2138c0aa0d175fbad22a/scanpy/preprocessing/_qc.py#L402-L428https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_theislab_scanpy_blob_2f160ea403d124d237fc2138c0aa0d175fbad22a_scanpy_preprocessing_-5Fqc.py-23L402-2DL428&d=DwMCaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=j6EgtEBZ-6pbDONgnwVzuTHJ6L-gWcikckOhZCwVjPc&m=j6WdQxFMfGL2BOrFUsLxuCIlMEWSPOGMoQ3sfPhU4hw&s=6bLtPWJSoSBSNQcRvwP4Pp-mHjA7d51Ees_qcvGEJYk&e=

@team: We should include numba in the package versions list.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_theislab_scanpy_issues_1341-23issuecomment-2D666246043&d=DwMCaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=j6EgtEBZ-6pbDONgnwVzuTHJ6L-gWcikckOhZCwVjPc&m=j6WdQxFMfGL2BOrFUsLxuCIlMEWSPOGMoQ3sfPhU4hw&s=AQB0zBmdSewipAgOqteV0t7F27jeGZaU-I4LngYZYus&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ALCZ3CPX4MWCNAHOFNSYRC3R6EZYFANCNFSM4PMLDJKQ&d=DwMCaQ&c=eRAMFD45gAfqt84VtBcfhQ&r=j6EgtEBZ-6pbDONgnwVzuTHJ6L-gWcikckOhZCwVjPc&m=j6WdQxFMfGL2BOrFUsLxuCIlMEWSPOGMoQ3sfPhU4hw&s=XT2LyZ_ZXTKdnWcyG0sd7w1UgmMmxTpvj_8IWLPy44I&e=.

flying-sheep commented 4 years ago

You can find the newest version on the PyPI page: https://pypi.org/project/numba/

stuartarchibald commented 4 years ago

Think this is a duplicate of https://github.com/numba/numba/issues/5955

flying-sheep commented 4 years ago

Thank you @stuartarchibald, it sure is! The error happens when numba tries to JIT-compile top_segment_proportions_sparse_csr.

taopeng1100 commented 4 years ago

Hi Philip, can you kindly help me here? How can I get “sc.pp.calculate_qc_metrics(adata, qc_vars=["mt"], inplace=True)” to work for me?

flying-sheep commented 4 years ago

Maybe downgrade numba for the time being? IDK to which version though. @stuartarchibald has more insight here. Please follow numba/numba#5955 for updates!

taopeng1100 commented 4 years ago

I am using Anaconda/Jupyter in my PC. When I try to downgrade numba, I run into issues of numba dependency packages in Anaconda so I am stuck!

stuartarchibald commented 4 years ago

@flying-sheep I might know a work around for scanpy whilst we try and fix the root cause in numba, would that help?

flying-sheep commented 4 years ago

Sure, thank you! Care to do a quick PR? Then we can point @taopeng1100 in the direction of installing scanpy’s dev version and everyone’s happy.

@taopeng1100 please reply by GitHub comment and not by email anymore, it spams up this comment section. I always have to remove some junk your email program adds.

stuartarchibald commented 4 years ago

@flying-sheep done https://github.com/theislab/scanpy/pull/1359

flying-sheep commented 4 years ago

Okay, it’s merged! @taopeng1100, please install the dev version of scanpy like this, and retry:

pip install git+https://github.com/theislab/scanpy.git
# or
pip install --user git+https://github.com/theislab/scanpy.git
taopeng1100 commented 4 years ago

It all works NOW! I appreciate your help!

taopeng1100 commented 4 years ago

It all works NOW! I appreciate your help!

taopeng1100 commented 4 years ago

It all works NOW! I appreciate your help!

taopeng1100 commented 4 years ago

It all works NOW! I appreciate your help!