Closed SidG13 closed 1 year ago
Goultard59
commented
1 year ago Have you checked the consistency between Chromosome in your custom annotation dataframe, your ranking database, your anndata matrix ?
As I remember the time, I build my scenic database. You need to check that cellranger does not modify your gene's names in case of multiple duplicates (for example U6 family genes). You need that every gene of your cell ranger matrix are in your database and in your custom annotation.
SidG13
commented
1 year ago Hi, thanks for your response. Here's what I did to check my files:
# Check file integrity (Genes) #
> library(anndata)
> ad <- read_h5ad("path/to/my/adata.h5ad")
> ad
AnnData object with n_obs × n_vars = 3865 × 15421
obs: 'cell_cluster', 'doublet_score', 'predicted_doublet'
var: 'gene_ids'
uns: 'cell_cluster_colors', 'scrublet'
obsm: 'X_umap'
> TSS_annot <- read.table('TSS_annot.txt', header=T, sep='\t') # also called `custom_annot` in python
> ad$var$gene_ids[!(ad$var$gene_ids %in% TSS_annot$Gene)]
character(0) # no genes from my RNA counts matrix are missing in the TSS annotation file
# Check file integrity (Chr names) #
> library(arrow)
> rankings_db <- arrow::read_feather('path/to/regions_vs_motifs.rankings.feather')
> sort(unique(str_extract(colnames(rankings_db),'.*(?=:)')))
[1] "ma1" "ma2" "ma3" "ma4" "ma5" "ma6" "ma7" "mi1" "mi10" "mi2" "mi3" "mi4" "mi5" "mi6" "mi7" "mi8" "mi9" "myo1" "myo3" "Z"
> sort(unique(TSS_annot$Chromosome))
[1] "ma1" "ma2" "ma3" "ma4" "ma5" "ma6" "ma7" "mi1" "mi10" "mi2" "mi3" "mi4" "mi5" "mi6" "mi7" "mi8" "mi9" "myo1" "un187" "Z"Though there are some chromosomes which are in one set and not the other (because there is a gene on it, but no motifs annotated since it was blacklisted during peak calling), I don't think this should cause a KeyError naming issue since the other names are the same and should be paired/recognized properly.
Sorry for the trouble, but do you see anything wrong perhaps? Is there another dataset I should double check for consistency?
Goultard59
commented
1 year ago I think that there are no overlaps between the "foreground" variable and your annotation here : https://github.com/aertslab/pycistarget/blob/0ed8289bda8feb8b137f243864213cee6b06d7f0/pycistarget/motif_enrichment_dem.py#L521
resulting in an "NA" value; you should check that you're not out of chromosome with default promoter space and your custom annotation.
SidG13
commented
1 year ago Here's what my fg_pr_overlap object looks like below. You can see a few overlaps are found, the rest are 0. I don't see NAs. Though, I'm not sure what the data is supposed to look like, @SeppeDeWinter does this look correct to you? I don't know why there are two different dataframes here, but again I don't know what the data is supposed to look like and maybe this is fine.
print(fg_pr_overlap)
+--------------+-----------+-----------+------------------+
| Chromosome | Start | End | NumberOverlaps |
| (category) | (int32) | (int32) | (int64) |
|--------------+-----------+-----------+------------------|
| Z | 37835199 | 37835699 | 0 |
| Z | 22469724 | 22470224 | 0 |
| Z | 15777279 | 15777779 | 0 |
| Z | 89932921 | 89933421 | 0 |
| ... | ... | ... | ... |
| mi10 | 2433294 | 2433794 | 1 |
| mi10 | 2266775 | 2267275 | 0 |
| mi10 | 185654 | 186154 | 1 |
| mi10 | 1576246 | 1576746 | 0 |
+--------------+-----------+-----------+------------------+
Unstranded PyRanges object has 2,868 rows and 4 columns from 18 chromosomes.
For printing, the PyRanges was sorted on Chromosome.
+--------------+-----------+-----------+------------------+
| Chromosome | Start | End | NumberOverlaps |
| (category) | (int32) | (int32) | (int64) |
|--------------+-----------+-----------+------------------|
| Z | 18489754 | 18490254 | 0 |
| Z | 60540869 | 60541369 | 0 |
| Z | 99620210 | 99620710 | 0 |
| Z | 67880911 | 67881411 | 0 |
| ... | ... | ... | ... |
| ma5 | 14717191 | 14717691 | 0 |
| ma5 | 37661785 | 37662285 | 0 |
| ma5 | 14718445 | 14718945 | 0 |
| ma5 | 14721267 | 14721767 | 0 |
| ma6 | 11729879 | 11730379 | 0 |
| ma6 | 30263065 | 30263565 | 0 |
| ma7 | 52411710 | 52412210 | 0 |
+--------------+-----------+-----------+------------------+
Unstranded PyRanges object has 23 rows and 4 columns from 8 chromosomes.
For printing, the PyRanges was sorted on Chromosome.Just to be thorough I checked if any values were indeed NA like so, but got nothing:
print(fg_pr_overlap[fg_pr_overlap.NumberOverlaps == 'NA'])
Empty PyRanges
SeppeDeWinter
commented
1 year ago Hi @SidG13
Sorry for the late reply.
Probably the issue is caused because one of your region sets does not contain any promoters, resulting in an empty pyranges object. This results in an error in the coord_to_region_names function, this is a bug.
The bug should be fixed with this commit https://github.com/aertslab/pycistarget/commit/5c355396393a44458934032e000bd22876a4d071.
Now the function returns an empty list instead of crashing, see example below:
In [1]: coord_to_region_names(pr.PyRanges())
Out[1]: []p.s. @Goultard59, thanks for the help!
I hope this fixes your issue?
Best,
Seppe
SidG13
commented
1 year ago Thanks @SeppeDeWinter, the wrapper certainly does go much further now! I don't want to paste all the output here, but I've got all the following folders made (and populated with all htmls)
But I do get the following error right at the very end, unfortunately I'm not very helpful but it's probably related to pickle?
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
Cell In [8], line 8
5 custom_annot = pd.read_csv('/home/administrator/Desktop/ExtraDrive1/Sid/sc_multiome/SCENICplus_multiome/data_manipulation/TSS_annot.txt', header=0, sep='\t')
6 #custom_annot
----> 8 run_pycistarget(
9 region_sets = region_sets,
10 species = 'custom',
11 save_path = os.path.join(work_dir, 'motifs'),
12 custom_annot = custom_annot,
13 ctx_db_path = rankings_db,
14 dem_db_path = scores_db,
15 path_to_motif_annotations = motif_annotation,
16 run_without_promoters = True,
17 n_cpu = 4,
18 _temp_dir = os.path.join(tmp_dir, 'ray_spill'),
19 annotation_version = 'v1',
20 )
File ~/Desktop/ExtraDrive1/Sid/sc_multiome/SCENICplus_multiome/scenicplus/src/scenicplus/wrappers/run_pycistarget.py:331, in run_pycistarget(region_sets, species, save_path, custom_annot, save_partial, ctx_db_path, dem_db_path, run_without_promoters, biomart_host, promoter_space, ctx_auc_threshold, ctx_nes_threshold, ctx_rank_threshold, dem_log2fc_thr, dem_motif_hit_thr, dem_max_bg_regions, annotation, motif_similarity_fdr, path_to_motif_annotations, annotation_version, n_cpu, _temp_dir, exclude_motifs, exclude_collection, **kwargs)
329 log.info('Saving object')
330 with open(os.path.join(save_path,'menr.pkl'), 'wb') as f:
--> 331 dill.dump(menr, f, protocol=-1)
333 log.info('Finished! Took {} minutes'.format((time.time() - start_time)/60))
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:336, in dump(obj, file, protocol, byref, fmode, recurse, **kwds)
334 _kwds = kwds.copy()
335 _kwds.update(dict(byref=byref, fmode=fmode, recurse=recurse))
--> 336 Pickler(file, protocol, **_kwds).dump(obj)
337 return
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:620, in Pickler.dump(self, obj)
618 raise PicklingError(msg)
619 else:
--> 620 StockPickler.dump(self, obj)
621 return
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:487, in _Pickler.dump(self, obj)
485 if self.proto >= 4:
486 self.framer.start_framing()
--> 487 self.save(obj)
488 self.write(STOP)
489 self.framer.end_framing()
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:1251, in save_module_dict(pickler, obj)
1248 if is_dill(pickler, child=False) and pickler._session:
1249 # we only care about session the first pass thru
1250 pickler._first_pass = False
-> 1251 StockPickler.save_dict(pickler, obj)
1252 log.info("# D2")
1253 return
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:971, in _Pickler.save_dict(self, obj)
968 self.write(MARK + DICT)
970 self.memoize(obj)
--> 971 self._batch_setitems(obj.items())
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:997, in _Pickler._batch_setitems(self, items)
995 for k, v in tmp:
996 save(k)
--> 997 save(v)
998 write(SETITEMS)
999 elif n:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:1251, in save_module_dict(pickler, obj)
1248 if is_dill(pickler, child=False) and pickler._session:
1249 # we only care about session the first pass thru
1250 pickler._first_pass = False
-> 1251 StockPickler.save_dict(pickler, obj)
1252 log.info("# D2")
1253 return
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:971, in _Pickler.save_dict(self, obj)
968 self.write(MARK + DICT)
970 self.memoize(obj)
--> 971 self._batch_setitems(obj.items())
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:997, in _Pickler._batch_setitems(self, items)
995 for k, v in tmp:
996 save(k)
--> 997 save(v)
998 write(SETITEMS)
999 elif n:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:603, in _Pickler.save(self, obj, save_persistent_id)
599 raise PicklingError("Tuple returned by %s must have "
600 "two to six elements" % reduce)
602 # Save the reduce() output and finally memoize the object
--> 603 self.save_reduce(obj=obj, *rv)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:717, in _Pickler.save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
715 if state is not None:
716 if state_setter is None:
--> 717 save(state)
718 write(BUILD)
719 else:
720 # If a state_setter is specified, call it instead of load_build
721 # to update obj's with its previous state.
722 # First, push state_setter and its tuple of expected arguments
723 # (obj, state) onto the stack.
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:1251, in save_module_dict(pickler, obj)
1248 if is_dill(pickler, child=False) and pickler._session:
1249 # we only care about session the first pass thru
1250 pickler._first_pass = False
-> 1251 StockPickler.save_dict(pickler, obj)
1252 log.info("# D2")
1253 return
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:971, in _Pickler.save_dict(self, obj)
968 self.write(MARK + DICT)
970 self.memoize(obj)
--> 971 self._batch_setitems(obj.items())
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:997, in _Pickler._batch_setitems(self, items)
995 for k, v in tmp:
996 save(k)
--> 997 save(v)
998 write(SETITEMS)
999 elif n:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:603, in _Pickler.save(self, obj, save_persistent_id)
599 raise PicklingError("Tuple returned by %s must have "
600 "two to six elements" % reduce)
602 # Save the reduce() output and finally memoize the object
--> 603 self.save_reduce(obj=obj, *rv)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:717, in _Pickler.save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
715 if state is not None:
716 if state_setter is None:
--> 717 save(state)
718 write(BUILD)
719 else:
720 # If a state_setter is specified, call it instead of load_build
721 # to update obj's with its previous state.
722 # First, push state_setter and its tuple of expected arguments
723 # (obj, state) onto the stack.
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/site-packages/dill/_dill.py:1251, in save_module_dict(pickler, obj)
1248 if is_dill(pickler, child=False) and pickler._session:
1249 # we only care about session the first pass thru
1250 pickler._first_pass = False
-> 1251 StockPickler.save_dict(pickler, obj)
1252 log.info("# D2")
1253 return
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:971, in _Pickler.save_dict(self, obj)
968 self.write(MARK + DICT)
970 self.memoize(obj)
--> 971 self._batch_setitems(obj.items())
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:997, in _Pickler._batch_setitems(self, items)
995 for k, v in tmp:
996 save(k)
--> 997 save(v)
998 write(SETITEMS)
999 elif n:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:603, in _Pickler.save(self, obj, save_persistent_id)
599 raise PicklingError("Tuple returned by %s must have "
600 "two to six elements" % reduce)
602 # Save the reduce() output and finally memoize the object
--> 603 self.save_reduce(obj=obj, *rv)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:692, in _Pickler.save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
690 else:
691 save(func)
--> 692 save(args)
693 write(REDUCE)
695 if obj is not None:
696 # If the object is already in the memo, this means it is
697 # recursive. In this case, throw away everything we put on the
698 # stack, and fetch the object back from the memo.
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:886, in _Pickler.save_tuple(self, obj)
884 if n <= 3 and self.proto >= 2:
885 for element in obj:
--> 886 save(element)
887 # Subtle. Same as in the big comment below.
888 if id(obj) in memo:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:886, in _Pickler.save_tuple(self, obj)
884 if n <= 3 and self.proto >= 2:
885 for element in obj:
--> 886 save(element)
887 # Subtle. Same as in the big comment below.
888 if id(obj) in memo:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:603, in _Pickler.save(self, obj, save_persistent_id)
599 raise PicklingError("Tuple returned by %s must have "
600 "two to six elements" % reduce)
602 # Save the reduce() output and finally memoize the object
--> 603 self.save_reduce(obj=obj, *rv)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:692, in _Pickler.save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
690 else:
691 save(func)
--> 692 save(args)
693 write(REDUCE)
695 if obj is not None:
696 # If the object is already in the memo, this means it is
697 # recursive. In this case, throw away everything we put on the
698 # stack, and fetch the object back from the memo.
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:886, in _Pickler.save_tuple(self, obj)
884 if n <= 3 and self.proto >= 2:
885 for element in obj:
--> 886 save(element)
887 # Subtle. Same as in the big comment below.
888 if id(obj) in memo:
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:603, in _Pickler.save(self, obj, save_persistent_id)
599 raise PicklingError("Tuple returned by %s must have "
600 "two to six elements" % reduce)
602 # Save the reduce() output and finally memoize the object
--> 603 self.save_reduce(obj=obj, *rv)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:692, in _Pickler.save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
690 else:
691 save(func)
--> 692 save(args)
693 write(REDUCE)
695 if obj is not None:
696 # If the object is already in the memo, this means it is
697 # recursive. In this case, throw away everything we put on the
698 # stack, and fetch the object back from the memo.
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:901, in _Pickler.save_tuple(self, obj)
899 write(MARK)
900 for element in obj:
--> 901 save(element)
903 if id(obj) in memo:
904 # Subtle. d was not in memo when we entered save_tuple(), so
905 # the process of saving the tuple's elements must have saved
(...)
909 # could have been done in the "for element" loop instead, but
910 # recursive tuples are a rare thing.
911 get = self.get(memo[id(obj)][0])
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:560, in _Pickler.save(self, obj, save_persistent_id)
558 f = self.dispatch.get(t)
559 if f is not None:
--> 560 f(self, obj) # Call unbound method with explicit self
561 return
563 # Check private dispatch table if any, or else
564 # copyreg.dispatch_table
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:839, in _Pickler.save_picklebuffer(self, obj)
835 if in_band:
836 # Write data in-band
837 # XXX The C implementation avoids a copy here
838 if m.readonly:
--> 839 self.save_bytes(m.tobytes())
840 else:
841 self.save_bytearray(m.tobytes())
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:806, in _Pickler.save_bytes(self, obj)
804 else:
805 self.write(BINBYTES + pack("<I", n) + obj)
--> 806 self.memoize(obj)
File ~/anaconda2/envs/py38/lib/python3.8/pickle.py:508, in _Pickler.memoize(self, obj)
506 if self.fast:
507 return
--> 508 assert id(obj) not in self.memo
509 idx = len(self.memo)
510 self.write(self.put(idx))
AssertionError:When I try the next codeblock to open the menr.pkl file:
SeppeDeWinter
commented
1 year ago Hi @SidG13
That's very unfortunate ... I have never seen this so I'm not sure what happened.
I would suggest running the function again but setting save_partial to True. That way intermediate results will be saved i the folders you showed above. Then you have a backup incase the saving fails again.
Best,
Seppe
SidG13
commented
1 year ago Thanks @SeppeDeWinter, I'm not sure how to use the backup files that were made since I can't proceed with the workflow, in fact even running with save_partial = True still throws a (what I think is) dill error. It's really odd that the menr.pkl file is made however, so it's probably just being corrupted somehow. Do you think it's a python version/dill version error?
Would it help if I sent you the requisite data (annotation, database files etc.) to test on your end to see if the error can be replicated?
SeppeDeWinter
commented
1 year ago Hi @SidG13
If you don't mind sending me the data then I'll do some tests after the weekend. I'm really not sure what is going wrong.
You can send it to seppe.dewinter@kuleuven.be
Best
Seppe
SidG13
commented
1 year ago Thanks so much! Just sent.
Goultard59
commented
1 year ago Hi,
After trying the pipelines with more samples, I encounter the same two errors.
Traceback (most recent call last):
File "/home/adufour/save/scripts/omicscenic.py", line 166, in <module>
run_pycistarget(
File "/work/adufour/scenicplus/src/scenicplus/wrappers/run_pycistarget.py", line 331, in run_pycistarget
dill.dump(menr, f, protocol=-1)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 336, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 620, in dump
StockPickler.dump(self, obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 487, in dump
self.save(obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 717, in save_reduce
save(state)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 717, in save_reduce
save(state)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 971, in save_dict
self._batch_setitems(obj.items())
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 997, in _batch_setitems
save(v)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 886, in save_tuple
save(element)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 886, in save_tuple
save(element)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 886, in save_tuple
save(element)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 603, in save
self.save_reduce(obj=obj, *rv)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 692, in save_reduce
save(args)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 901, in save_tuple
save(element)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 560, in save
f(self, obj) # Call unbound method with explicit self
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 839, in save_picklebuffer
self.save_bytes(m.tobytes())
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 806, in save_bytes
self.memoize(obj)
File "/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/pickle.py", line 508, in memoize
assert id(obj) not in self.memo
AssertionError
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x7feb9adf3f06]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x20028e5) [0x7feb9adeb8e5]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x1f27e09) [0x7feb9ad10e09]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x7feb9adf4a3d]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x1f25948) [0x7feb9ad0e948]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x7feb9adf4a3d]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x1ee0b46) [0x7feb9acc9b46]
/work/adufour/anaconda3/envs/scenicplus/lib/python3.8/site-packages/pyarrow/libarrow.so.900(+0x194546a) [0x7feb9a72e46a]
/lib64/libc.so.6(+0x39ce9) [0x7fec466a7ce9]
/lib64/libc.so.6(+0x39d37) [0x7fec466a7d37]
/lib64/libc.so.6(__libc_start_main+0xfc) [0x7fec4669055c]
python(+0x1ce125) [0x5566c3af3125]
/var/spool/slurm/d/job42616761/slurm_script : ligne 12 : 25552 Abandon python /home/adufour/save/scripts/omicscenic.pyupgrading to pyarrow >10.0 solves the problems for me.
SidG13
commented
1 year ago Unfortunately I'm still getting the same error as before. I updated to pyarrow v10.0.1. My guess is it's a dill error, my version is 0.3.5.1
SeppeDeWinter
commented
1 year ago @SidG13
I finally got around to trying to debug the issue using your data, my apologies for the delay (it has been a very busy period).
I was able to save the motif enrichment dictionary (menr) for your data using protocol 4 of dill. I pushed a change to the developmental branch so this protocol is used by default (https://github.com/aertslab/scenicplus/commit/f322c5b09b54213b8ec5062f25cc0b4143e92ee4).
If you pull the code from that branch (git clone --branch development https://github.com/aertslab/scenicplus.git) and reinstall scenicplus you should be able to save the motif enrichment results.
I hope this fixes this annoying issue.
Best,
Seppe
SidG13
commented
1 year ago Thank you, this worked!
Describe the bug I'm getting a
KeyErrorwhen running the wrapper function. The function works well up until the 'Creating contrast groups' in the motif_enrichment_dem.py script. I ran into this issue when using @Goultard59's original PR. When doing a bit of my own line-by-line troubleshooting, thefg_probject referenced as the point where the script stops working is created properly, so I'm not entirely sure what exactly is causing this issue.To Reproduce
Error output
Screenshots Here's what my
custom_annotpandas df looks like:Here's the structure of my
region_sets:Version (please complete the following information):
Additional context Thank you for your help!