Closed Sayyam-Shah closed 1 month ago
SeppeDeWinter
commented
1 year ago Hi
That's an annoying error..
Which version of dill are you using?
Can you try saving it with the protocol=-1 argument (i.e. dill.dump(scplus_obj, f, protocol=-1))
Best,
Seppe
Sayyam-Shah
commented
1 year ago Hello Seppe,
Thank you for getting back to me. I am using dill version 0.3.5.1. I previously tried using protocol -1 to save it, however, I got the same error. I recently ran the build grn step and the function saved the object as a pkl. However, I get the below error when trying to read in the object.
EOFError Traceback (most recent call last)
/tmp/ipykernel_35234/2263231967.py in <module>
1 infile = open('/cluster/projects/scenicplus_full/scplus_obj.pkl', 'rb')
----> 2 scplus_obj = pickle.load(infile)
3 infile.close()
EOFError: Ran out of inputI recently submitted another slurm job to run the build grn step while testing protocol -1 at the very end, however, I assume it is already embedded in the run_scenicplus function.
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
The run_scenicplus job just finished and I got the below error. There are a couple of future warnings and an error with dill in the scenicplus_workflow.py script.
2023-01-05 03:22:19,818 SCENIC+_wrapper INFO Saving object
/cluster/home/.local/lib/python3.7/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.
warnings.warn(msg, category=FutureWarning)
/cluster/home/.local/lib/python3.7/site-packages/loomxpy/loomxpy.py:459: FutureWarning: The default value of regex will change from True to False in a future version.
regulons.columns = regulons.columns.str.replace("_?\\(", "_(")
/cluster/home/.local/lib/python3.7/site-packages/loomxpy/loomxpy.py:437: FutureWarning: The default value of regex will change from True to False in a future version.
regulons.columns = regulons.columns.str.replace("_?\\(", "_(")
Traceback (most recent call last):
File "scenicplus_workflow.py", line 71, in <module>
_temp_dir = '/cluster/home/grntemp1')
File "/cluster/home/.local/lib/python3.7/site-packages/scenicplus/wrappers/run_scenicplus.py", line 366, in run_scenicplus
dill.dump(scplus_obj, f, protocol = -1)
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 336, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 620, in dump
StockPickler.dump(self, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1788, in save_type
if OLD37 or (not obj._field_defaults):
AttributeError: type object 'r2g' has no attribute '_field_defaults'Best Regards, Sayyam
SidG13
commented
1 year ago Is this perhaps related to this issue?
Sayyam-Shah
commented
1 year ago Hello,
Yup, it seems they are related issues. I updated my pyarrow and I'm getting the same error. It would be amazing if we can debug this since I'm excited to analyze the data and use it to contribute to our developing story.
SeppeDeWinter
commented
1 year ago @Sayyam-Shah
Sorry to hear it did not fix the issue.
I'm trying to debug this issue (it might indeed be related to the one of @SidG13; https://github.com/aertslab/scenicplus/issues/60).
Could you try with version 4 of the dill protocol, i.e. dill.dump(scplus_obj, f, protocol=4).
I hope this helps.
Best,
Seppe
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
Thank you for getting back to me. Interesting, it seems they are not related issues. I used protocol =4 and got the same error below.
Traceback (most recent call last):
File "scenicplus_workflow.py", line 77, in <module>
dill.dump(scplus_obj, f, protocol=4)
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 336, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 620, in dump
StockPickler.dump(self, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1251, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/cluster/tools/software/centos7/python/3.7.2/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/cluster/home/.local/lib/python3.7/site-packages/dill/_dill.py", line 1788, in save_type
if OLD37 or (not obj._field_defaults):
AttributeError: type object 'r2g' has no attribute '_field_defaults'Best Regards, Sayyam
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
How do you recommend we best approach this error?
I believe it may be due to the data being added from buildgrn. I am running the buildgrn step on the cluster so I don't have access to the network. Could it be an issue with the class of the object?
2023-01-07 01:31:49,023 SCENIC+_wrapper INFO /cluster/projects/dickgroup/scenicplus_full folder already exists.
2023-01-07 01:31:49,023 SCENIC+_wrapper INFO Build eGRN
2023-01-07 01:31:49,023 GSEA INFO Thresholding region to gene relationships
2023-01-07 01:32:15,029 INFO worker.py:1515 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
100%|██████████| 14/14 [17:26<00:00, 74.74s/it]2023-01-07 01:49:46,733 GSEA INFO Subsetting TF2G adjacencies for TF with motif.
2023-01-07 01:50:06,681 INFO worker.py:1515 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
2023-01-07 01:50:09,652 GSEA INFO Running GSEA...
initializing: 0%| | 66/26272 [00:15<1:51:42, 3.91it/s](_ray_run_gsea_for_e_module pid=56432) /cluster/home/.local/lib/python3.7/site-packages/gseapy/algorithm.py:71: RuntimeWarning: divide by zero encountered in true_divideIs it normal to encounter the above runtime warning? Could it be the driver of the serializing error?
I asked ChatGPT to help debug this error and I got the below response. I hope this helps!
The error message "attributeError: type object 'r2g' has no attribute '_field_defaults'" suggests that the object you are trying to serialize is missing a required attribute "_field_defaults".
It is likely that this attribute is specific to the library or package you are using, so it may be necessary for the proper functioning of the library or package.
As for the protocol to use, since the error message does not seem to be related to the protocol, using protocol 4 or -1 should not affect the error message. If you are using protocol 4 and you are still getting the error, you may need to check that the object you are trying to serialize has the required attribute "_field_defaults", and ensure that it is properly set.
It would be helpful if you can provide more context on the object 'r2g' to understand the reason of the error.
Additionally, if the object is part of a package, you should check the documentation of the package to see if there are any specific requirements for serialization.Hi @Sayyam-Shah
I'm not entirely sure what is going wrong here...
Do you have access to the scplus_obj after running the build_grn step?
if so can you show the output of:
scplus_obj.uns['eRegulons']
scplus_obj.uns['eRegulons'][0].regions2genes
Hello @SeppeDeWinter,
Thank you for getting back to me. I agree this is a very weird error since the build grn step is running fine. I am running the build_grn function on the cluster using 500 GB and 10 cores so I don't have the ability to easily play around with the object like in a jupyter notebook. I ran it again while outputting the eRegulons.
print(scplus_obj.uns['eRegulons'])Output
[eRegulon for TF AHR in context frozenset({'BASC binarized', 'positive r2g', 'Top 5 region-to-gene links per gene', '0.95 quantile', 'Top 10 region-to-gene links per gene', 'Cistromes_Unfiltered', '0.9 quantile', 'positive tf2g', '0.85 quantile', 'Top 15 region-to-gene links per gene'}).
This eRegulon has 317 target regions and 257 target genes., eRegulon for TF ARNT in context frozenset({'positive r2g', '0.95 quantile', 'Cistromes_Unfiltered', '0.9 quantile', 'positive tf2g', '0.85 quantile'}).
This eRegulon has 115 target regions and 119 target genes., eRegulon for TF ATF2 in context frozenset({'BASC binarized', 'positive r2g', 'Top 5 region-to-gene links per gene', '0.95 quantile', 'Top 10 region-to-gene links per gene', 'Cistromes_Unfiltered', '0.9 quantile', 'positive tf2g', '0.85 quantile', 'Top 15 region-to-gene links per gene'}).
This eRegulon has 166 target regions and 170 target genes., eRegulon for TF ATF3 in context frozenset({'BASC binarized', 'positive r2g', 'Top 5 region-to-gene links per gene', '0.95 quantile', 'Top 10 region-to-gene links per gene', 'Cistromes_Unfiltered', '0.9 quantile', 'positive tf2g', '0.85 quantile', 'Top 15 region-to-gene links per gene'}).
This eRegulon has 451 target regions and 237 target genes., eRegulon for TF ATF4 in context frozenset({'BASC binarized', 'positive r2g', 'Top 5 region-to-gene links per gene', '0.95 quantile', 'Top 10 region-to-gene links per gene', 'Cistromes_Unfiltered', '0.9 quantile', 'positive tf2g', '0.85 quantile', 'Top 15 region-to-gene links per gene'}).
(more in the output)
print(scplus_obj.uns['eRegulons'][0].regions2genes)Output:
[r2g(region='chr11:62678259-62679441', target='TMEM179B', importance=0.04323113098650579, rho=0.053295189402782994, importance_x_rho=0.002304011314022347, importance_x_abs_rho=0.002304011314022347), r2g(region='chr15:85329403-85331514', target='AKAP13', importance=0.0246915082631104, rho=0.17430356117307924, importance_x_rho=0.004303817820994655, importance_x_abs_rho=0.004303817820994655), r2g(region='chr3:119809021-119810640', target='COX17', importance=0.08976936593196429, rho=0.06518595500453615, importance_x_rho=0.0058517018484267646, importance_x_abs_rho=0.0058517018484267646), r2g(region='chr11:73280488-73282224', target='RELT', importance=0.02843790904629129, rho=0.09733910687489802, importance_x_rho=0.0027681206679555773, importance_x_abs_rho=0.0027681206679555773), r2g(region='chr3:30606147-30607439', target='TGFBR2', importance=0.16323459386608047, rho=0.10076587794556702, importance_x_rho=0.016448477162003667, importance_x_abs_rho=0.016448477162003667), r2g(region='chr1:160030854-160032761', target='CFAP45', importance=0.1477836361714969, rho=0.06560931845453029, importance_x_rho=0.009695983647944181, importance_x_abs_rho=0.009695983647944181), r2g(region='chr8:124650974-124651929', target='MTSS1', importance=0.015311566307064551, rho=0.22288130499738268, importance_x_rho=0.0034126618800725027, importance_x_abs_rho=0.0034126618800725027),
... - more in the outputDo you recommend I save the two slots separately and add them manually to an existing scenic plus object?
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
I tried to save the eRegulons object below and got the same error. It has something to do with build_grn and the object classes. I can maybe share the output file from the slurm job after outputting the eRegulons slot.
''' with open('/cluster/projects/scripts/scplus_obj.pkl', 'wb') as f: dill.dump(scplus_obj.uns['eRegulons'], f, protocol=4) '''
''' 2023-01-19 02:22:48,997 GSEA INFO Storing eRegulons in .uns[eRegulons].
Traceback (most recent call last):
File "scenicplus_workflow.py", line 78, in
SeppeDeWinter
commented
1 year ago Hi @Sayyam-Shah
It would indeed be useful if you could share the object.
I've also been working on some code to save the splus_obj as a MuData. We are trying to get away from pickling the object (precisely because of the kinds of errors you're getting).
You can also give that a try, even though the code might still be a bit buggy.
The function is in the development branch: https://github.com/aertslab/scenicplus/blob/development/src/scenicplus/io/object_converted.py . And it will return two objects, one for the motif enrichment information and another for the SCENIC+ inference itself.
You will also need to install the development version of pycistarget for this: https://github.com/aertslab/pycistarget/tree/MuData.
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
Thank you! That sounds amazing. I'll try MuData.
I would love to share the data to further scenic plus development and help me debug this serializing error. How would you like me to share it? Just an FYI, I am sending you the single cells and I have been using 500 GB to run the buildgrn step.
Best Regards, Sayyam
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
Upon trying to save the object, I got the below error. I commented out the code that would save the eRegulons since I want to first test it on a lighter step before I run buildgrn.
I'm not exactly sure how to debug this since I inputted the same # of cells from the cistopic and adata objects.
ValueError Traceback (most recent call last)
/scratch/7806817/ipykernel_38252/192711628.py in <module>
----> 1 mdata = scenicplus_object_to_mudata(scplus_obj)
/scratch/7806817/ipykernel_38252/1918707288.py in scenicplus_object_to_mudata(scplus_obj, search_space_key, region_to_gene_key, TF_to_gene_key, eRegulon_AUC_key, eRegulon_metadata_key)
73 uns = uns)
74
---> 75 mdata_menr = dict_motif_enrichment_results_to_mudata(scplus_obj.menr)
76
77 return mdata, mdata_menr
/scratch/7806817/ipykernel_38252/3780393144.py in dict_motif_enrichment_results_to_mudata(menr)
139 if type(menr[key]) == dict and all([isinstance(x, cisTarget) for x in menr[key].values()]):
140 mudata_constructor.update(
--> 141 {f"{key}_{region_set_name}": adata for region_set_name, adata in _dict_of_cisTarget_to_AnnData(menr[key])})
142 elif isinstance(menr[key], DEM):
143 mudata_constructor.update(
/scratch/7806817/ipykernel_38252/3780393144.py in <dictcomp>(.0)
139 if type(menr[key]) == dict and all([isinstance(x, cisTarget) for x in menr[key].values()]):
140 mudata_constructor.update(
--> 141 {f"{key}_{region_set_name}": adata for region_set_name, adata in _dict_of_cisTarget_to_AnnData(menr[key])})
142 elif isinstance(menr[key], DEM):
143 mudata_constructor.update(
/scratch/7806817/ipykernel_38252/3780393144.py in _dict_of_cisTarget_to_AnnData(cistarget_obj_dict)
64 def _dict_of_cisTarget_to_AnnData(cistarget_obj_dict: Mapping[str, cisTarget]) -> Tuple[str, AnnData]:
65 for key in cistarget_obj_dict.keys():
---> 66 yield (key, cisTarget_to_AnnData(cistarget_obj_dict[key]))
67
68 def DEM_to_AnnData(dem_obj: DEM) -> Iterator[Tuple[str, AnnData]]:
/scratch/7806817/ipykernel_38252/3780393144.py in cisTarget_to_AnnData(cistarget_obj)
47 X = motif_hit_matrix, dtype = np.uint8,
48 obs = obs,
---> 49 var = var)
50 #add unstructured data
51 adata.uns['name'] = cistarget_obj.name
~/.local/lib/python3.7/site-packages/anndata/_core/anndata.py in __init__(self, X, obs, var, uns, obsm, varm, layers, raw, dtype, shape, filename, filemode, asview, obsp, varp, oidx, vidx)
303 varp=varp,
304 filename=filename,
--> 305 filemode=filemode,
306 )
307
~/.local/lib/python3.7/site-packages/anndata/_core/anndata.py in _init_as_actual(self, X, obs, var, uns, obsm, varm, varp, obsp, raw, layers, dtype, shape, filename, filemode)
519 _move_adj_mtx({"uns": self._uns, "obsp": self._obsp})
520
--> 521 self._check_dimensions()
522 self._check_uniqueness()
523
~/.local/lib/python3.7/site-packages/anndata/_core/anndata.py in _check_dimensions(self, key)
1842 if "obs" in key and len(self._obs) != self._n_obs:
1843 raise ValueError(
-> 1844 "Observations annot. `obs` must have number of rows of `X`"
1845 f" ({self._n_obs}), but has {self._obs.shape[0]} rows."
1846 )
ValueError: Observations annot. `obs` must have number of rows of `X` (35019), but has 119297 rows.
Hello @SeppeDeWinter,
How do you recommend I integrate the mudata object back into the scenicplus workflow? Because of the above error, I'm wondering if it is necessary to save the mdata_menr since I already have the cistarget data saved using pickle.
Best Regards, Sayyam
SeppeDeWinter
commented
1 year ago Hi @Sayyam-Shah
Unfortunately for now we don't have a convenient function (yet) to convert back to the sceniplus object. I will put a snippet of code below.
It's indeed not necessary to save the mdata_menr.
Here is the snippet of code to convert back to a scenicplus object. This will become a function soon in the package.
With some luck you might be able to save this object using dill.
import mudata
import pickle
from scenicplus.scenicplus_class import SCENICPLUS
mdata = mdata.read_h5mu("<PATH_TO_MDATA>")
menr = pickle.load(open("<PATH_TO_MENR>", "rb"))
scplus_obj = SCENICPLUS(
X_ACC = mdata['ACC'].X.T,
X_EXP = mdata['EXP'].X,
metadata_regions = mdata['ACC'].var,
metadata_genes = mdata['EXP'].var,
metadata_cell = mdata.obs,
menr = menr ,
dr_cell = {k: dict(mdata.obsm)[k] for k in dict(mdata.obsm).keys() if k not in mdata.mod.keys()},
dr_region = {},
uns = mdata.uns)
scplus_obj.uns['eRegulon_AUC'] = {
'Gene_based': mdata['AUC_target_genes'].to_df(),
'Region_based': mdata['AUC_target_regions'].to_df()}
Hope this helps.
Best,
Seppe
Sayyam-Shah
commented
1 year ago Hello @SeppeDeWinter,
Thank you so much. I'll try converting the mudata object using your code above and test dill post-conversion. It helps a lot and we are excited to see how we can apply it to all our multiome data. It will be interesting to as well play around with perturbation analysis with scenic+.
Would you like me to send you our scenic+ object before running build_grn to potentially debug the pickle seralization error? How would you like me to send it to you/to what email?
Best Regards, Sayyam
dyeramosu
commented
1 year ago Hi, wondering if there was any update on this issue? I've been running into the same error when I try to save the scenic plus object during the run_scenicplus step:
from scenicplus.wrappers.run_scenicplus import run_scenicplus
try:
run_scenicplus(
scplus_obj = scplus_obj,
variable = ['GEX_states'],
species = 'hsapiens',
assembly = 'hg38',
tf_file = '/cromwell_root/fc-371bc8c3-78d6-4af4-886b-0d68f87bcc9c/PDAC_scenic_plus/scenicplus_dir/PDAC_scenic_plus_output/utoronto_human_tfs_v_1.01.txt',
save_path = 'scenic_plus_output_wdl/',
biomart_host = 'http://sep2019.archive.ensembl.org/',
upstream = [1000, 150000],
downstream = [1000, 150000],
calculate_TF_eGRN_correlation = False,
calculate_DEGs_DARs = True,
export_to_loom_file = True,
export_to_UCSC_file = True,
path_bedToBigBed = 'tmpdir/',
n_cpu = 24,
_temp_dir = os.path.abspath("tmpdir/"))
except Exception as e:
#in case of failure, still save the object
dill.dump(scplus_obj, open(os.path.join('scenic_plus_output_wdl', 'scplus_obj.pkl'), 'wb'), protocol=-1)
raise(e)my error:
/usr/local/lib/python3.7/dist-packages/sklearn/manifold/_t_sne.py:783: FutureWarning: The default initialization in TSNE will change from 'random' to 'pca' in 1.2.
FutureWarning,
/usr/local/lib/python3.7/dist-packages/sklearn/manifold/_t_sne.py:793: FutureWarning: The default learning rate in TSNE will change from 200.0 to 'auto' in 1.2.
FutureWarning,
/usr/local/lib/python3.7/dist-packages/scanpy/preprocessing/_normalization.py:170: UserWarning: Received a view of an AnnData. Making a copy.
view_to_actual(adata)
/usr/local/lib/python3.7/dist-packages/anndata/_core/anndata.py:1235: ImplicitModificationWarning: Trying to modify attribute `.obs` of view, initializing view as actual.
df[key] = c
... storing 'ACC_sample_id' as categorical
/usr/local/lib/python3.7/dist-packages/scenicplus/diff_features.py:76: FutureWarning: X.dtype being converted to np.float32 from int32. In the next version of anndata (0.9) conversion will not be automatic. Pass dtype explicitly to avoid this warning. Pass `AnnData(X, dtype=X.dtype, ...)` to get the future behavour.
index=scplus_obj.cell_names), var=pd.DataFrame(index=scplus_obj.region_names))
... storing 'ACC_sample_id' as categorical
/usr/local/lib/python3.7/dist-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.
warnings.warn(msg, category=FutureWarning)
/usr/local/lib/python3.7/dist-packages/loomxpy/loomxpy.py:459: FutureWarning: The default value of regex will change from True to False in a future version.
regulons.columns = regulons.columns.str.replace("_?\\(", "_(")
/usr/local/lib/python3.7/dist-packages/loomxpy/loomxpy.py:437: FutureWarning: The default value of regex will change from True to False in a future version.
regulons.columns = regulons.columns.str.replace("_?\\(", "_(")
/usr/local/lib/python3.7/dist-packages/scenicplus/enhancer_to_gene.py:879: RuntimeWarning: invalid value encountered in true_divide
S = (scores - scores.min()) / (scores.max() - scores.min())
/usr/local/lib/python3.7/dist-packages/scenicplus/enhancer_to_gene.py:893: RuntimeWarning: invalid value encountered in true_divide
S = (scores - scores.min()) / (scores.max() - scores.min())
Traceback (most recent call last):
File "<stdin>", line 36, in <module>
File "/usr/local/lib/python3.7/dist-packages/scenicplus/wrappers/run_scenicplus.py", line 380, in run_scenicplus
dill.dump(scplus_obj, f, protocol = -1)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 235, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 394, in dump
StockPickler.dump(self, obj)
File "/usr/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1652, in save_type
if not obj._field_defaults:
AttributeError: type object 'r2g' has no attribute '_field_defaults'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 39, in <module>
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 235, in dump
Pickler(file, protocol, **_kwds).dump(obj)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 394, in dump
StockPickler.dump(self, obj)
File "/usr/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1186, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/lib/python3.7/pickle.py", line 840, in _batch_appends
save(x)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/lib/python3.7/pickle.py", line 633, in save_reduce
save(cls)
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 388, in save
StockPickler.save(self, obj, save_persistent_id)
File "/usr/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/lib/python3.7/dist-packages/dill/_dill.py", line 1652, in save_type
if not obj._field_defaults:
AttributeError: type object 'r2g' has no attribute '_field_defaults'Any help would be appreciated!
Best, Deepika
SeppeDeWinter
commented
1 month ago Closing as we don't pickle the object anymore
Hello,
Thank you for the amazing tool! I am running build_grn, however, I am getting an error when trying to pickle the object. May you please help me debug the error? I've been able to pickle the object easily before, however, it seems I only encounter the error after running build_grn.
Error: