search

dpeerlab / Palantir

Single cell trajectory detection
https://palantir.readthedocs.io
GNU General Public License v2.0
213 stars 50 forks source link

compute_gene_trends throws ValueError: X data must not contain Inf nor NaN #98

Closed widsquid closed 1 year ago

widsquid commented 1 year ago

Thanks for creating this tool, it has been interesting so far. Sadly after much success I now get this error with the trend computation function. I have checked for NAs and Infs and zeros. I have tried to create a tester imp_df of random floats (no NAs nan Inf zeros) to input instead and still the same result. Any idea what could be the problem? I installed new from from git clone yesterday. All my data comes straight from 10x output. I am wondering could the NA be from the pr_res data? Thanks so much.

gene_trends = palantir.presults.compute_gene_trends(pr_res, imp_df.loc[:, genes])

_RemoteTraceback Traceback (most recent call last) _RemoteTraceback: """ Traceback (most recent call last): File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 428, in _process_worker r = call_item() File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 275, in call return self.fn(*self.args, *self.kwargs) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 620, in call return self.func(args, **kwargs) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py", line 289, in call for func, args, kwargs in self.items] File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py", line 289, in for func, args, kwargs in self.items] File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py", line 162, in gam_fit_predict y_pred = gam.predict(pred_x) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py", line 434, in predict return self.predict_mu(X) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py", line 414, in predict_mu features=self.feature, verbose=self.verbose) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py", line 273, in check_X name='X data', verbose=verbose) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py", line 171, in check_array raise ValueError('{} must not contain Inf nor NaN'.format(name)) ValueError: X data must not contain Inf nor NaN """

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last) /tmp/ipykernel_1341918/4064161686.py in ----> 1 gene_trends = palantir.presults.compute_gene_trends(pr_res, imp_df.loc[:, genes])

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py in compute_gene_trends(pr_res, gene_exprs, lineages, n_splines, spline_order, n_jobs) 120 spline_order 121 ) --> 122 for gene in gene_exprs.columns 123 ) 124

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in call(self, iterable) 1096 1097 with self._backend.retrieval_context(): -> 1098 self.retrieve() 1099 # Make sure that we get a last message telling us we are done 1100 elapsed_time = time.time() - self._start_time

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in retrieve(self) 973 try: 974 if getattr(self._backend, 'supports_timeout', False): --> 975 self._output.extend(job.get(timeout=self.timeout)) 976 else: 977 self._output.extend(job.get())

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout) 565 AsyncResults.get from multiprocessing.""" 566 try: --> 567 return future.result(timeout=timeout) 568 except CfTimeoutError as e: 569 raise TimeoutError from e

~/anaconda3/envs/env_palantir/lib/python3.7/concurrent/futures/_base.py in result(self, timeout) 433 raise CancelledError() 434 elif self._state == FINISHED: --> 435 return self.__get_result() 436 else: 437 raise TimeoutError()

~/anaconda3/envs/env_palantir/lib/python3.7/concurrent/futures/_base.py in __get_result(self) 382 def __get_result(self): 383 if self._exception: --> 384 raise self._exception 385 else: 386 return self._result

ValueError: X data must not contain Inf nor NaN

ManuSetty commented 1 year ago

Can you please run the function with n_jobs=1. This will help isolate where the issue is for better debugging.

widsquid commented 1 year ago

Thanks so much for the reply. Here is the output. Let me know if there is any other info that might help.

gene_trends = palantir.presults.compute_gene_trends(pr_res, imp_df.loc[:, genes], n_jobs=1)

ValueError Traceback (most recent call last) /tmp/ipykernel_1341918/2426816647.py in ----> 1 gene_trends = palantir.presults.compute_gene_trends(pr_res, imp_df.loc[:, genes], n_jobs=1)

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py in compute_gene_trends(pr_res, gene_exprs, lineages, n_splines, spline_order, n_jobs) 120 spline_order 121 ) --> 122 for gene in gene_exprs.columns 123 ) 124

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in call(self, iterable) 1083 # remaining jobs. 1084 self._iterating = False -> 1085 if self.dispatch_one_batch(iterator): 1086 self._iterating = self._original_iterator is not None 1087

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator) 899 return False 900 else: --> 901 self._dispatch(tasks) 902 return True 903

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in _dispatch(self, batch) 817 with self._lock: 818 job_idx = len(self._jobs) --> 819 job = self._backend.apply_async(batch, callback=cb) 820 # A job can complete so quickly than its callback is 821 # called before we get here, causing self._jobs to

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback) 206 def apply_async(self, func, callback=None): 207 """Schedule a func to be run""" --> 208 result = ImmediateResult(func) 209 if callback: 210 callback(result)

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py in init(self, batch) 595 # Don't delay the application, to avoid keeping the input 596 # arguments in memory --> 597 self.results = batch() 598 599 def get(self):

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in call(self) 287 with parallel_backend(self._backend, n_jobs=self._n_jobs): 288 return [func(*args, **kwargs) --> 289 for func, args, kwargs in self.items] 290 291 def reduce(self):

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in (.0) 287 with parallel_backend(self._backend, n_jobs=self._n_jobs): 288 return [func(*args, **kwargs) --> 289 for func, args, kwargs in self.items] 290 291 def reduce(self):

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py in gam_fit_predict(x, y, weights, pred_x, n_splines, spline_order) 160 if pred_x is None: 161 pred_x = x --> 162 y_pred = gam.predict(pred_x) 163 164 # Standard deviations

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py in predict(self, X) 432 containing predicted values under the model 433 """ --> 434 return self.predict_mu(X) 435 436 def _modelmat(self, X, term=-1):

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py in predict_mu(self, X) 412 X = check_X(X, nfeats=self.statistics['m_features'], 413 edge_knots=self.edgeknots, dtypes=self.dtype, --> 414 features=self.feature, verbose=self.verbose) 415 416 lp = self._linear_predictor(X)

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py in check_X(X, n_feats, min_samples, edge_knots, dtypes, features, verbose) 271 # basic diagnostics 272 X = check_array(X, force_2d=True, n_feats=n_feats, min_samples=min_samples, --> 273 name='X data', verbose=verbose) 274 275 # check our categorical data has no new categories

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py in check_array(array, force_2d, n_feats, ndim, min_samples, name, verbose) 169 # check finite 170 if not(np.isfinite(array).all()): --> 171 raise ValueError('{} must not contain Inf nor NaN'.format(name)) 172 173 # check ndim

ValueError: X data must not contain Inf nor NaN

ManuSetty commented 1 year ago

Can you please check if there is NA or inf in imp_df. imp_df.isna().sum() and imp_df.isinf().sum()

widsquid commented 1 year ago

Sure thanks so much.

Screen Shot 2023-02-18 at 10 46 39 PM Screen Shot 2023-02-18 at 10 46 23 PM
ManuSetty commented 1 year ago

Can you please share the output of pr_res by running the palantir.plot.plot_palantir_results? Looks like it is an issue in there and not from imputation.

widsquid commented 1 year ago
Screen Shot 2023-02-21 at 1 36 55 AM
ManuSetty commented 1 year ago

Do any of pr_res.pseudotime or pr_res.branch_probs contain infinities or NA?

widsquid commented 1 year ago

It does not appear that way. Is this the correct way of determining that? Thanks again. W

From: Manu Setty @.> Sent: February 21, 2023 9:52 AM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Do any of pr_res.pseudotime or pr_res.branch_probs contain infinities or NA?

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1438881842, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X7NTKQMOPQJNW3HM2LWYT6FVANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

This is puzzling! Do you mind sharing a pickle object of pr_res and imp_df? I can examine it in my end to better understand this.

widsquid commented 1 year ago

Archive.zip

widsquid commented 1 year ago

sure, thanks for your efforts. please let me know if you do not successfully receive the files. W

ManuSetty commented 1 year ago

Sorry, can you please clarify what format the file is saved and how I can load it.

widsquid commented 1 year ago

Unzip to find two .pkl files I believe. W

On Mar 4, 2023, at 8:49 PM, Manu Setty @.***> wrote:



Sorry, can you please clarify what format the file is saved and how I can load it.

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1454989545, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X7JRFRFZVSG4KYIWOLW2QLODANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

widsquid commented 1 year ago

were you able to successfully unzip the file? my mistake I should have said two .obj files saved using pkl

ManuSetty commented 1 year ago

Looks like the pr_res file might be corrupted. Can you please share again?

image
widsquid commented 1 year ago

[https://res-geo.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]pr_res.objhttps://1drv.ms/u/s!Au68to3eRf-VgQllb57d_tBHrgMQ [https://res-geo.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]imp_df.objhttps://1drv.ms/u/s!Au68to3eRf-VgQpWY0tr_okzo3ym

From: Manu Setty @.> Sent: March 9, 2023 12:49 PM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Looks like the pr_res file might be corrupted. Can you please share again? [image]https://user-images.githubusercontent.com/2151404/224153963-e4d4fbbc-b1ab-4845-bb7f-e4854f0553f0.png

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1462783909, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X4CEAK54KHYS5AGJ4DW3I64FANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

Still the same issue.

image

What version of pickle are you using to save?

widsquid commented 1 year ago

I tried to save the files again. Let me know if you can get them loaded. Thanks again for all the trouble, everything else looks great for the figures I am trying to make I except I would really like to have some of these plots in there as well. I hope we can get to the bottom of this. [https://res-geo.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]imp_df.objhttps://1drv.ms/u/s!Au68to3eRf-VgQuTVwUbkpsnIBv0 [https://res-geo.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]pr_res.objhttps://1drv.ms/u/s!Au68to3eRf-VgQxeDA9qYi_khK-8

W

From: Manu Setty @.> Sent: March 9, 2023 9:45 PM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Still the same issue. [image]https://user-images.githubusercontent.com/2151404/224233546-0f121573-fb60-4094-96a4-0286db27b6fe.png

What version of pickle are you using to save?

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1463307171, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X5RWQPIHAK4ZS4JIOLW3K5XPANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

Sorry still the same issue

image

Perhaps you can save the information in there as text: pr_res.pseudotime.to_csv('pt.csv') pr_res.branch_probs.to_csv('branch_probs.csv') pr_res.entropy.to_csv('entropy.csv')

widsquid commented 1 year ago

[https://res-geo.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]branch_probs.csvhttps://1drv.ms/u/s!Au68to3eRf-VgQ3g0zf8xiFj29Fs [https://res-geo.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]entropy.csvhttps://1drv.ms/u/s!Au68to3eRf-VgQ5AV0sHwxrf6r66 [https://res-geo.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]pt.csvhttps://1drv.ms/u/s!Au68to3eRf-VgQ_92woP8mm_u6Q4 My apologies, certainly, here they are. W

From: Manu Setty @.> Sent: March 11, 2023 6:20 PM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Sorry still the same issue [image]https://user-images.githubusercontent.com/2151404/224520554-f2cb0331-760c-464f-bde1-3c67c3b957ab.png

Perhaps you can save the information in there as text: pr_res.pseudotime.to_csv('pt.csv') pr_res.branch_probs.to_csv('branch_probs.csv') pr_res.entropy.to_csv('entropy.csv')

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1465073729, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2XY43EOUK5PFYQGOLOTW3UXGZANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

Can you please email to manu.n.setty@gmail.com? The above links are causing Onedrive to try and convert the files which never finishes.

ManuSetty commented 1 year ago

This looks an issue with a subset of the branches. If you use the Peri or SRC lineages, the trends will be computed without issues

image

For the Failed lineage, the maximum probability of any given cell is 0.4. When assigning cells to branches, our code looks for cells with at least 0.7 probability (https://github.com/dpeerlab/Palantir/blob/4d4f9fcdbfebcf47b4ca1b7ac8a0fb1b8e5fc089/src/palantir/presults.py#L94). You will need to make a tweak in this line to get trends for this lineage.

widsquid commented 1 year ago

Amazing I can not wait try it. I will let you know if it works. Thanks so much. W

On Mar 12, 2023, at 7:00 PM, Manu Setty @.***> wrote:



This looks an issue with a subset of the branches. If you use the Peri or SRC lineages, the trends will be computed without issues [image]https://user-images.githubusercontent.com/2151404/224591154-7b76be98-83d4-4685-a8d0-49b7ad8f8399.png

For the Failed lineage, the maximum probability of any given cell is 0.4. When assigning cells to branches, our code looks for cells with at least 0.7 probability (https://github.com/dpeerlab/Palantir/blob/4d4f9fcdbfebcf47b4ca1b7ac8a0fb1b8e5fc089/src/palantir/presults.py#L94). You will need to make a tweak in this line to get trends for this lineage.

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1465393474, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X6ACRSQFXIOIQ23XJLW32ET3ANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

widsquid commented 1 year ago

So you are correct, it works if I restrict the lineages to src and peri but still does not work for failed even if the setting is > 0.1. Does that makes sense?

widsquid commented 1 year ago

Same result if I modify the threshold to 0.0. Would that be typical in a case like this? Also what does it say about this particular lineage if it is not calling high probability cells?

ManuSetty commented 1 year ago

It is puzzling that lowering the threshold did not work - are you seeing the error at the same code location?

Low probability on a lineage indicates that it might not be a true terminal state.

widsquid commented 1 year ago

there is a possibility it is not a true terminal state, perhaps rather a failed trajectory. There are however interesting differentially expressed genes in that region. Is it still possible to plot those on a trajectory? I suppose perhaps lowering the threshold should allow this if working?

This is the error, seems like the same outcome at least (of NA or undefined values being generated):

_RemoteTraceback Traceback (most recent call last) _RemoteTraceback: """ Traceback (most recent call last): File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 428, in _process_worker r = call_item() File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py", line 275, in call return self.fn(*self.args, *self.kwargs) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 620, in call return self.func(args, **kwargs) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py", line 289, in call for func, args, kwargs in self.items] File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py", line 289, in for func, args, kwargs in self.items] File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py", line 162, in gam_fit_predict y_pred = gam.predict(pred_x) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py", line 434, in predict return self.predict_mu(X) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/pygam.py", line 414, in predict_mu features=self.feature, verbose=self.verbose) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py", line 273, in check_X name='X data', verbose=verbose) File "/home/w/anaconda3/envs/env_palantir/lib/python3.7/site-packages/pygam/utils.py", line 171, in check_array raise ValueError('{} must not contain Inf nor NaN'.format(name)) ValueError: X data must not contain Inf nor NaN """

The above exception was the direct cause of the following exception:

ValueError Traceback (most recent call last) /tmp/ipykernel_2663296/3306120243.py in 3 imp_df = pd.DataFrame(ad[:, genes].layers['MAGIC_imputed_data'], 4 index=ad.obs_names, columns=genes) ----> 5 gene_trends = palantir.presults.compute_gene_trends( pr_res, imp_df.loc[:, genes])#, lineages = ['SRC'], n_jobs = 1) 6 #gene_trends = palantir.presults.compute_gene_trends( pr_res, imp_df.loc[:, genes], n_jobs = 1)

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/palantir/presults.py in compute_gene_trends(pr_res, gene_exprs, lineages, n_splines, spline_order, n_jobs) 120 spline_order 121 ) --> 122 for gene in gene_exprs.columns 123 ) 124

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in call(self, iterable) 1096 1097 with self._backend.retrieval_context(): -> 1098 self.retrieve() 1099 # Make sure that we get a last message telling us we are done 1100 elapsed_time = time.time() - self._start_time

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/parallel.py in retrieve(self) 973 try: 974 if getattr(self._backend, 'supports_timeout', False): --> 975 self._output.extend(job.get(timeout=self.timeout)) 976 else: 977 self._output.extend(job.get())

~/anaconda3/envs/env_palantir/lib/python3.7/site-packages/joblib/_parallel_backends.py in wrap_future_result(future, timeout) 565 AsyncResults.get from multiprocessing.""" 566 try: --> 567 return future.result(timeout=timeout) 568 except CfTimeoutError as e: 569 raise TimeoutError from e

~/anaconda3/envs/env_palantir/lib/python3.7/concurrent/futures/_base.py in result(self, timeout) 433 raise CancelledError() 434 elif self._state == FINISHED: --> 435 return self.__get_result() 436 else: 437 raise TimeoutError()

~/anaconda3/envs/env_palantir/lib/python3.7/concurrent/futures/_base.py in __get_result(self) 382 def __get_result(self): 383 if self._exception: --> 384 raise self._exception 385 else: 386 return self._result

ValueError: X data must not contain Inf nor NaN

ManuSetty commented 1 year ago

Can you please check that br_cells parameter has non-zero cells here: https://github.com/dpeerlab/Palantir/blob/4d4f9fcdbfebcf47b4ca1b7ac8a0fb1b8e5fc089/src/palantir/presults.py#L94 after you reduce the threshold and reinstall the package.

widsquid commented 1 year ago

Hello again, sorry for the delay. I was able to get back to it and I have run the script again I believe with the modified threshold. Can we assume that the threshold is set to zero based on running the command in my attached image and seeing the 0.0 in that line? Thanks again.

Screen Shot 2023-04-22 at 7 55 06 PM Screen Shot 2023-04-22 at 7 54 48 PM
widsquid commented 1 year ago
Screen Shot 2023-04-22 at 8 41 11 PM Screen Shot 2023-04-22 at 8 43 41 PM
ManuSetty commented 1 year ago

This should have worked. I can investigate this on my end - can you please reshape the files again? Sorry, I cannot seem to locate them.

widsquid commented 1 year ago

Certainly, my pleasure, thanks for taking a look. W

widsquid commented 1 year ago

[https://res-h3.public.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]pr_res.objhttps://1drv.ms/u/s!Au68to3eRf-VgSCjrJrn3-xmPvSc [https://res.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]entropy.csvhttps://1drv.ms/u/s!Au68to3eRf-VgSH4vvIIahbL9OpF [https://res.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]branch_probs.csvhttps://1drv.ms/u/s!Au68to3eRf-VgSICvCrLNuKza4Sb [https://res-h3.public.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png]imp_df.objhttps://1drv.ms/u/s!Au68to3eRf-VgSPrSsrskMTAMixn [https://res.cdn.office.net/assets/mail/file-icon/png/csv_16x16.png]pt.csvhttps://1drv.ms/u/s!Au68to3eRf-VgSTuS6aOaJLAsc8u

Are these all the files you need? Thanks. W

From: Manu Setty @.> Sent: April 25, 2023 12:05 PM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

This should have worked. I can investigate this on my end - can you please reshape the files again? Sorry, I cannot seem to locate them.

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1522278217, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X2USIL5Y3W6CYBL3F3XDAN7TANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

widsquid commented 1 year ago

FYI when computing gene trends, "SRC13" trajectory works alone, "Peri" works alone but they do not work together and "Failed" will not work alone or with any combination.

ManuSetty commented 1 year ago

The links dont seem to be working - can you please me again manu.n.setty <AT> gmail.com

widsquid commented 1 year ago

sorry, yes no problem! W

On Apr 26, 2023, at 4:00 PM, Manu Setty @.***> wrote:



The links dont seem to be working - can you please me again manu.n.setty gmail.com

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1523972792, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X6DJBXCDIJNE7S6GIDXDF5E3ANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

Hello again

The changes we discussed did work for me and the gene trend computation was successful for all lineage and genes you shared with me

image
widsquid commented 1 year ago

Ok that is interesting. Any thoughts as to why it might be returning that error on my end? I could try a fresh install of the package? W

From: Manu Setty @.> Sent: April 30, 2023 9:57 PM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Hello again

The changes we discussed did work for me and the gene trend computation was successful for all lineage and genes you shared with me

[image]https://user-images.githubusercontent.com/2151404/235408863-431e794a-1853-42de-a409-03bd4784ecdc.png

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1529346219, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X7E3NGQR7Y6RNW33M3XD47FDANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

ManuSetty commented 1 year ago

Yes - please pull the latest repo, make the changes and then reinstall. Please try starting your notebook server only after re-installing the package.

widsquid commented 1 year ago

Sounds good, I will give it a try with my fingers crossed and let you know how it goes. Thanks again! W

From: Manu Setty @.> Sent: May 1, 2023 10:55 AM To: dpeerlab/Palantir @.> Cc: widsquid @.>; Author @.> Subject: Re: [dpeerlab/Palantir] compute_gene_trends throws ValueError: X data must not contain Inf nor NaN (Issue #98)

Yes - please pull the latest repo, make the changes and then reinstall. Please try starting your notebook server only after re-installing the package.

— Reply to this email directly, view it on GitHubhttps://github.com/dpeerlab/Palantir/issues/98#issuecomment-1530011445, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIYR2X5ZULQWE23DJDUCII3XD72JXANCNFSM6AAAAAAU63KD64. You are receiving this because you authored the thread.Message ID: @.***>

widsquid commented 1 year ago

Some progress? After a reinstall and open a new notebook I got a second trajectory on the trend plots but it ignored the third one.

['SRC13', 'Peri', 'Failed']

Screen Shot 2023-05-01 at 5 47 33 PM Screen Shot 2023-05-01 at 5 01 26 PM
ManuSetty commented 1 year ago

Does the gene trends object not contain the results for the other lineages?

Manu

On Mon, May 1, 2023 at 2:48 PM widsquid @.***> wrote:

Some progress? After a reinstall and open a new notebook I got a second trajectory on the trend plots but it ignored the third one.

['SRC13', 'Peri', 'Failed'] [image: Screen Shot 2023-05-01 at 5 47 33 PM] https://user-images.githubusercontent.com/36773215/235537423-4f89b5a8-e0ab-48d3-8bfc-89af168958ba.png [image: Screen Shot 2023-05-01 at 5 01 26 PM] https://user-images.githubusercontent.com/36773215/235537449-ec8bd4ae-cf58-4866-b775-4ab3ff47d8a6.png

— Reply to this email directly, view it on GitHub https://github.com/dpeerlab/Palantir/issues/98#issuecomment-1530357266, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQNH3HJ37H7WFQ5CY5GQXDXEAVSPANCNFSM6AAAAAAU63KD64 . You are receiving this because you commented.Message ID: @.***>

widsquid commented 1 year ago

No good point, it does not. Looks like just for the two that are plotted, trends and std each (49 genes x 500 waypoints).

widsquid commented 1 year ago

I sent some information to your gmail. Thanks again. W

widsquid commented 1 year ago

Thank you, with the clues you provided I was able to work out where the problem was coming from. I tested a number of terminal cell combinations and and thresholds and determined that with certain combinations of terminal cells, the probabilities were indeed dropping well below the threshold. We were also able to reproduce this error in another dataset that had previously resolved four trajectories by simply setting an additional terminal cell too close to one of the existing terminal cells, thereby reducing the confidence of the existing trajectory to below the threshold I suppose. In any case thank you for your efforts.