A dataset that will not cluster

[Stein-ErikG]

Hi,

I have been enjoying FastPG now for some time, clustering up to 8 million cells in one hour or more on my iMAC.

I am now stuck on a new dataset that just will not cluster. Im using the same code but it seems to hang at some point during clustering. If i subset to a low enough total number of cells (less than 200 000), it will however complete in less than 30 seconds. If I subset to 300 000 or more cells in total, it will not finish even if left over night.

I have gone back to my bigger dataset and there i clustered 1 million cells in 10 mins, so there seems to be no issues with the local installation of FastPG in R.

I appreciate that this might be really difficult to debug for you. Any suggestions on what to check would be helpful.

As a note, the dataset that is troublesome has quite a lot more heterogeneity of phenotypes between than my other and bigger dataset.

[sararselitsky]

I'm glad FastPG has been working well for you! We've had a similar issue described before. Can you run each step separately to see where it hangs?

On Thu, Oct 14, 2021 at 7:40 AM Stein-ErikG @.***> wrote:

Hi,

I have been enjoying FastPG now for some time, clustering up to 8 million cells in one hour or more on my iMAC.

I am now stuck on a new dataset that just will not cluster. Im using the same code but it seems to hang at some point during clustering. If i subset to a low enough total number of cells (less than 200 000), it will however complete in less than 30 seconds. If I subset to 300 000 or more cells in total, it will not finish even if left over night.

I have gone back to my bigger dataset and there i clustered 1 million cells in 10 mins, so there seems to be no issues with the local installation of FastPG in R.

I appreciate that this might be really difficult to debug for you. Any suggestions on what to check would be helpful.

As a note, the dataset that is troublesome has quite a lot more heterogeneity of phenotypes between than my other and bigger dataset.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sararselitsky/FastPG/issues/17, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQD6T6JSTRLUBCPAM5QO53UG26RHANCNFSM5F7OGOBA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Stein-ErikG]

i managed to run each step separately and it hung on (did not complete)

Parallel Louvain clustering

clusters <- FastPG::parallel_louvain( links )

with no error messages. I clustered 131 000 cells in 30 seconds, but 258 000 cells did not finish even for over 12 hours. How can i debug this?

[sararselitsky]

Mahantesh created Grappollo, which is the step that had the issue. Mahantesh, can you help us debug this?

On Sun, Oct 17, 2021 at 2:53 AM Stein-ErikG @.***> wrote:

i managed to run each step separately and it stopped on Parallel Louvain clustering

clusters <- FastPG::parallel_louvain( links )

with no error messages. I clustered 131 000 cells in 30 seconds, but 258 000 cells did not finish even for over 12 hours. How can i debug this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sararselitsky/FastPG/issues/17#issuecomment-945060875, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQD6T4XGG64G2MU6K76QJ3UHJXHTANCNFSM5F7OGOBA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[sararselitsky]

Hi Sara,

I think the graph did not get built correctly. Is it possible to share the input?

In particular, what was the last line that Grappolo printed?

Regards, Mahantesh P.S. Sent from my iPhone. Please excuse the typos and brevity.

On Oct 17, 2021, at 7:04 AM, Sara Selitsky @.***> wrote:

 Check twice before you click! This email originated from outside PNNL.

Mahantesh created Grappollo, which is the step that had the issue. Mahantesh, can you help us debug this?

On Sun, Oct 17, 2021 at 2:53 AM Stein-ErikG @.**@.>> wrote:

i managed to run each step separately and it stopped on

Parallel Louvain clustering

clusters <- FastPG::parallel_louvain( links )

with no error messages. I clustered 131 000 cells in 30 seconds, but 258 000 cells did not finish even for over 12 hours. How can i debug this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsararselitsky%2FFastPG%2Fissues%2F17%23issuecomment-945060875&data=04%7C01%7CMahantesh.Halappanavar%40pnnl.gov%7C4c1e8246ed004fd776eb08d991770f0d%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637700762808191716%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZpC5aYXQ32ML9UVWuX674rdBodbm2q2GyLqQelWMnh4%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FACQD6T4XGG64G2MU6K76QJ3UHJXHTANCNFSM5F7OGOBA&data=04%7C01%7CMahantesh.Halappanavar%40pnnl.gov%7C4c1e8246ed004fd776eb08d991770f0d%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637700762808201663%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gko%2FIj5shHQ0xjzNO5hc8OMHs%2BvK8VEqD4g9PpsAzDI%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CMahantesh.Halappanavar%40pnnl.gov%7C4c1e8246ed004fd776eb08d991770f0d%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637700762808201663%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=vdE2A0%2BABfUyqRsqddvRBiFkb3DEZdvb2uaxSgSDY20%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CMahantesh.Halappanavar%40pnnl.gov%7C4c1e8246ed004fd776eb08d991770f0d%7Cd6faa5f90ae240338c0130048a38deeb%7C0%7C0%7C637700762808211618%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BItocY9GFM2NYKWM2jGFp1Ek4jehr6n7QYHV9C%2FO7nc%3D&reserved=0.