search

KrishnaswamyLab / Multiscale_PHATE

Creating multi-resolution embeddings and clusters from high dimensional data
Other
48 stars 11 forks source link

Slow/stalled run of multiscale phate on large datasets using python 3.7.6 #12

Open PaulFremont3 opened 1 year ago

PaulFremont3 commented 1 year ago

Hello,

I am having trouble running multiscale phate using a dataset with size (>200000, 20).

My code:

" data = load_matrix('data_filename.txt') stats = load_vector('stations_filename.txt ') unigenes = load_vector('genes_filename.txt') df = pd.DataFrame(data, columns = stats, index=unigenes) n_cores=40

print(type(df)) print(df.shape) mp_op = multiscale_phate.Multiscale_PHATE(n_jobs=n_cores, random_state=1) levels = mp_op.fit(df) "

output:: " <class 'pandas.core.frame.DataFrame'> (274774, 20) Calculating Multiscale PHATE tree... Calculating PCA... Calculated PCA in 0.33 seconds. Calculating partitions... " After running PCA quite fast, the algorithm seems to be stalled when calculating partitions: I let it run for several days of calculation using multiple cores (which I verified are running) but the job is still at the same point ('calculating partitions...').

From your examples, multiscale PHATE should run much faster than this on this kind of dataset.

My dataset is: in rows: genes expression (scaled between 0 and 1 with many 0) in columns: different samples

Let me know if you have some insights on this.

Thank you

bai101315 commented 1 week ago

Hello. I've also run into the same mistake as you. The program seems to be stuck in the 'Calculating partitions...' operation all the time. But when I was reproducing the results in the paper, this operation was relatively fast.I would greatly appreciate it if you could provide me with some assistance. Thank you.