jwijffels
commented
3 years ago Thanks for the report. Function top2vec uses dbscan::hdbscan to cluster the result of the embedding from UMAP (which is a document embedding space with for your example 2 dimensions.
The error is coming fromdbscan::hdbscan. Maybe this should be asked at this repository (https://github.com/mhahsler/dbscan)
In order to debug this, you will need to provide the matrix which is passed on to hdbscan and report to that repository.
To get it, you can do the following to reproduce the error and provide embedding_umap to the authors at https://github.com/mhahsler/dbscan
emb <- as.matrix(d2v)
embedding_umap <- uwot::tumap(emb , n_neighbors = 100L, n_components = 2, metric = "cosine")
thisshouldfail <- dbscan::hdbscan(embedding_umap , minPts = 25)
I have been running doc2vec and top2vec on a Unix server. However, as I increase the data size I ran into the following error:
This is happening when calling the following top2vec code:
Which refers to the following doc2vec line (note I have also tried using doc2vec models with fewer dimensions (50) and iterations (25)):
This only happens when running t2v on my "full" data, which has 137649 rows, 4 columns (
doc_id,date,origin,text) and takes up around 200mb. When running on a subset of the data (20% sample), I do not run into this error. With the full data, doc2vec runs correctly; the issue is only with top2vec.This seems to happen regardless of the options I specify for t2v (I've tried with different combinations of
minPts,n_neighbors, andn_components). I've also tried increasing the amount of RAM. With this same dataset, I've tried using as much as 600GB at a time, with the same error.I am happy to provide any other information that may be useful, and can email the data itself if that would be helpful.
Here is the traceback:
Here is the output of
sessionInfo():