jc-healy
commented
4 years ago Hi there,
My longer term and probably less useful answer is that we're currently working through the details and theory of how best to apply UMAP style embeddings to text documents. It can be useful to remove global effects of words from your representation. Tfidf does a decent job of downweighting these effects but it very much looks like you need to apply stronger transformers to remove the global (or at least broad) language effects that are tying together your document space. Our preliminary work, in case you are interested, can be found here: https://github.com/TutteInstitute/TextMAP. We don't have much documentation or a completed paper ready yet but the building blocks for various pre-UMAP transformers can be found there if you are feeling very keen.
That said, I think the crux of your problem is that you'd like to incorporate some knowledge of your labels into your unsupervised embedding. Have you tried incorporating the semi-supervised aspects of UMAP to attempt to learn an embedding that better separates your site categories or such things?
On Thu, Aug 13, 2020 at 12:52 PM Dennis notifications@github.com wrote:
I am embedding large noisy text docs (websites) using UMAP for further downstream tasks, including classification (site category), regression (advertising performance) and clustering.
Sometimes the embedding works well and others less so! I am still not able to understand how to tune the process and ensure we can achieve an optimal embedding, the idea is to have a single embedding which can be used across the various tasks (if possible).
The main issue I am having is that often there is not huge amount of separation in the data, often the result is as below: [image: image] https://user-images.githubusercontent.com/10243849/90162867-322a9480-dd8d-11ea-9d21-215816a99a67.png
Where we have one large group and then a few well defined smaller groupings. This can be explained that lost of internet / news content can be quite generic, but I am still looking for a better way to separate articles even say if they are both news yet one is financial news whilst the other climate.
How does UMAP deal with more data for example is one global embedding learnt on millions on various sites a good idea, or am I better off to split this up in categorised embeddings?
What hyperparams would be important in such a task?
One more piece of useful info would be I am applying tfidf on my raw docs before passing data to UMAP.
Sorry about the long rant and many questions 👍
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/lmcinnes/umap/issues/478, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3IUWWTIIUL7TAWEIF2USDSAQK6LANCNFSM4P6UPHZA .
ydennisy
I am embedding large noisy text docs (websites) using UMAP for further downstream tasks, including classification (site category), regression (advertising performance) and clustering.
Sometimes the embedding works well and others less so! I am still not able to understand how to tune the process and ensure we can achieve an optimal embedding, the idea is to have a single embedding which can be used across the various tasks (if possible).
The main issue I am having is that often there is not huge amount of separation in the data, often the result is as below:
Where we have one large group and then a few well defined smaller groupings. This can be explained that lost of internet / news content can be quite generic, but I am still looking for a better way to separate articles even say if they are both
newsyet one isfinancial newswhilst the otherclimate.How does UMAP deal with more data for example is one global embedding learnt on millions on various sites a good idea, or am I better off to split this up in categorised embeddings?
What hyperparams would be important in such a task?
One more piece of useful info would be I am applying tfidf on my raw docs before passing data to UMAP.
Sorry about the long rant and many questions 👍
EDIT: just one more Q, is it ever a good idea to scale the output of UMAP?