koheiw
commented
1 year ago Thanks for the nice example code. Please see my inline comments.
# load the required packages
library(quanteda)
library(seededlda)
# Create a corpus from your text data
my_corpus <- corpus(c("The field of artificial intelligence is expanding with new breakthroughs.",
"Data analysis and machine learning are important in extracting insights.",
"Cybersecurity measures are crucial to protect sensitive information.",
"Health and fitness play a significant role in overall well-being.",
"The economy is affected by global market trends and trade policies.",
"Education is essential for personal growth and career opportunities.",
"Climate change and environmental sustainability are pressing issues.",
"Social media platforms are shaping communication and information sharing.",
"The entertainment industry is evolving with new digital platforms.",
"Urbanization and infrastructure development are transforming cities."))
# Define the seed words for each topic
seed_list <- list(topic1 = c("artificial intelligence", "machine learning", "data analysis"),
topic2 = c("cybersecurity", "privacy", "data protection"),
topic3 = c("climate change", "sustainability", "environmental impact"))
dict <- dictionary(seed_list)
# Create a document-feature matrix with unigram and bigram features
toks <- tokens(my_corpus, remove_punct = TRUE, remove_symbols = TRUE, remove_number = TRUE) |>
tokens_remove(stopwords("en"), min_nchar = 2) |> # remove common words using stopwords
#tokens_ngrams(n = 1:2) |> # unigrams of phrases usually appear in the same topic
tokens_compound(dict) # for multi-word expressions
dfmt <- dfm(toks) |>
dfm_remove(stopwords('en')) |>
dfm_trim(min_termfreq = 0.5, termfreq_type = "quantile") # max_docfreq might remove too many words
# Run seeded LDA
slda <- textmodel_seededlda(dfmt, dict, residual = TRUE) # consider using batch_size and auto_iter when data is large
# extract topic classification for each document
topics <- topics(slda)
terms(slda)
#> topic1 topic2 topic3
#> [1,] "artificial_intelligence" "cybersecurity" "climate_change"
#> [2,] "data_analysis" "information" "sustainability"
#> [3,] "machine_learning" "expanding" "play"
#> [4,] "field" "new" "education"
#> [5,] "new" "crucial" "essential"
#> [6,] "breakthroughs" "protect" "personal"
#> [7,] "important" "affected" "growth"
#> [8,] "measures" "market" "environmental"
#> [9,] "sensitive" "trade" "pressing"
#> [10,] "significant" "career" "issues"
#> other
#> [1,] "platforms"
#> [2,] "extracting"
#> [3,] "insights"
#> [4,] "health"
#> [5,] "fitness"
#> [6,] "economy"
#> [7,] "trends"
#> [8,] "policies"
#> [9,] "field"
#> [10,] "artificial_intelligence"I have never used LDAvis seriously but please try if this works. I could add a method for this to my package.
library(LDAvis)
#> Loading required package: LDAvis
lda_vis <- createJSON(phi = slda$phi,
theta = slda$theta,
doc.length = ntoken(slda$data),
vocab = featnames(slda$data),
term.frequency = featfreq(slda$data))
serVis(lda_vis)Created on 2023-06-06 with reprex v2.0.2
hahoangnhan
Hi @koheiw ,
I have two issues that I hope to hear from you some advice and suggestion.
I - tokens or tokens_ngrams
Just to know and do my case correctly. Please give me any advice or suggestion.
I have a list of guided keywords which are both uni and n-grams. So, when tokenizing my corpus, I am not sure about using tokens or tokens_ngrams is the most suitable one. Q1: What should I do if I expect
Q2: If I have n-grams, should I use
tokens_compound?Q3: do I need to remove stop words when using n-grams?
Please see my example code below.
II - Use LDAvis to visualize and modify keywords, topics
As I need to do several rounds of seeded LDA to make the topics are separated clearly and combine topics if they are quite close to each other. I would like to use LDAvis to see more intuitively. I tried and got the error as below:
Q4: Is it because LDAvis does not work with seededlda output? If it works, what should I do? Is there any alternative visualization tools?
Thank you so much for your time and consideration.
Best,
HHN