juliasilge
commented
4 years ago Hello @kathy-j-lee! 👋 I am sorry you are having trouble; that sounds frustrating. I just worked up this example, and I believe things are working as expected:
library(tidyverse)
library(tidytext)
library(stm)
#> stm v1.3.5 successfully loaded. See ?stm for help.
#> Papers, resources, and other materials at structuraltopicmodel.com
library(janeaustenr)
books <- austen_books() %>%
group_by(book) %>%
mutate(chapter = cumsum(str_detect(text, regex("^chapter ", ignore_case = TRUE)))) %>%
ungroup() %>%
filter(chapter > 0) %>%
unite(document, book, chapter, remove = FALSE)
austen_sparse <- books %>%
unnest_tokens(word, text) %>%
anti_join(stop_words) %>%
count(document, word) %>%
cast_sparse(document, word, n)
#> Joining, by = "word"
topic_model <- stm(
austen_sparse,
K = 6,
init.type = "Spectral",
verbose = FALSE
)
summary(topic_model)
#> A topic model with 6 topics, 269 documents and a 13908 word dictionary.
#> Topic 1 Top Words:
#> Highest Prob: elizabeth, darcy, bennet, jane, miss, bingley, time
#> FREX: darcy, bennet, bingley, wickham, collins, lydia, lizzy
#> Lift: offenses, entailed, ponies, bennets, corps, deigned, phillips's
#> Score: darcy, elizabeth, bennet, bingley, jane, wickham, lydia
#> Topic 2 Top Words:
#> Highest Prob: emma, miss, harriet, weston, knightley, elton, jane
#> FREX: emma, weston, knightley, elton, woodhouse, fairfax, churchill
#> Lift: bangs, broadway, brunswick, cleverer, curtseys, delicately, drizzle
#> Score: emma, weston, knightley, elton, woodhouse, harriet, fairfax
#> Topic 3 Top Words:
#> Highest Prob: catherine, miss, tilney, time, isabella, thorpe, morland
#> FREX: tilney, thorpe, morland, allen, eleanor, northanger, fullerton
#> Lift: heroic, france, gloucestershire, lid, thorpes, tilney, tilneys
#> Score: tilney, catherine, thorpe, morland, allen, isabella, eleanor
#> Topic 4 Top Words:
#> Highest Prob: elinor, marianne, time, sister, dashwood, mother, edward
#> FREX: elinor, marianne, dashwood, jennings, willoughby, brandon, ferrars
#> Lift: tumbling, waistcoat, westward, resembled, riches, enquire, margaret's
#> Score: elinor, marianne, dashwood, jennings, willoughby, lucy, brandon
#> Topic 5 Top Words:
#> Highest Prob: anne, captain, elliot, lady, wentworth, sir, charles
#> FREX: elliot, wentworth, walter, russell, musgrove, uppercross, kellynch
#> Lift: 1760, 1784, 1785, 1787, 1789, 1791, 1800
#> Score: elliot, wentworth, anne, walter, russell, musgrove, louisa
#> Topic 6 Top Words:
#> Highest Prob: fanny, crawford, miss, sir, edmund, time, thomas
#> FREX: crawford, edmund, bertram, norris, rushworth, mansfield, julia
#> Lift: bertrams, edmund's, _daughters_, _miss, adequately, attic, baronet's
#> Score: crawford, fanny, edmund, thomas, bertram, rushworth, norris
chapters <- books %>%
group_by(document) %>%
summarize(text = str_c(text, collapse = " ")) %>%
ungroup() %>%
inner_join(books %>%
distinct(document, book))
#> Joining, by = "document"
chapters
#> # A tibble: 269 x 3
#> document text book
#> <chr> <chr> <fct>
#> 1 Emma_1 "CHAPTER I Emma Woodhouse, handsome, clever, and rich, with… Emma
#> 2 Emma_10 "CHAPTER X Though now the middle of December, there had yet… Emma
#> 3 Emma_11 "CHAPTER XI Mr. Elton must now be left to himself. It was n… Emma
#> 4 Emma_12 "CHAPTER XII Mr. Knightley was to dine with them--rather ag… Emma
#> 5 Emma_13 "CHAPTER XIII There could hardly be a happier creature in t… Emma
#> 6 Emma_14 "CHAPTER XIV Some change of countenance was necessary for e… Emma
#> 7 Emma_15 "CHAPTER XV Mr. Woodhouse was soon ready for his tea; and w… Emma
#> 8 Emma_16 "CHAPTER XVI The hair was curled, and the maid sent away, a… Emma
#> 9 Emma_17 "CHAPTER XVII Mr. and Mrs. John Knightley were not detained… Emma
#> 10 Emma_18 "CHAPTER XVIII Mr. Frank Churchill did not come. When the t… Emma
#> # … with 259 more rows
effects <- estimateEffect(1:3 ~ book, topic_model, chapters)
summary(effects)
#>
#> Call:
#> estimateEffect(formula = 1:3 ~ book, stmobj = topic_model, metadata = chapters)
#>
#>
#> Topic 1:
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 0.017611 0.024589 0.716 0.475
#> bookPride & Prejudice 0.792547 0.042627 18.593 <2e-16 ***
#> bookMansfield Park -0.002047 0.035036 -0.058 0.953
#> bookEmma 0.009849 0.035917 0.274 0.784
#> bookNorthanger Abbey 0.002644 0.039941 0.066 0.947
#> bookPersuasion 0.026252 0.045608 0.576 0.565
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#>
#> Topic 2:
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 0.019051 0.020107 0.948 0.344
#> bookPride & Prejudice 0.001541 0.026667 0.058 0.954
#> bookMansfield Park -0.004227 0.026910 -0.157 0.875
#> bookEmma 0.880291 0.033919 25.953 <2e-16 ***
#> bookNorthanger Abbey 0.003975 0.032122 0.124 0.902
#> bookPersuasion -0.006641 0.030269 -0.219 0.827
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#>
#> Topic 3:
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 0.018046 0.021297 0.847 0.398
#> bookPride & Prejudice 0.003139 0.028206 0.111 0.911
#> bookMansfield Park 0.017119 0.033818 0.506 0.613
#> bookEmma 0.001544 0.031178 0.050 0.961
#> bookNorthanger Abbey 0.870453 0.047399 18.365 <2e-16 ***
#> bookPersuasion 0.079599 0.050629 1.572 0.117
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
tidy(effects)
#> # A tibble: 18 x 6
#> topic term estimate std.error statistic p.value
#> <int> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 1 (Intercept) 0.0176 0.0243 0.722 4.71e- 1
#> 2 1 bookPride & Prejudice 0.793 0.0428 18.5 1.33e-49
#> 3 1 bookMansfield Park -0.00215 0.0351 -0.0612 9.51e- 1
#> 4 1 bookEmma 0.00973 0.0358 0.272 7.86e- 1
#> 5 1 bookNorthanger Abbey 0.00273 0.0402 0.0680 9.46e- 1
#> 6 1 bookPersuasion 0.0262 0.0455 0.575 5.66e- 1
#> 7 2 (Intercept) 0.0189 0.0202 0.938 3.49e- 1
#> 8 2 bookPride & Prejudice 0.00185 0.0267 0.0692 9.45e- 1
#> 9 2 bookMansfield Park -0.00424 0.0268 -0.158 8.74e- 1
#> 10 2 bookEmma 0.880 0.0345 25.6 3.16e-73
#> 11 2 bookNorthanger Abbey 0.00464 0.0324 0.143 8.86e- 1
#> 12 2 bookPersuasion -0.00617 0.0302 -0.205 8.38e- 1
#> 13 3 (Intercept) 0.0178 0.0214 0.833 4.06e- 1
#> 14 3 bookPride & Prejudice 0.00370 0.0285 0.130 8.97e- 1
#> 15 3 bookMansfield Park 0.0172 0.0339 0.509 6.11e- 1
#> 16 3 bookEmma 0.00166 0.0313 0.0529 9.58e- 1
#> 17 3 bookNorthanger Abbey 0.871 0.0473 18.4 3.01e-49
#> 18 3 bookPersuasion 0.0804 0.0505 1.59 1.12e- 1Created on 2020-03-07 by the reprex package (v0.3.0)
Can you try to put together a small reproducible example to demonstrate what is leading to this problem?
kathy-j-lee
ghkoo
github-actions[bot]
thanks for this package!
posting in hopes there might be a quick answer to a problem i've been running into. I've done a fair amount of searching but haven't come across an answer. (tried deleting .Rhistory and .RData, but no help)
I consistently get the following error when running
tidy(estimateEffect object). Error in object$parameters[[topic]] : subscript out of boundsapologies i don't have a fully reproducible example, but any tips/pointers would be greatly appreciated! thx!