quanteda / spacyr

R wrapper to spaCy NLP
http://spacyr.quanteda.io
250 stars 38 forks source link

nounphrase_consolidate() gives unexpected results #207

Open rjake opened 3 years ago

rjake commented 3 years ago

spacyr has been great to learn! Thanks for making this available. I might be doing something wrong but sometimes nounphrase_consolidate() gives me unexpected results. It seems to be related to how much data spacyr has to process. In the results below, sometimes processing more rows gives the expected results (orange) and sometimes processing fewer rows gives the expected results (green). I hope you can shed some light on this for me.

library(tidyverse)
library(spacyr)

odd_words <- "right|most|paw|clear"

df <- 
  tibble(
    doc_id = seq_along(sentences),
    text = tolower(sentences)
  )

all_rows <-
  df |> 
  spacy_parse(nounphrase = TRUE) |> 
  nounphrase_consolidate()

filtered_rows <-
  df |>
  filter(str_detect(text, odd_words)) |> # <--- this line is different 
  spacy_parse(nounphrase = TRUE) |> 
  nounphrase_consolidate()

filtered_rows |> 
  select(doc_id:token) |> 
  left_join(
    select(all_rows,doc_id, sentence_id, token_id, token2 = token)
  ) |> 
  filter(token != token2)

image