Problem adding newspaper articles using LexisNexisTools

[JKerms89]

Dear DNA members,

I am experimenting with the DNA interface with the help of the DNA manual. However, I am experiencing some problems with trying to add articles using the LexisNexisTools package. I am following the suggested workflow (see pg. 110 of the DNA manual)

library("LexisNexisTools")

This places a sample TXT file in your working directory in

lnt_sample()

Look for TXT files

my_files <- list.files(pattern = "TXT", ignore.case = TRUE)

If this contains other files not from LexisNexis you can subset the vector

my_files <- my_files[grepl("^sample.TXT$", my_files)]

Now the files can be converted into an LNToutput object

LNToutput <- lnt_read(my_files)

And this object can be converted to work in rDNA

docs <- lnt_convert(LNToutput, to = "rDNA")

But I then get the following error message:

docs_orig <- dna_getDocuments(conn) RcallMethod: attempt to call a method of a NULL object.

I would be grateful if someone could help me out.

Many thanks,

Jan

[JBGruber]

The line dna_getDocuments(conn) only works when there are documents in your database already. You don't show how you created conn but based on the error I assume that you didn't run the commands in the manual in sequence. In this case, this would be a complete working sequence of commands:

library("LexisNexisTools")
# This places a sample TXT file in your working directory in
lnt_sample()
# Look for TXT files
my_files <- list.files(pattern = "TXT", ignore.case = TRUE)
LNToutput <- lnt_read(my_files)
# And this object can be converted to work in rDNA
docs <- lnt_convert(LNToutput, to = "rDNA")

library(rDNA)
dna_init()
conn <- dna_connection(dna_sample())

docs_orig <- dna_getDocuments(conn)
# We create new unique IDs by IDs with the highest ID from the original data
docs$id <- docs$id + max(docs_orig$id)
# We use coder 1 to import the new documents
unique(docs_orig$coder)
## [1] 2
docs$coder <- 2
# Now we combine the original with the new documents and push it to DNA
docs <- rbind(docs_orig, docs)
dna_setDocuments(conn, documents = docs, simulate = FALSE)
docs <- dna_getDocuments(conn)

nrow(docs_orig)
nrow(docs)

Also note that LexisNexis changed their interface last year and the new recommended download format is DOCX (Word files). If you download files in this format, you need to replace pattern = "TXT" with pattern = "DOCX". The rest of the code should be fine.

[JKerms89]

Many thanks JBGruber. Your quick response is greatly appreciated. The lnt.sample seems to work now although the final step (namely to open the lnt.sample on DNA has eluded me!). The other thing that confuses me is in the code above, there is both the dna_sample and the lnt_sample but are they not too separate datasets?

I'm also having problems importing my own LexisNexis DOCX files onto DNA. I've managed to import them onto R but I'm unable to export these files onto the DNA. I've amended the pattern = "TXT" with pattern = "DOCX" as you advised. Are you kindly able to give a clear step-by-step guide on how to import LexisNexis articles onto R and then open them on the DNA interface? Thank's so much for your help.

All the best,

Jan

[JBGruber]

You first need a .dna database. Look to the manual for advice on how to create one (Chapter 4). I simply used the sample files from LexisNexisTools and rDNA for demonstration purposes. Then you connect to this database and fill it with a data.frame which you create with lnt_convert:

# connect to own database named "my_data.dna" as an example
library(rDNA)
dna_init()
conn <- dna_connection("my_data.dna")

# read in and convert docx files
library("LexisNexisTools")
# Look for DOCX files in current working directory
my_files <- list.files(pattern = "DOCX", ignore.case = TRUE)
# read files to R
LNToutput <- lnt_read(my_files)
# And this object can be converted to work in rDNA
docs <- lnt_convert(LNToutput, to = "rDNA")

# add docs to dna db
dna_setDocuments(conn, documents = docs, simulate = FALSE)

Hope that helps.

[JKerms89]

It's working now!! Thank you so much and complements on the wonderful LexisNexis R Package. It should certainly make my life a lot easier when it comes to coding my dataset. Out of curiosity, how well does a normal spec computer (8gb RAM) handle a newspaper dataset of between 2-3000 articles on DNA? Do you think my computer should handle it fine or should I invest in more powerful hardware?

Many thanks again for your prompt assistance.

Jan

All the best,

Jan

[leifeld]

I can't speak to the LexisNexis side of things, but I have worked with much larger datasets (~30,000 documents) in DNA without problems. The SQL database scales relatively well. I would give it a try and only buy new hardware when/if you hit a wall.

[JKerms89]

Many thanks for your assistance. My remaining hurdle is how to download LexisNexis articles in English - it seems as though the LexisNexis translation tool only works online but when I proceed to download the article it returns to the original language. Are you able to suggest some alternative solutions? I understand that DeepL is a reliable translation tool but I would of course prefer the DOCX files all translated before they are imported on the DNA platform.

Many thanks,

Jan

[leifeld]

If @JBGruber (package author of LexisNexisTools) does not have any suggestions, I'll close this issue. I have never done translations with LexisNexis.