ropensci / qualtRics

Download ⬇️ Qualtrics survey data directly into R!
https://docs.ropensci.org/qualtRics
Other
215 stars 70 forks source link

Data imported to R is not current #36

Closed cmbohn closed 7 years ago

cmbohn commented 7 years ago

When importing survey data into R, is there a delay between what is current in the Qualtrics system and what is being downloaded?

Is it possible that the data is having problems importing? I enter the following into R: loyperc <- getSurvey("SV_5nIBD5iRalXmdNz", useLabels=FALSE, useLocalTime=TRUE, dateWarning=FALSE, convertStandardColumns=TRUE, force_request=TRUE) I get the following warning: Warning message: In scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : invalid input found on input connection 'C:/Users/cb37543/AppData/Local/Temp/RtmpeEgLjr/Loyalty perceptions (Jamie).csv' I can only import 88 observations, when there is actually much more than this.

JasperHG90 commented 7 years ago

Thanks for your comment.

When importing survey data into R, is there a delay between what is current in the Qualtrics system and what is being downloaded?

I'm not aware of such a delay. Essentially, you're just making a call to the Qualtrics API in which you request a csv file of the response exports.

Is it possible that the data is having problems importing?

Yes, that is possible. (see e.g. #13). In this case, I'm guessing there are special characters in your csv file. I googled that error and found the following:

As you can see here, the readSurvey() function uses UTF-8 for the 'fileEncoding' argument.

Here's what I suggest you try:

  1. Re-run your code until you see the error
  2. Execute the following:
# Get temp dir
td <- tempdir()
# List files and show
list.files(td)
# Copy the name of your survey
file_name <- paste(td, "/", "<YOUR-SURVEY>.csv")
# Read it into R
surv <- read.csv(file=file_name,
                        header = FALSE,
                        sep = ',',
                        stringsAsFactors=FALSE,
                        fileEncoding = "latin1",
                        skip = 3)
  1. Let me know if that worked. If it does, I'll add 'fileEncoding' as an optional argument.

Best,

J.

cmbohn commented 7 years ago

The code provided isn’t connecting. I have tried a number of variations for inputting the file name: file_name <- paste(td, "/", ".csv")

All attempts have resulted in the system not being able to find the file or directory. The temp directory did not list a csv file. So I tried entering the files that exist in the temp file. Below is just one example of the same results I received.

Error in file(file, "rt", encoding = fileEncoding) : cannot open the connection In addition: Warning message: In file(file, "rt", encoding = fileEncoding) : cannot open file 'C:\Users\cb37543\AppData\Local\Temp\RtmpKiJALO / SV_5nIBD5iRalXmdNz.rds': No such file or directory

Chad

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 1:01 AM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Thanks for your comment.

When importing survey data into R, is there a delay between what is current in the Qualtrics system and what is being downloaded?

I'm not aware of such a delay. Essentially, you're just making a call to the Qualtrics API in which you request a csv file of the response exports.

Is it possible that the data is having problems importing?

Yes, that is possible. (see e.g. #13https://github.com/JasperHG90/qualtRics/issues/13). In this case, I'm guessing there are special characters in your csv file. I googled that error and found the following:

As you can see herehttps://github.com/JasperHG90/qualtRics/blob/master/R/readSurvey.R#L46-L51, the readSurvey() function uses UTF-8 for the 'fileEncoding' argument.

Here's what I suggest you try:

  1. Re-run your code until you see the error
  2. Execute the following:

Get temp dir

td <- tempdir()

List files and show

list.files(td)

Copy the name of your survey

file_name <- paste(td, "/", ".csv")

Read it into R

surv <- read.csv(file=file_name,

                    header = FALSE,

                    sep = ',',

                    stringsAsFactors=FALSE,

                    fileEncoding = "latin1",

                    skip = 3)
  1. Let me know if that worked. If it does, I'll add 'fileEncoding' as an optional argument.

Best,

J.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311574567, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVYpfPkJIy5Z3bwJHmD_ONDNWivlaks5sIfpDgaJpZM4OG_o9.

JasperHG90 commented 7 years ago

Hi chad,

I didn't realize you're on a windows machine. Try:

file_name <- paste(td, "\", ".csv")

On 28 Jun 2017, 18:15 +0200, cmbohn notifications@github.com, wrote:

The code provided isn’t connecting. I have tried a number of variations for inputting the file name: file_name <- paste(td, "/", ".csv")

All attempts have resulted in the system not being able to find the file or directory. The temp directory did not list a csv file. So I tried entering the files that exist in the temp file. Below is just one example of the same results I received.

Error in file(file, "rt", encoding = fileEncoding) : cannot open the connection In addition: Warning message: In file(file, "rt", encoding = fileEncoding) : cannot open file 'C:\Users\cb37543\AppData\Local\Temp\RtmpKiJALO / SV_5nIBD5iRalXmdNz.rds': No such file or directory

Chad

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 1:01 AM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Thanks for your comment.

When importing survey data into R, is there a delay between what is current in the Qualtrics system and what is being downloaded?

I'm not aware of such a delay. Essentially, you're just making a call to the Qualtrics API in which you request a csv file of the response exports.

Is it possible that the data is having problems importing?

Yes, that is possible. (see e.g. #13https://github.com/JasperHG90/qualtRics/issues/13). In this case, I'm guessing there are special characters in your csv file. I googled that error and found the following:

As you can see herehttps://github.com/JasperHG90/qualtRics/blob/master/R/readSurvey.R#L46-L51, the readSurvey() function uses UTF-8 for the 'fileEncoding' argument.

Here's what I suggest you try:

  1. Re-run your code until you see the error
  2. Execute the following:

Get temp dir

td <- tempdir()

List files and show

list.files(td)

Copy the name of your survey

file_name <- paste(td, "/", ".csv")

Read it into R

surv <- read.csv(file=file_name,

header = FALSE,

sep = ',',

stringsAsFactors=FALSE,

fileEncoding = "latin1",

skip = 3)

  1. Let me know if that worked. If it does, I'll add 'fileEncoding' as an optional argument.

Best,

J.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311574567, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVYpfPkJIy5Z3bwJHmD_ONDNWivlaks5sIfpDgaJpZM4OG_o9. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

cmbohn commented 7 years ago

I thought of that too and tried it before this message. ☺ Because it is a windows machine, it actually needs 2 slashes. file_name <- past(td, “\”, “.csv”)

And unfortunately, I still got the same error and warning.

It isn’t a deal breaker or a problem for me to download the CSV from Qualtrics. That isn’t holding me up. But I would like to continue to help figure out what is going on.

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 11:27 AM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Hi chad,

I didn't realize you're on a windows machine. Try:

file_name <- paste(td, "\", ".csv")

On 28 Jun 2017, 18:15 +0200, cmbohn notifications@github.com<mailto:notifications@github.com>, wrote:

The code provided isn’t connecting. I have tried a number of variations for inputting the file name: file_name <- paste(td, "/", ".csv")

All attempts have resulted in the system not being able to find the file or directory. The temp directory did not list a csv file. So I tried entering the files that exist in the temp file. Below is just one example of the same results I received.

Error in file(file, "rt", encoding = fileEncoding) : cannot open the connection In addition: Warning message: In file(file, "rt", encoding = fileEncoding) : cannot open file 'C:\Users\cb37543\AppData\Local\Temp\RtmpKiJALO / SV_5nIBD5iRalXmdNz.rds': No such file or directory

Chad

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 1:01 AM To: JasperHG90/qualtRics qualtRics@noreply.github.com<mailto:qualtRics@noreply.github.com> Cc: Bohn, Chad Chad.Bohn@jetblue.com<mailto:Chad.Bohn@jetblue.com>; Author author@noreply.github.com<mailto:author@noreply.github.com> Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Thanks for your comment.

When importing survey data into R, is there a delay between what is current in the Qualtrics system and what is being downloaded?

I'm not aware of such a delay. Essentially, you're just making a call to the Qualtrics API in which you request a csv file of the response exports.

Is it possible that the data is having problems importing?

Yes, that is possible. (see e.g. #13https://github.com/JasperHG90/qualtRics/issues/13). In this case, I'm guessing there are special characters in your csv file. I googled that error and found the following:

As you can see herehttps://github.com/JasperHG90/qualtRics/blob/master/R/readSurvey.R#L46-L51, the readSurvey() function uses UTF-8 for the 'fileEncoding' argument.

Here's what I suggest you try:

  1. Re-run your code until you see the error
  2. Execute the following:

Get temp dir

td <- tempdir()

List files and show

list.files(td)

Copy the name of your survey

file_name <- paste(td, "/", ".csv")

Read it into R

surv <- read.csv(file=file_name,

header = FALSE,

sep = ',',

stringsAsFactors=FALSE,

fileEncoding = "latin1",

skip = 3)

  1. Let me know if that worked. If it does, I'll add 'fileEncoding' as an optional argument.

Best,

J.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311574567, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVYpfPkJIy5Z3bwJHmD_ONDNWivlaks5sIfpDgaJpZM4OG_o9. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311730469, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVWKlatVVExV5NGR7HoD_QdJ7JoLRks5sIoz_gaJpZM4OG_o9.

JasperHG90 commented 7 years ago

Thanks for helping out :-). It's useful for me to know such issues so I can solve them.

You are right: without specifying a directory in which to save the export the csv file is deleted from the temporary folder.

Can you follow the steps in the image below and tell me if this works for you?

rconsole

Best,

J.

JasperHG90 commented 7 years ago

As an aside: is the data you download from Qualtrics manually and load into R complete?

cmbohn commented 7 years ago

Yes. The data appears to be complete (meaning that it has the number of rows in the CSV matches what is in Qualtrics). However, the way the CSV export is formatted it appears to be different from an API. As a result I have to do some recoding to the CSV dataset or accommodate the CSV format in my R code.

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 1:35 PM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

As an aside: is the data you download from Qualtrics manually and load into R complete?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311764468, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVdUpVAMyy_0Z6N0BjIsjsB-ofxhLks5sIqr-gaJpZM4OG_o9.

cmbohn commented 7 years ago

I ran the code you gave me below. It brought in more data but not the complete data. The complete data has around 1085 observations. This time around it only brought in 273 observations – 88 when I first contacted you. So there is that… I am still getting a warning too.

Warning messages: 1: In scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : invalid input found on input connection 'C:\Users\cb37543\AppData\Local\Temp\RtmpKiJALO\Loyalty perceptions (Jamie).csv' 2: In scan(file = file, what = what, sep = sep, quote = quote, dec = dec, : EOF within quoted string

From: Jasper Ginn [mailto:notifications@github.com] Sent: Wednesday, June 28, 2017 1:29 PM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Thanks for helping out :-). It's useful for me to know such issues so I can solve them.

You are right: without specifying a directory in which to save the export the csv file is deleted from the temporary folder.

Can you follow the steps in the image below and tell me if this works for you?

[rconsole]https://user-images.githubusercontent.com/8079803/27656317-d21a9f18-5c48-11e7-81a7-484c89d9a526.png

Best,

J.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311763069, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVZs7pSgx1ZzwrOziiTI2YBklZRUrks5sIqmOgaJpZM4OG_o9.

JasperHG90 commented 7 years ago

Hm.

Last thing I suggest you try:

After downloading the csv in a temporary folder, copy-paste the temporary folder in your file manager and open de csv file in excel.

Two things I'd like to know about your survey:

  1. Does your survey contain text that is not in English with special characters?
  2. Does your survey contain questions that ask the participant to input lines of text? (e.g. a sentence or two/three)

Thanks for your help!

cmbohn commented 7 years ago
  1. I don’t believe the open text responses contain any non-English characters. I haven’t found any.

  2. The open text responses could contain a couple sentences in some cases. We limited the character count, but I do see some comments with about 2 sentences.

I notice when I open the temp file, there isn’t a copy of the csv in the temp folder. Only rds files. I tried taking a copy of the csv file that I downloaded directly from Qualtrics and put it in the temp folder. It didn’t recognize it and gave me the same warning message that I sent previously.

From: Jasper Ginn [mailto:notifications@github.com] Sent: Thursday, June 29, 2017 12:47 AM To: JasperHG90/qualtRics qualtRics@noreply.github.com Cc: Bohn, Chad Chad.Bohn@jetblue.com; Author author@noreply.github.com Subject: Re: [JasperHG90/qualtRics] Data imported to R is not current (#36)

Hm.

Last thing I suggest you try:

After downloading the csv in a temporary folder, copy-paste the temporary folder in your file manager and open de csv file in excel.

Two things I'd like to know about your survey:

  1. Does your survey contain text that is not in English with special characters?
  2. Does your survey contain questions that ask the participant to input lines of text? (e.g. a sentence or two/three)

Thanks for your help!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/JasperHG90/qualtRics/issues/36#issuecomment-311877649, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcXCVVJC4oiU4ADd9eMSHuhjLTFBithEks5sI0h7gaJpZM4OG_o9.

JasperHG90 commented 7 years ago

Ok. Thanks for trying out these suggestions. I'll close this issue for now because I can't reproduce it on my end, but I'll get back to this at a later point in time.

JasperHG90 commented 7 years ago

Issue should be solved. Please look at #38