fetch_survey produces "Error: Qualtrics API raised a bad request (400) error"

[kerem-a]

Hello! I'm currently facing an issue while attempting to import surveys into R using qualtRics. The function all_surveys() operates smoothly, indicating that my credentials are likely in order. However, when I use fetch_survey, I encounter a "Qualtrics API raised a bad request (400) error."

I hold a student account at a university, and although I successfully obtained the API key, I'm wondering if there could still be a permission-related problem. Any insights or suggestions on resolving this issue would be greatly appreciated.

Thank you!

qualtrics_api_credentials(api_key = "XXXXX", 
                             base_url = "XXX.qualtrics.com",
                             install = TRUE,
                           overwrite = TRUE

all_surveys()

Get me a nice list as it should: id name ownerId lastModified creationDate isActive 1 XXX FFQ_adults_2… UR_4Vh… 2023-04-04T… 2023-03-09T… TRUE
2 XXX FFQ_iris01… UR_d6W… 2023-10-01T… 2023-01-04T… FALSE
3 XXX FFQ - medica… UR_d6W… 2023-11-23T… 2023-06-30T… TRUE
....

fetch_survey(surveyID="XXX")

Throw :

Error in `qualtrics_response_codes()`:
Qualtrics API reported a bad request error (400):
Please report this on https://github.com/ropensci/qualtRics/issues

Backtrace:
    ▆
 1. └─qualtRics::fetch_survey(surveyID = "SV_8CUhVfr6of7X2ei", force_request = TRUE)
 2.   └─qualtRics:::export_responses_request(...)
 3.     └─qualtRics:::export_responses_init(surveyID = surveyID, body = body)
 4.       └─qualtRics:::qualtrics_api_request(...)
 5.         └─qualtRics:::qualtrics_response_codes(res)
 6.           └─rlang::abort(error_message)

[juliasilge]

Hello @kerem-a! There are two common issues that cause folks to run into this error.

• The first is not having your base URL set correctly. Notice that when you register your credentials:

The base URL you pass to the qualtRics package should either look like yourdatacenterid.qualtrics.com or like yourorganizationid.yourdatacenterid.qualtrics.com

• The second is Qualtrics proxying your request. Do you see any messages about this? For example, what do you see when you run this at the command line (not in R)?

curl --request GET \
--url https://XXX.qualtrics.com/API/v3/whoami \
--header 'x-api-token: YOURLONGAPITOKENHERE'

You would need to use your URL and your API token. Do you see anything in the results like:

Notice: Request proxied. For faster response times, use this host instead:

• Another thing to try -- can you run this code to see a bit more detail about what is happening?

## put your survey ID here:
fetch_url <- qualtRics::generate_url(query = "fetchsurvey", surveyID = "SV_xxx")
raw_payload <- qualtRics:::create_raw_payload(
    label = TRUE,
    start_date = NULL,
    end_date = NULL,
    limit = NULL,
    time_zone = NULL,
    unanswer_recode = NULL,
    unanswer_recode_multi = NULL,
    include_display_order = TRUE,
    include_questions = NULL,
    breakout_sets = NULL
)

res <- qualtRics:::qualtrics_api_request("POST", url = fetch_url, body = raw_payload)
res$meta
#> $requestId
#> [1] "f52b2929-b24a-4c9d-88a6-e7c9d3ff660d"
#> 
#> $httpStatus
#> [1] "200 - OK"

^{Created on 2022-04-04 by the reprex package (v2.0.1)}

[kerem-a]

Thank you for your fast reply, After taking a more careful look, I realized that I used the "Organization ID" instead of the Datacenter ID in the base_url value. Replacing this value fixed the issue with fetch_survey.

I'm not sure if it makes sense, but this code: fetch_url <- qualtRics::generate_url(query = "fetchsurvey", surveyID = "SV_XXX") still throws an error even after replacing the base_url with the correct one.

<error/rlang_error>
Error in `qualtRics::generate_url()`:
Internal error: invalid URL generation query

Backtrace:
    x
 1. \-qualtRics::generate_url(query = "fetchsurvey", surveyID = "SV_9BUaLjV9ztl7Tjo")
 2.   \-rlang::abort("Internal error: invalid URL generation query")

But I guess you can close the issue anyway. On a side note, perhaps it's a good idea to add something to this error message, like: " 'Did you set the URL correctly? Remove the 'https://' and use the Organization ID (not the Datacenter ID).'"

Thank you very much for this handy package!

[jmobrien]

@juliasilge this seems like something that shows up pretty often as a point of user confusion--maybe we could find a more user-friendly approach to configuration?

The screenshot above--a dropdown from one of the API testing modules at api.qualtrics.com--is, I think, all the datacenters that Qualtrics has? And, if you click through them, just below this you see each's associated string prefix that would go in xxxx.qualtrics.com. What if we deprecated the base_url arg in favor of datacenter_id in the credentials function, which then checks whether it receives one of the strings found here?

While I'm at it, something like that could pave the way for an extra string option like "mock", which would then configure to use the mocking server. Thinking that probably makes sense as a direction to go for tests, vs. saved VCR's of normal accounts like we currently do--but, implementing that under our current URL generation scheme would require some super kluge-y changes.

(Throw it on the pile, I know; I've been too buried to do development here for a while now. I haven't forgotten, though!)

[juliasilge]

Oh gosh, being able to test against their mock server would be huge. I think this could be a really good way to go; this is a fantastic idea @jmobrien. 👍

[jmobrien]

👍

Realistically, doing it right entails a bit of a re-build of URL generation, perhaps scrapping much of the generate_url() system I set up. The idea was to give us better modularity & extensibility (and maybe it did a bit), but knowing what I do now I would do it differently today. I think it's already gotten a bit unwieldy, while also mostly locking us out of some important flexibility, especially w/r/t mocking.