fetch_survey() results in bad request error

[cndunham]

Each time I try to fetch_survey(...) I get "Error in qualtrics_response_codes(): ! Qualtrics API reported a bad request error (400)." Running survey_questions(...) with the same survey id and credentials does not produce an error. I have encountered this issue with every survey available to me.

[jmobrien]

Not sure what the issue is without more information, but it could be permissions. If you're a collaborator on these surveys with "Edit" permissions but not "View Reports" permissions (or some of the sub-permissions under those categories, I believe that could produce the results you're seeing. You can look at the survey content (Edit), but not the response content (which would need View Reports).

Does that match your scenario, or is it likely something else?

[cndunham]

Thank you for your quick reply. My permissions seem to be OK. I am able to export survey results via the web just fine. Basically, what was working just months ago will now produce this error. Nothing has changed, as far as I know -- same credentials, same survey, same function call. I wondered if something about the Qualtrics API changed, but perhaps not. Is there some other information I can provide?

[jmobrien]

Hmm. For the next step, what's your current version # of qualtRics? In between the prior time when it worked vs now (when it doesn't), has the package been updated?

[cndunham]

I'm using the very latest version of qualtRics, 3.1.7. I first tried on a new machine with a fresh install of qualtRics and my credentials. I then tested on the server on which this originally worked, and there the package has been updated since it was last run successfully, also using the latest version.

[juliasilge]

OK, next thing to check -- do you get messages about your request being proxied? Like in #229 and #211?

Another thing to try -- can you run this code to see a bit more detail about what is happening?

## put your survey ID here:
fetch_url <- qualtRics::generate_url(query = "fetchsurvey", surveyID = "SV_xxx")
raw_payload <- qualtRics:::create_raw_payload(
    label = TRUE,
    start_date = NULL,
    end_date = NULL,
    limit = NULL,
    time_zone = NULL,
    unanswer_recode = NULL,
    unanswer_recode_multi = NULL,
    include_display_order = TRUE,
    include_questions = NULL,
    breakout_sets = NULL
)

res <- qualtRics:::qualtrics_api_request("POST", url = fetch_url, body = raw_payload)
res$meta
#> $requestId
#> [1] "f52b2929-b24a-4c9d-88a6-e7c9d3ff660d"
#> 
#> $httpStatus
#> [1] "200 - OK"

^{Created on 2022-04-04 by the reprex package (v2.0.1)}

[cndunham]

I get this error on the first step: Error in qualtRics::generate_url(): ! Internal error: invalid URL generation query Run rlang::last_error() to see where the error occurred.

Backtracking the error does not produce anything more.

Oddly, I get the same error when running generate_url(query = "surveyquestions" ...), even though I can successfully run survey_questions() with the same surveyID and credentials.

[juliasilge]

I'm sorry; I may not have shared enough info. You need replace "SV_xxx" with your own real survey ID.

[cndunham]

I am using a real surveyID. Using my credentials and a valid surveyID I can get survey questions, for instance, with surveyquestions('SV***').

But when I run qualtRics::generate_url(query = 'fetch survey') with that same surveyID it errors with "! Internal error: invalid URL generation query."

I looked inside generate_url and saw that the endpoint_template step does not have a match for "fetch_survey," so I tried generate_url again with another query option that was listed, "fetchdescription". Now I can create the fetch_url object per your instructions above, but running res <- qualtRics:::qualtrics_api_request("POST", url = fetch_url, body = raw_payload) produces the same error that I received initially.

"Error in qualtrics_response_codes(): ! Qualtrics API reported a bad request error (400): • Please report this on https://github.com/ropensci/qualtRics/issues:

[cndunham]

Update: I switch POST to GET in res <- qualtRics:::qualtrics_api_request("POST", url = fetch_url, body = raw_payload) and that worked.

Now trying query = fetchdescription or query = exportresponses will work all the way through.

Looking into the result of res, I can see that this is in fact a proxy issue. It gave me the new base url to use (the other had worked fine just months ago) and now fetch_survey(...) will work!

[jmobrien]

So, you had something like [datacenter].qualtrics.com, or just www.qualtrics.com, when you needed [differentdatacenter].qualtrics.com for the url? Is that right?

If so, that would makes sense. Qualtrics used to be more permissive about the datacenter (what forms the "base URL") that you could use. If you used the wrong one it would be slower, but it would still work, generally with a message about being proxied to a different center (which from your post you seem to know about). That is no longer the case, though; if you give the wrong data center, it just errors out.

My previous experience with that change was around using "www." rather than a specific datacenter for reqeusts, which we use in our package tests for universality. That also no longer works--but, it produces a different error than the 400 you got. So, apologies that I didn't know to consider that angle when attempting to help you. And +1 to you for tracking it down yourself.

Given what you found, I'm thinking we should add some extra info to the 400 error outputs to suggest the datacenter check. I'll definitely include credit to you on that PR, which will likely help others going forward.

[cndunham]

That's right, I was using my org's custom subdomain in the base url, which works fine for some requests like survey_questions() and used to work for fetch_survey(). But now it seems requests like fetch_survey() and fetch_description() require a different subdomain, which I would not have been able to guess without seeing the full error message produced by the de-bugging steps above. I very much appreciate your and Julia's help with this issue and all that you do to make these tools available to the public.

[jmobrien]

Glad you got it worked out. As for why survey_questions() still works, I suppose they didn’t make these changes to the older legacy API endpoints, one of which that function depends on. We’ll eventually replace functions using older endpoints with updated equivalents, at which point I think the need for accurate datacenter will be a package universal.

Oh, and BTW: the legacy API dependency in survey_questions() also means it's output isn’t always perfect to how Qualtrics currently operates. It’s usually still good enough for many, if not most, purposes, but there are some more complex and/or precision-demanding scenarios where it might fall short. Obviously I don’t know your specific situation, but it's worth mentioning as something to be aware of generally.

[jmobrien]

Amending this with a note after some tests--it looks like it's not a legacy vs. modern endpoint thing, but rather something granular. Functions like fetch_description() use the newer v3 endpoints, but still work if the datacenter is wrong. fetch_survey() is specifically the one that produces these issue--so, data rather than metadata. (Might also be seen in other types of data like contacts, mailinglists, etc., but I haven't tested those).

[minahkim-dspp]

I want to echo that I had the exact same issue... I had an error using fetch_sruvey() (saying it's a bad request 400 ), fetch_describe() works for me. I also tried applying the code from @juliasilge but I faced an error in general_url stage. It says "Internal error: invalid URL generation query". I'm still a novice, but I'll also be trying to solve the problem. Any advice will be greatly appreciated!

[juliasilge]

@minahkim-dspp If you are still having this problem, can you open a new issue outlining the details? That's usually a better call than commenting on an old, closed issue.