Error with `get_interviews`

[ashwinikalantri]

While trying to get interviews from the "primary" workspace, I received this error - Error: Internal Server Error (HTTP 500). But was able to get the interviews from another workspace.

The primary workspace has 2 questionnaires deployed with more than 30,000 interviews. While the second workplace has 1 questionnaire with 3000 interviews.

[ashwinikalantri]

I get the same error whole getting interviews from a specific version of the questionnaire using get_interviews_for_questionnaire. This version has be most interviews (~23000). Can the volume of the interviews cause this?

[arthur-shaw]

Interesting. Let me check with the SuSo dev team. The 500 error code should mean that the error is with SuSo/GraphQL server rather than with get_interviews or get_interviews_for_questionnaire.

Many thanks for testing this--and finding an issue--so quickly!

[SlyNet]

Does adding a take argument with sane value fixes described issue?

[ashwinikalantri]

@SlyNet I don't understand what take argument means here. The function permits workspace, server, user and password arguments only.

[SlyNet]

Maybe package does not allow to pass such args, but there are skip and take for paging:

[ashwinikalantri]

@SlyNet using the take argument fixes the issue for smaller values of take. I tried using take: 10000, this seemed to work on some tries while not on other.

[arthur-shaw]

@SlyNet , thanks for the great idea! I could definitely implement this on the backend.

I'm guessing the underlying issue is that there is--and probably needs to be--a timeout rule for satisfying queries. Are there any good rules for how many observations a client should take at once, as a function of query complexity? For example, would it make sense to revert to the 40 records ceiling used for REST endpoints? Or should API clients target smaller numbers of obs since the GraphQL API handles more complex requests than REST? (See what the query for the get_interviews function here. It's a kitchen-sink query that aims simply to fetch all available attributes of every interview.)

[arthur-shaw]

Next actions after talking with SuSo dev team:

• Do not overwhelm the server. Rather than request all records for a query, request results by sane-sized chunks. In the back-end, page through results. On first request, take the first 100 results. On subsequent results, skip the 100*(number of prior requests) records and take the next 100. Continue until finish paging through all results.
• Add user parameters to change these chunk sizes. This is meant to handle cases where 100 records is too big of a request for the target server to handle. If the chunk size is > 100, it will be reset to 100. If the chunk size is < 100, it will be used.

A future version of SuSo will set a 100 record limit on the server.