philchalmers / mirtCAT

Computerized Adaptive Testing with Multidimensional Item Response Theory
http://philchalmers.github.io/mirtCAT/
89 stars 28 forks source link

Multiple sessions result in 'please provide valid response' #22

Closed kilzone closed 5 years ago

kilzone commented 5 years ago

Dear Phil,

The library you created is awesome, but we're having some trouble using it; we're probably doing something wrong and require some help. We've developed a test using your library, which is working great, tested the responses and so far everything seems exactly like we want - however we've some trouble scaling.

When we host the application on a Shiny R Server we're running into the fact that the package seems to merge sessions (responses of one user, affects the others). Are we doing something wrong?!

Stephen

philchalmers commented 5 years ago

Thanks for the report. This mainly has to do with the setup of the shiny server, which apparently deals with open sessions in a particularly odd way (all sessions seem to be run on the same port, or something to that extent, and therefore are all interacting with the same active session). I've noticed this as well in some of my recent server-based tests.

The fix that I can do on my end, I think, is to limit only one session to be run at a time by checking to see if a temporary file exists on the computer, which would only be created at run-time. Then, any sign-ins from mirtCAT could be checked and rejected if their is currently a busy session. I've hesitated to do this though because it feels like the solution should be done on the R server side, but I'm unsure exactly how to go about this. Perhaps asking this type of question on stack overflow would be helpful, pointing to this issue?

philchalmers commented 5 years ago

I found a few relevant links to this problem, listed below:

Some individuals claim that Shiny Proxy is the way to go to solve this problem, but I get the feeling that the basic R shiny server should be handling mirtCAT sessions better (e.g., multiple users should slow down the processes, not somehow interact between sessions). This likely has something to do with the socket-based implementations, but I'm certainly no expert in this area to offer an official answer. I'll leave this issue open for now until a good explanation of the problem is found.

philchalmers commented 5 years ago

@kilzone any progress on your end? From what I could tell from the above links, the most promising solution was to host your server app using ShinyProxy