search

ucsb-cs / submit

Repository for tracking issues, suggestions with the submit.cs system
https://ucsb-cs.github.io/submit/
BSD 2-Clause "Simplified" License
9 stars 5 forks source link

Submissions that echo input #2

Closed kpavery closed 10 years ago

kpavery commented 10 years ago

I'm not sure if this has already come up, but it might be nice to have an option which supports disallowing echoing all the input to the output. By doing this, students can see the input of hidden test cases, given the output.

bboe commented 10 years ago

@kpavery I would love a way to absolutely prevent this, however, any such method that I can think of is fairly easily circumvented. However, I think the system limit on the student's view of (1) maximum line length and (2) only showing three diff lines is sufficient to prevent this behavior.

Students who make subsequent submissions to extract longer chains of input should be penalized for trying to circumvent the submission system.

From an project designer's point of view, you can also try to prevent this behavior by having all expected output go to a named file which is unknown to the student's code.

If you have any other suggestions, I'd love to hear them.

kpavery commented 10 years ago

@bboe I agree that it would be difficult to make something that is circumvention-proof. An idea that I had was to add an option that does not show the output if certain "disallowed" trigger strings are found. Then we can specify some part of the input that should never appear in the output, and maybe prevent some people who echo it.

kpavery commented 10 years ago

@bboe Of course this requires project designers (us) to come up with them and design the projects accordingly, but it seemed pretty simple / reasonable for the benefit added in my head.

bboe commented 10 years ago

@kpavery I like that you're thinking about ideas on how to solve it. Professor Franklin and I thought about it quite a bit as well. We've since decided that it's not worth the effort to handle a few simple cases since that will likely prompt suggestions for gradually handling subsequently more complicated cases. Our suggestion is that if it's that important that the test cases don't leak, then don't include them in the student's view. This can be accomplished by creating a "hidden" testable. The students view of their score will not show or include any results from the hidden testable.

Please let me know if you would like any other suggestions on how to prevent test leakage. In my experience with the system, I have not really noticed any significant attempt to leak the test cases. Please privately share with my any information you have to suggest otherwise. Thanks!