search

github-education-resources / classroom

GitHub Classroom automates repository creation and access control, making it easy for teachers to distribute starter code and collect assignments on GitHub.
https://classroom.github.com
1.35k stars 569 forks source link

Allow teachers to configure black-box testing through CI for students submissions #913

Closed nwoodthorpe closed 7 years ago

nwoodthorpe commented 7 years ago

It would be cool if teachers could configure some sort of black-box testing through a CI provider for submissions.

E.g. Students are making a simple web-app, and when submitted, this CI system compiles it and sends a few sample requests and compares responses with expectations.

This could also be expanded to anything else that a CI system can do, which is basically anything.

We don't have support for this right now, but I wouldn't be surprised if some professors had home-rolled their own solution. @mozzadrella have you heard of anything like this being done before?

johndbritton commented 7 years ago

@nwoodthorpe Is the idea here that the tests wouldn't be visible to the students? Technically I think this can be done already with some off-the-shelf CI tools but it'd take some work on the teachers' part.

nwoodthorpe commented 7 years ago

@johndbritton By black-box, I meant more in terms of specs that hit the running application (integration vs unit specs) since those are usually independent of implementation.

This can 100% be done right now, but the teacher has to set up each assignment repo 1 by 1 with the CI system. A professor with 100 students and 10 classrooms would have to configure CI 1000 times.

ryansobol commented 7 years ago

We include an automated test suite in all our assignment repositories. Running the suite allows both students and teachers to know how close a student is to completing the assignment.

However, we'd love to include a TravisCI badge at the top of the README.md file and a webhook for each student assignment. That way, our teachers wouldn't have to manually clone and run the test suite for each assignment when grading.

nwoodthorpe commented 7 years ago

Thanks for the feedback!

This is 100% a feature we're looking to implement at some point. It's on our roadmap, so once we find a bit of time, we'll get it done. We'll give you a shout when there's movement on this :)

obcode commented 7 years ago

@ryansobol if you have access to a Jenkins server, you can use Jenkinsfile with the Jenkins-GitHub-Plugin to automate the process.

stale[bot] commented 7 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.