Assignment instructions are confusing causing problems with online format class

In the last semester, it became apparent that the instructions in docs/EducationalAssignments/ are often confusing, and do not cover the scope of what is required during the assignment.

Most common issue was that the instructions don't do a good enough job of introducing the compatibility requirements that need to be built into the reference monitor so it will not break the functionality of the entire environment.

Example: The majority of students failed to build their reference monitor with support for cases when the file was reopened after closing. This caused issues during the second assignment, when we could not test the quality of the security monitor, because they had not understood the whole scope of the assignment.

Online format classes aggravate this issue significantly. I propose clarifying the requirements of the assignments for future classes. I will also volunteer myself to do so if there is no other candidate.

This is an interesting issue, which I thank you for raising. The issue you raise is an intended part of students failing to think of cases they need to handle. This is just one of a host of other issues students should have tested for and handled in their defensive code.

The point of the defensive assignment is to have students have an easy assignment where they have to handle these sorts of issues. Many students overlook this issue and in past experience nearly all overlook at least some issue. Then in the attack portion other students are generally quite effective at pointing out the problem. This makes students reason better about how their defensive code can fail as they write attacks.

However, I also think it may be possible that the specific assignment you are using has some additional lack of clarity. If so, do you want to talk about which version of the assignment you are using and in which way you think it should be clarified?

On Sat, Jan 19, 2019 at 5:14 AM Kurlee notifications@github.com wrote:

In the last semester, it became apparent that the instructions in docs/EducationalAssignments/ are often confusing, and do not cover the scope of what is required during the assignment.

Most common issue was that the instructions don't do a good enough job of introducing the compatibility requirements that need to be built into the reference monitor so it will not break the functionality of the entire environment.

Example: The majority of students failed to build their reference monitor with support for cases when the file was reopened after closing. This caused issues during the second assignment, when we could not test the quality of the security monitor, because they had not understood the whole scope of the assignment.

Online format classes aggravate this issue significantly. I propose clarifying the requirements of the assignments for future classes. I will also volunteer myself to do so if there is no other candidate.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/SeattleTestbed/docs/issues/53, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XDxjCGAk3hX71WRzISl2PYOubep_Mks5vEvAFgaJpZM4aJEWK .

The main issue we ran into was that students had a difficult time doing the assignment because they had difficulty grasping the entire scope of the project and the base functionality. The assignment we worked on this past semester was the ABopenfile series.

I would characterize the issue the majority of the students had as having a hard time linking the assignment, documentation, and guidance peppered in the class forums by TA's. I was really excited about this assignment and it was disappointing to see that I could not properly asses the majority of reference monitors during assignment two because basic functionality was not there. Upon inspecting the source code, it was apparent that the students had implemented features to protect against basic invalid write attacks.

I think that the functionality of the openfile method should be clarified in the original assignment as this was the most common issue. Sort of a "What you should be able to do" and "What you shouldnt be able to do" kind of thing

Okay, I think with that context, this makes sense. Do you want to send a PR?

On Sat, Jan 19, 2019 at 8:07 PM Kurlee notifications@github.com wrote:

The main issue we ran into was that students had a difficult time doing the assignment because they had difficulty grasping the entire scope of the project and the base functionality. The assignment we worked on this past semester was the ABopenfile series.

I would characterize the issue the majority of the students had as having a hard time linking the assignment, documentation, and guidance peppered in the class forums by TA's. I was really excited about this assignment and it was disappointing to see that I could not properly asses the majority of reference monitors during assignment two because basic functionality was not there. Upon inspecting the source code, it was apparent that the students had implemented features to protect against basic invalid write attacks.

I think that the functionality of the openfile method should be clarified in the original assignment as this was the most common issue. Sort of a "What you should be able to do" and "What you shouldnt be able to do" kind of thing

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SeattleTestbed/docs/issues/53#issuecomment-455828555, or mute the thread https://github.com/notifications/unsubscribe-auth/AA0XD7_ekhOZcuWett9VI7cTCgCVU2V6ks5vE8E1gaJpZM4aJEWK .

Yes, I will go ahead and send a PR in the next couple days

Thank you, Professor!

Hi, I wanted to piggyback off this mainly because issue is similar in that I believe there is an issue with instructions, and I didn't know a way to phrase the title without giving anything away for students because I also have a very specific issue with instructions as well. Apologies for wall of text in advance. Please let me know what you think

Here's a TLDR:

Instructions create what I believe to be a vulnerability.
Instructions unclear. Need unambiguous expectations of what resource monitor allows to occur.or clear well defined goals. Specifications and scope are too broad

1 Instructions cause what I believe to be a vulnerability.

I noticed in the ABopenfile series, in the reference monitor part one it states:

When close() is called on the file, if a file is not valid, it is discarded. If both files are valid, the older one is discarded.

Implying that you keep the a or b file. Keeping the readonly a file, can cause issues if someone tries to open it. I can email you with what issues it might cause that I think is just an inherant vulnerability. The other issue is that the assignment is presented as simulating keeping a backup in case something goes wrong you have a backup, but it is never specified that the original file be updated, either with the new valid file or the backup. Also I am not sure why you would keep either of the files after updating the original file.

2 Instructions unclear. Specifications and scope are too broad

This being said, the other issue is that it is still confusing as to what the reference monitor is suppose to do to operate correctly. While it is suppose to be open ended and is subject to interpretation, I feel like it does become confusing when there aren't examples of expected behavior. We have an example where file should not have been written to before closing, but not what files we should end up with after closing. It's hard to assess students when students have different interpretations of what is expected.

This causes stress as students are unsure what is and is not valid behavior and if something is a thing students should allow. This also causes problems when students write attack code and believe they have found a vulnerability where the defender believes that what attacker did was valid behavior and depending on interpretation both can be correct.

While some of my examples seem intuitive such as updating the original file, because why would you have the newer file be a file.b file what if you need to update again you would need to update on file.b and create a file.b.b and so forth however a student might not consider that and take it at face value (as we are accustomed to math problems where the premise is grounded in some strange reality where Bob goes to the store bought 500 watermelons, but Amy ate 30 of them how many are left) combined with not having well defined guidelines expected behavior it becomes ambiguous. I think to solve this you would have to have well defined guidelines, and describing how it applies to a real world situation will help greatly + make the assignment more intuitive. We already have the concept of the file backup for updating firmware. Perhaps we could outline what that entails aside from the fact that updates create a backup just in case and tie it into the assignment.

The language is also a little unclear, while I found "the older one is discarded" to obviously mean the valid backup as we only want the valid written to file, it left students concerned with timing. I don't feel like you would be giving anything away by stating that we would keep the valid write only b file if both files are valid.

Okay, thanks for pointing these out. Would you like to submit a pull request with proposed changes to the assignment text? I'd prefer the instructions are clearer even if they give things away, but I can be convinced otherwise.

On Sat, Sep 21, 2019 at 4:42 AM Victoria Zhong notifications@github.com wrote:

Hi, I wanted to piggyback off this mainly because issue is similar in that I believe there is an issue with instructions, and I didn't know a way to phrase the title without giving anything away for students because I also have a very specific issue with instructions as well. Apologies for wall of text in advance. Please let me know what you think

Here's a TLDR:

Instructions create what I believe to be a vulnerability.

Instructions unclear. Need unambiguous expectations of what resource monitor allows to occur.or clear well defined goals. Specifications and scope are too broad

1 Instructions cause what I believe to be a vulnerability.

I noticed in the ABopenfile series, in the reference monitor part one it states:

When close() is called on the file, if a file is not valid, it is discarded. If both files are valid, the older one is discarded.

Implying that you keep the a or b file. Keeping the readonly a file, can cause issues if someone tries to open it. I can email you with what issues it might cause that I think is just an inherant vulnerability. The other issue is that the assignment is presented as simulating keeping a backup in case something goes wrong you have a backup, but it is never specified that the original file be updated, either with the new valid file or the backup. Also I am not sure why you would keep either of the files after updating the original file.

2 Instructions unclear. Specifications and scope are too broad

This being said, the other issue is that it is still confusing as to what the reference monitor is suppose to do to operate correctly. While it is suppose to be open ended and is subject to interpretation, I feel like it does become confusing when there aren't examples of expected behavior. We have an example where file should not have been written to before closing, but not what files we should end up with after closing. It's hard to assess students when students have different interpretations of what is expected.

This causes stress as students are unsure what is and is not valid behavior and if something is a thing students should allow. This also causes problems when students write attack code and believe they have found a vulnerability where the defender believes that what attacker did was valid behavior and depending on interpretation both can be correct.

While some of my examples seem intuitive such as updating the original file, because why would you have the newer file be a file.b file what if you need to update again you would need to update on file.b and create a file.b.b and so forth however a student might not consider that and take it at face value (as we are accustomed to math problems where the premise is grounded in some strange reality where Bob goes to the store bought 500 watermelons, but Amy ate 30 of them how many are left) combined with not having well defined guidelines expected behavior it becomes ambiguous. I think to solve this you would have to have well defined guidelines, and describing how it applies to a real world situation will help greatly + make the assignment more intuitive. We already have the concept of the file backup for updating firmware. Perhaps we could outline what that entails aside from the fact that updates create a backup just in case and tie it into the assignment.

The language is also a little unclear, while I found "the older one is discarded" to obviously mean the valid backup as we only want the valid written to file, it left students concerned with timing. I don't feel like you would be giving anything away by stating that we would keep the valid write only b file if both files are valid.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SeattleTestbed/docs/issues/53?email_source=notifications&email_token=AAGROD6F2DUW5GEAHFJPOHLQKXM67A5CNFSM4GREIWFKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7INQDA#issuecomment-533780492, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGROD26KB3OZDS3N7ZDXQLQKXM67ANCNFSM4GREIWFA .

Sure, I'll start working on it!

SeattleTestbed / docs

Assignment instructions are confusing causing problems with online format class #53

1 Instructions cause what I believe to be a vulnerability.

2 Instructions unclear. Specifications and scope are too broad