User Testing Feedback WT21/22

[Clashsoft]

Please collect comments, feedback, opinions here.

Template:

# 2022-XX-XX, XX Assignment X

## General

## Feature Requests

## Improvement Suggestions

## Bugs

[Clashsoft]

2021-11-15, PM Assignment 2

General

• Autofill for Remark is very useful.
• Sometimes forgot to click "-1 points" and gave 0 instead.
• "Dark matter" points that did not factor into the totals made tasks harder to specify:
  • Could be solved by setting parent tasks to 0 points, but:
  • Parent tasks with 0 points were displayed in the rendered issue.
  • Could be solved with "separator tasks" like - ===== General ===== (-5P).
• I set up a deadline in the past that did not trigger automatic import.

Improvement Suggestions

• [x] Editing an evaluation should keep the old remark - had to copy and paste.
• [ ] Simple deductions without remark involve a lot of clicking ("Grade" → "-1" → "Submit"). Could be reduced to a simple quick button next to "Grade".

Bugs

• [x] Editing tasks as markdown failed to parse ID correctly, lead to "deletion" of evaluations (could not be connected to the task any more, but were still visible in dev tools)

[Morphclue]

2021-11-15, PM Assignment 2

General

• Feels faster than copy-pasting by hand (and probably is)
• Grading works pretty good so far, didn't have any problems or trouble

  Improvement Suggestions

• [x] Assignee field could be a dropdown or a search-field
• [x] ?/Max-points should be highlighted if Post feedback to GitHub is clicked
  • [x] The question mark could also be filled out by the correct number (if you send the data, you calculate everything anyways so that shouldn't be that hard probably)
• [x] Add GitHub-name of the student in the solutions & tasks tab. I've had to click student info twice during grading and this felt kinda unnessary. Maybe next to the Deadline or something like that?
• [x] My workflow is opening up every grading and repo in a new tab. It could be useful adding a new checkbox Always add in new tab. I'm unsure about this one, because i am not sure how i will adjust my workflow with fulib.org in the following assignments.

[Clashsoft]

2021-11-08, PM Assignment 1

Feature Requests

• [x] Should be able to delete Evaluation. (also suggested by @Kuenssi)
• [x] Keyboard shortcuts for grading.
• [x] Buttons for setting min/max points.

Improvement Suggestions

• [x] Task list should be updated after submitting evaluation. (also suggested by @Kuenssi)
• [x] Upper/lower limit in points field (also suggested by @Kuenssi)

Bugs

• [x] Importing did not work (for everyone whose GitHub username was already in the database).
• [x] Cannot edit assignments using token - only owner (also suggested by @Kuenssi)
  • [x] Editing assignments when not logged in only warned about this fact when viewing the preview. This led to frustration when much time was invested in editing the assignment, only to find out it had to be discarded because I was not logged in.

[Kuenssi]

2021-11-15, PM Assignment 2

Feature Requests

• [x] Add a modal before posting an issue where you can see the most basic information, like calculated points

Improvement Suggestions

• [x] It could be usefull to be able to edit the assignment when i got the assignment via token -> Or have the possability to have a maintainer like role that can be given by the creator of the assignment

Bugs

• [x] Toasts in the overview are "behind" the fixed header
• [x] After the tasks were changed, the previously entered evaluations were not displayed anymore

[JulianWerner98]

2021-11-16, PM Assignment 2

General

Nice Tool! It feels much faster and shows me a better overview

Feature Requests

• [x] One empty Field at every Task/Subtask for mistakes who not in the Suggestions
• [x] Add a modal before posting an issue where you can see the most basic information, like calculated points

Improvement Suggestions

• [x] Make it possible to edit the Remark Text after sending

Bugs

• [x] Shortcut "Strg" + "-" is also a zoom shortcut for the browser

[Kuenssi]

2021-11-29, PM Assignment 03

General

• The Modal that opens before submitting an issue is perfect for my workflow. The ability to check the draft is great.
• VSCode Extension works fine. Selecting and completing the submissions work very intuativly.

Bug

• [x] Enter yourself as an assignee -> Search for your name -> Table doesn't show the previously added assignee entries (After a site reload it works again

[Clashsoft]

2021-11-29, PM Assignment 3

General

• Delete Snippet seems wonky. It never showed any Code Search notifications and also did not update the diagnostics. As a workaround, typing anything updates diagnostics.
• Sometimes, Alt-Enter on the comment for creating a snippet shows "No Actions available" if you are really fast. This is probably a problem with VSCode.

Feature Requests

• [x] Allow viewing the origin evaluation for a derived evaluation created by Code Search.

Improvement Suggestions

• [x] After selecting a task to create a snippet, I sometimes forgot which one I selected. Accidentally submitting an evaluation for the wrong task can mess up Code Search badly. The description should be visible either within the comment, or before sending.

Bugs

• [x] Invalid settings (e.g. invalid Mongo ID in Assignment or Solution ID) show cryptic error messages.
• [x] Updating an evaluation replaces the author "Code Search" in all other derived evaluations with the TA name.
• [x] Code Search does not copy the comments to derived snippets.

[Morphclue]

2021-11-29, PM Assignment 3

General

• VSCode Extension feels pretty good to use
• Code Search only works with static code examples and even then students seem to find many different ways of implementing functions

Feature Requests

• [x] IntelliJ Extension - fujaba/fulibFeedback#3

Improvement Suggestions

• [x] Update Code Search only after grading is done

[JulianWerner98]

2021-11-29, PM Assignment 3

General

• I used the function as a last corrector and could thus benefit a lot from CodeSearch. A quick look at the correction and you were done. It certainly saved me 30-90min of time.
• VSC Extension needs a little moment until you can use it flawlessly, but it is just a new tool. After that it helps great.

  Feature Requests

Improvement Suggestions

• [x] Highlighting in VSC for better readability

[Kuenssi]

2021-12-04, PM Assignment 3 (Second round)

General

• I like the new Extension and the way vscode and fulib.org harmonize while grading
• Code Search already worked after a few solutions saving me some time
• I think the new workflow is much better as long as you have more than one screen your working on

Bugs

• [x] The solution page has to be reloaded again to show the already graded points (Only happens sometimes)
• [x] Sometimes while writing a comment for a code snippet another code snippet (one or more) get created and you have to delete those and check the first comment again

[nataschu]

2021-12-06, PM Assignment 4

Improvement Suggestions

• [x] brain afk clicking Save & Close on "We value your privacy!" made me think sth broke
• [x] "clone in VSCode" not working when using the open source build of vscode

Bugs

• [x] After Grading, the blue "Grade" link only sometimes updates automatically. Most times needs a manual f5.
• [x] fulib feedback extension starts after opening a .java-file. I opened a .gradle-file and got confused, because the selected text wasn't shown when grading in fulib.org

Feature Requests

• [x] Maybe use a resizable text area for the "Remark" text field, to be able to see everything I wrote. Sometimes I write several sentences.

[Morphclue]

2021-12-06, PM Assignment 4

General

• The highlight feature really improved and the workflow is way better now
• Grading after selecting everything necessary is way better
• Comments taking over Code Search is a great feature and helped out a lot
• This tool autograded probably around 90% of all my assignments

Bugs

• [x] Different line marked in VSCode - the tool is showing the correct line of code

[Kuenssi]

2021-12-06, PM Assignment 04

Bugs

• [x] When someone else creates a solution and code search finds matches, while someone else is working on another grading. VSCode Highlighting and the fulib view does not update automatically

[Clashsoft]

2021-12-06, PM Assignment 4

General

• I was able to grade all submissions very quickly because most of Task 1 was done by Code Search.

Feature Requests

• [x] A way to select file names for Code Search. Would have helped to rule out all the "Not in package x.y.z", "Not in src/main/java" deductions.
• [ ] Some possibility to require multiple snippets to be in the same file or have an order. E.g., String name; could be found in both Game and Player. I had to select class Game { String name; to make it unambiguous. This does not work if there is more code in between. - #249

Improvement Suggestions

• [x] Remember solution search input when switching to solution detail view and back. I tended to open the solution in a new tab, but then clicked the "<" (Back) button and ended up in the unfiltered table with all submissions. Then I remembered to close the tab because the filtered table was still open in the original tab.
• [x] "Solutions Evaluated by Code Search" is not a useful statistic because in this assignment it was just 100%.
• Clone including the commit SHA. I don't think this is possible with the IDE URIs though.

[Kuenssi]

2021-12-13, PM Assignment 05

Improvement Suggestions

• [x] It would be awesome to see if i already graded something by one student in the overview for all people. I can't really check who i already graded without creating the github issue (Natascha and Julian H. would like to have this too)

[Morphclue]

2021-12-13, PM Assignment 05

General

• Button Open in GitHub was really useful to check Commits (VSCode needs extensions for that)

Improvement Suggestions

• [x] Token for the whole course instead of a single assignment. Additionally to that a dropdown inside VSCode to select the current assignment.

[Clashsoft]

2022-01-14, AlgoDS Assignment 1

General

• Generally well suited for Code Search.
• Many different variable names made it less efficient.
• Same code snippets in multiple files (e.g. int n = scanner.nextInt();) tripped up Code Search a little. Should have configured glob pattern to avoid this problem. In the end the effect on grades was negligible because everyone got this right.

Improvement Suggestions

• [x] Task Statistics are pointless right now when assuming all subtasks are graded for all solutions whether given zero, full or partial points. Instead of Points Total, Times Evaluated and % of All Solutions, it should show and sort by Average % of Points, calculated as Points Total / (Task Points * Number of Solutions with this Evaluation). This will work for positive as well as negative task points (deductions). Perhaps the other statistics could be shown after expanding the task row (like an accordion). In addition, the new space could be used to show an overview of Remarks and how often they were used.

Bugs

• [x] Solution by ... is useless if only Student ID is known.
• [x] In the solution table, the points badge highlighting prioritises Code Search over manual submission. I.e. the badge is teal when it should be yellow for mixed evaluations. Was indeed kinda irritating to not be able to see what was already viewed.

[Morphclue]

2022-01-24, PM Assignment 7

Bugs

• [x] Let us assume that a homework has 10P in total. If you find out that a project isn't executable you will grade -10P making it 0/10P in total. If autograde detects something wrong in a subtask it will also grade (for example) -1P. This results in a negative score in the homework: -1/10P. I think removing the possibility for negative scores would be a better solution, since negative scores shouldn't be possible in a homework.

[Kuenssi]

2022-01-24, PM Assignment 7

General

In this assignment were a lot of user errors while grading.

• Codesnippets without any context (just one line), which can come up in multiple files/methods
• Adding a codesnippet to a wrong task

Users will make errors for sure. And there is just so much you can do about it. The tool works as intended nothing to argue with that, but the ones grading have to be conscious about the dangers of blindly trusting code search.

The things i mentioned are just problems that no software can fix without double checking through other people therefore needing more time to grade something.

Maybe a little preview what codesearch would find with the selected codesnippet could help the graders to validate their input before making it "public".

[Clashsoft]

2022-01-24, PM Assignment 7

General

The assignment seemed promising for Code Search thanks to the fine-grained tasks, but user errors due to insufficient snippet context made it necessary to double-check code found by Code Search.

Feature Requests

• [x] View a list of solutions/evaluations affected by Code Search from a given evaluation.
• [x] Preview Code Search during selection.

Improvement Suggestions

• [x] If an evaluation is made with a remark for a task with negative points, it should show as -0P in the feedback. I believe that is more clear than 0P, because the latter might imply the parent task was failed whilst -0 clearly indicates that a comment was made with negative consequences.

Bugs

• [x] Clone method is broken as reported by @nataschu.

[Clashsoft]

2022-01-31, PM Assignment 8

General

The Code Search summary preview helped a lot for selecting snippets with sufficient context.

Improvement Suggestions

• [x] Don't highlight ungraded leaf tasks in green, it makes it harder to find what needs to be done still.
• [x] Statistics gets messed up when someone forgets to close the evaluation modal and goes for lunch - it should filter outliers, e.g. >1 min. Thanks @JulianWerner98 😉

Bugs

• [x] Users with token cannot view statistics.
• [x] If person A edits an assignment, then person B edits that, and person A edits again, the changes from B will be lost, because A had a saved draft from their first edit.
• [x] Code Search banner sometimes does not show up -- weird because it should always find at least one result: the solution that is currently being graded.
• [x] Code Search banner displays fractional hits:

[Clashsoft]

2022-02-07, PM Assignment 9

General

• Code Search preview banner still works well, but we sometimes ignored it being yellow or red when there was just one solution with more than one search result. For example, one person implemented light and dark mode FXMLs and had some duplicate snippets. Still, they got the task right.
• Code Search worked really well for this assignment.

[Clashsoft]

2022-02-11, PM Assignment 10

Bugs

• [x] Something was very wrong with Code Search today, it did not find any snippets containing a + (???), not even the one result that should always be from the repo in which it was selected.
  • Turns out this problem only affected views (Code Search tab and preview), because it came from incorrect encoding of the + in API calls.