search

oughtinc / mosaic

A web app for recursive question-answering with pointers
MIT License
37 stars 3 forks source link

It's quite hard to understand what's going on when trying to analyze an experiment #571

Open brachbach opened 5 years ago

brachbach commented 5 years ago

...if you're not super familiar with the app

this was driven home to me today by analyzing some workspaces with OpenAI folks today -- it was pretty hard for the OpenAI folks to figure out what was going on (they had a few hours of experience with the app before we met, either with the training materials or with trying to do the analysis on their own)

This is a more of an "epic" or something, really each of the hard/confusing things below should be its own issue, but I didn't think it was worth it for me to create all of them. Also I think generally dealing with this stuff is a lowish priority (see below). Mainly I just wanted to capture everything that I noticed today that wasn't already documented elsewhere, to be able to pluck the lowest-hanging fruits as far as quick app fixes, to make clear that the UX for analysis is pretty bad right now, and to show that doing "user testing" of analysis was a useful exercise as far as surfacing things that could be improved

Things that are hard/confusing:

  1. subtree view doesn't make sense for Factored Evaluation
  2. it's confusing that you have to log in (as an admin?) to view workspaces in the proper admin mode
  3. it's confusing that pointer numbers are different between each workspace and between any workspace and the compactTree view
  4. questions and answers are pointers that can be referred to by other workspaces, but don't appear as pointers (appear as plaintext) in the compactTree view
  5. can't see the history of a workspace
  6. hard to see what depth level of the compactTree hierarchy you're at (it's hard to count the indents). I'd propose that each level should its own background color to indicate its depth.
  7. no easy way to see where a pointer came from (command-f on the compactTree view is the best way)
  8. it's easy to get confused about what options are available to judges during the experiment vs. only appear in the analysis view (there was some confusion about whether the "open" button for subquestions appeared to judges during the experiment)
  9. the "Entire Tree" button in the analysis view of a workspace should go to the compactTree view, not the subtree view
  10. (probably not really an app concern, more like an analyst training thing) -- it's not clear where to start to analyze a tree. Long's intuition was to traverse the tree breadth-first -- I think depth-first (or actually, starting at the leaves and then going up) is pretty clearly easier to understand
  11. I'm thinking that instead of A1$X and A2$Y, they should just be A$X and A$Y. I don't think that the 1 or 2 really adds anything, and it's some visual noise
  12. It's weird that app features added later (such as the "judge subquestion") button show up when you're analyzing an earlier experiment

My guess is that 9 and 11 are super easy fixes that we should go ahead and make.

6 would also be nice if it's easy, but prob not if it would take more than a couple hours.

Everything else is probably not worth changing for now (and may well never be worth changing for Mosaic 1)

brachbach commented 5 years ago

Was thinking that an alternative approach to the ones of these that are about the workspace view would be to make the compactTree view very suited for analysis, or to create a similar tree view that's very suited for analysis, and stop using the workspace view for analysis.

The tree analysis view would then need to have everything that's currently in the workspace view:

  1. ability to see scratchpad
  2. ability to see which pointers are locked vs. unlocked
  3. ability to see input budget remaining

(This seems like a reasonable amount of work to get right, so would prob hold off. I think we'd really need at least all 3 of these things, and maybe also some other things I'm not thinking of, to be able to stop using the workspace view for analysis)

Mati-Roy commented 5 years ago

Vojtech mentions

Can you somehow add some id to each Q and A in the Tree View? Or somehow make sure that pointers in the Tree View do not change and we can refer to them in the google doc? It will be better than coping whole questions or answers to the google doc. In this moment it is probably no big problem but in future when will we use google doc for longer time, it will be crazy to going through the document looking for something which is relevant for me.