Add parametricDrawingOf (generalizes LSU's randomDrawingOf and guiDrawingOf)

[cdsmith]

FYI @alphalambda

I think I'm hopeful that randomDrawingOf and guiDrawingOf belong in the core CodeWorld API. They make for a nice transition between animations and more general parameterized drawings.

I'd like to combine them into a single entry point, though. Maybe parametricDrawingOf? That's a horrible name, but the idea is not to use "widget" because some of the common and earliest uses might not affect the UI at all. I'll keep thinking, though!

The API would look something like:

-- Main entry point:
parametricDrawingOf :: ([Number] -> Picture, [Parameter]) -> Picture

-- Specific parameters:
slider :: (Text, Range, Point) -> Parameter
randomNumber :: Range -> Parameter  -- (invisible)
randomButton :: (Text, Range) -> Parameter
prompt :: Text -> Parameter  -- (invisible after program start)
currentHour, currentMinute, currentSecond :: Parameter  -- (invisible)
currentTime :: Parameter  -- (invisible, time since program start)
timer :: Point -> Parameter  -- (visible, with start and stop buttons)

-- Utility type, mainly to prevent confusion between position and range
data Range where Between :: (Number, Number) -> Range

-- API for creating parameters: basically activities with local state and an output number.
parameterOf
    :: ([Number] -> state,  -- Input is stream of random numbers
        (state, Event) -> state,
        state -> Picture,
        state -> Number)
    -> Parameter

Example:

program = parametricDrawingOf(picture, params)

params = [
    slider("Size", Between(1, 10), (5, 5)),
    randomNumber(Between(-10, 10)),
    randomButton("Change color", Between(0, 1)),
    prompt("How many circles?"),
    currentSecond
    ]

picture([size, x, color, num, sec]) = ...

@alphalambda also has a scheme where transformations can be applied to a widget (such as withConversion, etc.) rather than reimplemented each time. I need to investigate this. It seems to require a more complex UI for defining widgets (and indeed, in Extras.Widget, there are no user-defined widgets at all), and yet not be sufficient for anything more than what can be accomplished by passing a range in as a parameter. In particular, transformations might be interesting if they could be changed over the life of the program, but they cannot in the current implementation. My guess is that students would rather just give the desired range as an argument.

@alphalambda also has a scheme for stateful controls, which looks way trickier, but maybe useful. Not sure how to think about that yet!

[cdsmith]

Some of the parameters above (like currentMinute, currentHour, prompt, etc.) would need unsafePerformIO in the implementation. I think that's probably okay, and better than exposing IO to the user in the API. In particular, since Parameteris an opaque type with no operations defined on it directly, the use of unsafePerformIO incurs a burden only to check that it's used correctly by the implementation of parametricDrawingOf.

[cdsmith]

prompt was a port from the Extras.Do by @alphalambda but it's really a poor fit here, because in the original, it didn't need to be a number! There's no good way, though, to support parameters of heterogeneous types, so we'd only get some of the functionality.

[cdsmith]

One concern: this is a LOT to add to the prelude, especially if it consumes common words like timer. And I could see growing a bigger collection of parameters in the future. That makes me wonder if we might still want to put this into a new Parameters module.

[cdsmith]

@alphalambda

We have a working implementation of this in codeworld-api now. I've been tinkering a bit lately. The big question I have now is this: should placement of the widgets/parameters be up to the individual control, or should it be automatic?

The current implementation leaves each component choosing its own position, so a student has to give x and y coordinates to draw it at. This is fine if you imagine yourself creating a GUI as a product. But if the idea is duplicate things like Desmos sliders with a very simple student UI, then it seems to make more sense to place them automatically. So a student could just write parametricDrawingOf(picture, [timer, slider(1, 5), randomNumber(1, 10)]), and the three components would automatically add themselves to the screen in different and non-overlapping locations. Perhaps they would even be draggable, in case the student wanted to move them around manually. This would make it impossible for a student to do something like build their own calculator. This is a loss, but it doesn't seem like a huge one: I don't think anyone's goal here is to teach the use of GUI component frameworks.

[alphalambda]

I have been discussing this same issue with some people, and they pointed out that in systems like Desmos the sliders are actually in a separate panel. They think it is important that students do not perceive the sliders as "just a way to interact with your image" but as a separate interface to control the underlying parametric model. If students perceive it as a way to interact with images, they will want more user interaction and probably will find the GUI to be artificially restrictive, which may demotivate them. On the other hand, if the students perceive the sliders as just a way to modify the parametric model, then they will be more content with them, just as they are with the Desmos interface.

I guess having the sliders in a separate panel requires too many changes to the user interface, so having them automatically place themselves could be a reasonable trade-off. I'm not sure about letting the user drag them, because they become again "interaction with the image". Probably, autoplaced sliders that can be hidden would accomplish the same goal. Rather than be draggable, I'd prefer having a key, such as the spacebar, that can be used to show/hide all the sliders at once.

Nevertheless, I would still like the conversion to be performed explicitly and not based on ranges, for pedagogical reasons, even if it they are slightly more inconvenient to use. And sliders should still have labels identifying them. So, I'd rather have your example be:

parametricDrawingOf([ timer("jump"), withConversion(slider("parameter 1"), \x -> 1 + 4x), withConversion(randomBox("New height"),\x -> 1 + 9x) ], mainPic)

On Tue, Dec 10, 2019 at 11:13:35AM -0800, Chris Smith wrote:

@alphalambda

We have a working implementation of this in codeworld-api now. I've been tinkering a bit lately. The big question I have now is this: should placement of the widgets/parameters be up to the individual control, or should it be automatic?

The current implementation leaves each component choosing its own position, so a student has to give x and y coordinates to draw it at. This is fine if you imagine yourself creating a GUI as a product. But if the idea is duplicate things like Desmos sliders with a very simple student UI, then it seems to make more sense to place them automatically. So a student could just write parametricDrawingOf(picture, [timer, slider(1, 5), randomNumber(1, 10)]), and the three components would automatically add themselves to the screen in different and non-overlapping locations. Perhaps they would even be draggable, in case the student wanted to move them around manually. This would make it impossible for a student to do something like build their own calculator. This is a loss, but it doesn't seem like a huge one: I don't think anyone's goal here is to teach the use of GUI component frameworks.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/google/codeworld/issues/1243#issuecomment-564188463

[cdsmith]

Great, sounds like we're basically in agreement on the overall direction.

I think it's worth seriously considering putting the controls into a separate collapsible panel outside of the program. The major disadvantage would be that we'd lose the ability to define new parameter types in CodeWorld; we'd be stuck with a fixed set of system options, which would be implemented in JavaScript with a postMessage-based protocol to communicate with the running program. But the major advantage would be that we can use accessible GUI widgets that work with the system clipboard, focus model, screen readers, etc. Also, they stay out of the way of the student picture and don't feel like part of the program, as you mentioned.

[cdsmith]

Actually, I think putting the controls in a different window or panel would be a bad idea. That's because it would make them inaccessible when you embed the result into its own frame. It still might be better to use native JavaScript controls that float over the canvas instead of being drawn within CodeWorld, but I'd want them to live in the run frame, not in the editor. This is a key difference from Desmos: in CodeWorld, you're making something that lives outside of the editor.

[alphalambda]

Wouldn't it be possible to have the runtime iframe contain 2 panels, one with the input widgets and the other with the image output?

I'm not sure whether it is worth the additional complexity or not, but I think it is possible. The same problem is currently present with the message panel. When the output is embedded (or fullscreeen) errors and traced outputs are not shown, which could be disconcerting sometimes. So, actually 3 panels (input,output,messages) seems to me the proper setting for the runtime frame.

Example: https://code.world/run.html?mode=haskell&dhash=DL6CpzqJlgVGxahR5vRaBTw

Corresponding source code: https://code.world/haskell#P7q2VD2vum01B_W1dsxAJxA

(Same thing could happen in the educational mode with my Do library)

On Tue, Dec 10, 2019 at 02:47:59PM -0800, Chris Smith wrote:

Actually, I think putting the controls in a different window or panel would be a bad idea. That's because it would make them inaccessible when you embed the result into its own frame. It still might be better to use native JavaScript controls that float over the canvas instead of are drawing within CodeWorld, but I'd want them to live in the run frame, not in the editor. This is a key difference from Desmos: in CodeWorld, you're making something that lives outside of the editor.

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/google/codeworld/issues/1243#issuecomment-564295006

[cdsmith]

I'm aware of the issue with stdout; when running in a frame of its own, it logs to the console instead. I think that's a more reasonable choice than adding a GUI output panel to every program. Students really don't think of programs as things that have standard output stream anyway, so I see stdout as a debugging tool, which should be hidden when the program is run outside of the IDE.

The parameterization stuff is less clear, but I'd rather take the less opinionated stance, which would be to hover over the program rather than adding a new panel. I'll try to build a sample implementation at chessai's Haskathon this weekend, and see what you think.

[cdsmith]

What do you think about this?

https://code.world/haskell#PPRc3vsmyMBZwfjT7mX_6sA

[cdsmith]

One thing that would need to be fixed here before release is the Inspect window. Currently it shows the structure of the widgets in addition to the user's own drawing. Entering inspect mode should probably hide the widgets so that only the user program is inspected.

[alphalambda]

These are very similar to what I had. Here are a few suggestions:

I made the sliders have no background so that they do not block too much of the picture underneath. If you want to keep them inside the output panel and want them have a background, then they should be draggable. Otherwise, the user basically loses a band 4 units wide at the left side of the output, so they may want to put the "center" of their picture at (2.5,0) instead of (0,0).

The timer I have can be restarted, and that is sometimes handy.

Showing labels and values all the time works only for short labels. Otherwise, they spill over too soon.

These are all minor things, but maybe you could consider them.

On Wed, Dec 11, 2019 at 06:09:04PM -0800, Chris Smith wrote:

What do you think about this?

https://code.world/haskell#PPRc3vsmyMBZwfjT7mX_6sA

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/google/codeworld/issues/1243#issuecomment-564817569

[alphalambda]

A couple more suggestions:

• Maybe the space bar should toggle hiding/showing all the widgets
• The currentHour, currentMinute, etc Parameters do not need associated visuals. I may want to use them to build my own clock, and having them show is not very useful.
• I don't see the point of having a constant parameter. Just because it can be done may not be a good reason to do it, and I'd rather have students use normal definitions to define constants.

On Wed, Dec 11, 2019 at 06:09:04PM -0800, Chris Smith wrote:

What do you think about this?

https://code.world/haskell#PPRc3vsmyMBZwfjT7mX_6sA

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/google/codeworld/issues/1243#issuecomment-564817569

[cdsmith]

Good point about names. One idea is that, even if we don't expose it to users, we could add a clipping feature to use internally. But I could also redesign the widgets to be less likely to overflow, since clipping is obviously not a great experience either.

I'm feeling more and more like these should be draggable. I don't think it's feasible to drop the background without making the controls moveable, because they will not be legible for many background colors. I do think a 90% alpha look might be good, though, to at least make it feel like there's something behind the control.

So I have something like this in mind:

+----------------------+
| paramName         ◯ |
+----------------------+
|                      |
| control ui           |
|                      |
+----------------------+

where the button in the title bar is a minimize button, other clicks on the title bar are drags, and clicks in the control ui area are handled by the control.

Other points:

• I agree about adding a restart for the timer.
• Adding a space-bar to hide controls sounds fine. I want to throw in some kind of interstitial to make sure students can find the controls again, but I can do that.
• I debated with myself for a while about params that don't have meaningful UI. I ended up deciding to include them just because otherwise it makes them feel like a different kind of thing. If we allow params to be minimized, and minimize these by default, and also allow space bar to hide params entirely, then perhaps this could minimize the disadvantages, but still present these as params for the user.

[cdsmith]

Missed one.

• I think the reason for having constant as a Parameter is that it makes it easier to temporarily set a constant value for something that was originally something else, like a time or random number. I can imagine doing this for debugging: I identify a broken test case using random, and temporarily set the param to constant instead while fixing that case. Then I set it back to random when I'm done. This is definitely debatable, and doesn't need to be decided right away.

[cdsmith]

I have made some, but not all, changes to this code.

1. All parameters are now floating windows that are draggable and collapsible.
2. Backgrounds are partially transparent so it's clear that part of the drawing extends behind them.
3. Relevant sections of the UI are clipped so that parameters cannot extend outside their windows.

Still to do:

• Fix the Inspect window.
• Add a UI to hide all params.
• Add restart to the timer.
• Fix some visuals, especially where I was lazy and used unicode emojis instead of better icons.
• Auto-collapse parameters that have no UI.

Overall, I'm feeling pretty good about this.

[cdsmith]

Currently just missing this:

• Fix some visuals, especially where I was lazy and used unicode emojis instead of better icons.

Example: https://code.world/haskell#PwgkSRmvsPV3E27dpRhV2AA

[cdsmith]

Also:

• Move param window to top when it's clicked.

[cdsmith]

Remaining work:

• Fix some visuals, especially where I was lazy and used unicode emojis instead of better icons.
• Port the API to codeworld-base.
• Final sanity check on the API.

My current test case: https://code.world/haskell#Pe8KuRQghqMVGvj2aIyR6lQ

@alphalambda Feedback is appreciated. I know I've made some different choices than you would have (e.g., always showing widgets even if there is no UI), and I am interested in whether for realistic examples you think the extra UI is too intrusive.

[cdsmith]

Updated for latest API changes: https://code.world/haskell#P0p4sdhr7YZ1527pWKqjoxQ

[cdsmith]

I have hidden parameterOf, for now. I find the argument compelling that there's no great need to add new parameter types, and unfortunately the interface for that got more complicated than I hoped.

[cdsmith]

Education variant: https://code.world/#PJ3GtJJXlBPKCsA9Yz_ieFQ

Still to-do:

• Fix some visuals, especially where I was lazy and used unicode emojis instead of better icons.
• Final sanity check on the API.