User interfaces, UI widgets and related issues

[jcelerier]

I'm opening this issue to start a discussion about user interface, and the semantic level on top of OSC.

As the various implementations start to become available, everybody wants to create nice UI overlays to access OSCQuery parameters. I don't think that the spec should enforce anything on this (what's the point) BUT I believe that it should be able to help - and allow at least some level of compatibility with existing frameworks & libraries which do UI generation.

For instance :

• The various utilities provided by @mrRay
• Interactive Shader Format
• Faust (hslider / vslider / groupbox / etc...)
• LV2 (http://lv2plug.in/ns/extensions/ui/ui.html) - yay for logarithmic sliders (http://lv2plug.in/ns/ext/port-props/port-props.html)
• I maintain a small list of UI libraries for creative coding as part of the one I'm working on with @aklevy, qml-creative-controls (at the bottom of the readme) : https://github.com/jcelerier/qml-creative-controls
• The folks at vvvv are working on RabbitControl which shares sensibly the same goals : https://github.com/rabbitControl/RCP and have an extensive widget specification : https://github.com/rabbitControl/RCP/blob/master/RCPWidget.md ; I'm pretty sure that any kind of collaboration on this would be fairly valuable. @i-n-g-o, @joreg, ping :D
• There's ofxRemoteUI : https://github.com/armadillu/ofxRemoteUI

So, the simplest thing for this would be to say :

Add a "WIDGET" field to the spec with the expected UI widget kind - "LineEdit", "Slider", "ColorChooser"

But is it the right solution ? I don't know. Another possibility would be to work on the semantic level, that's the way we are mostly following in ossia with the UNITS and EXTENDED_TYPE attributes.

[i-n-g-o]

one thing we figured when discussing widgets for rabbitControl is that the discussion is endless. we decided to only specify what we think are the most important widgets with the most important features. as with the datatypes the protocol (RCP) allows custom-widgets.

we hope that people start to come up with their own widgets, relevant widgets then could be integrated in a future specification.

custom widgets would either need a custom client of course, or a client with widget-plugins (type-plugins) - which could be downloaded automatically...

widgets in a rabbit-parameter are proposed widgets (a parameter wishes to be rendered as the widget it defines) - is that what you mean with recommended_widget? if the client does not support the widget it would fall back to the most simple widget: likely to be a text-input-field, or even just a non-changable textfield.

[mrRay]

this is a very good question.

no, i don't think that we should try to derive a WIDGETS attribute in this proposal.

if there was already an established widget protocol that we could pick up and use, i think it would be a good idea to consider its adoption in OSCQuery- but there isn't, and i don't think that this repository/project is the appropriate venue for creating such a protocol.

generally speaking, i'm on board with the whole notion of a "widget protocol"- i just don't think that we should be trying to define that here and now. over the last week, the discussions between dave and myself have been all over the place- a "widget protocol" encompasses a lot of things. there's a great deal to consider, and if it's going to be a well-formed proposal then it needs to take into consideration situations that are going to be outside the immediate concerns and limitations of OSCQuery, and it needs to include a substantially wider audience.

that being said, this question didn't come out of the blue: i think this was born of a desire that software be more consistent, and that's good. to that end, we were thinking of updating the proposal to include some examples of UI items to consider using for various OSC data types- nothing fancy, just a listing of the basic UI items and conventions in the wild that we're presently aware of (suggestions, not rules). here's what we've put together so far:

Basic UI Elements:

• Slider/Knob- some kind of UI item for sweeping across range of UI values
• Momentary Button- A UI item that sends an event that does not have an accompanying value ("bang" to borrow a term from max)
• Toggle Button- A UI item that sends an on/off value
• Text Field- A UI item that allows for text entry, capable of displaying both number values and string/character values
• Color Picker- A UI item that allows the user to specify a color- this can take any form (RGBA slider, HSL space, etc)
• Pop-Up Menu/Radio Button- a UI item that presents the user with a number of options they have to choose between.

UI Element suggestions for OSC Type Tags:

OSC Type Tag	Data Type	UI Item Suggestions
f,d	Floating-point numbers	Slider/Knob, Text Field
i,h	Integer numbers	Slider/Knob, Text Field, Toggle Button (if limited to 0/1)
s,S,c	Strings/Alphanumeric chars	Text Field
r	Color	Color Picker
T,F,N,I	No value (type is value)	Momentary Button
b	Data blob	None
m	MIDI data	None
t	OSC timetag	None

Existing Conventions:

• OSC methods that accept multiple values can be expressed using a combination of the above basic UI elements.
• The VALS attribute explicitly lists the acceptable values- if this is being used, you may want to consider using a pop-up menu or radio button to express the available choices, depending on the context.
• Some apps express integer-type values limited to 0 and 1 (this can be done either via the MIN/MAX or VALS attributes) as toggle buttons.
• Some apps express integer-type values with a relatively small range as a Pop-Up Menu or Radio Button, depending on the number of possible values.
• Some apps that work with images express OSC methods with the type tag string "ff" (two floating-point values) as a single UI item capable of specifying a two-dimensional point input.
  
  
  

...if there's something we've overlooked and you'd like to see added to the above list of conventions, please let me know...

p.s.- RabbitControl looks very, very interesting- thanks for sharing this!

[jcelerier]

T,F,N,I | No value (type is value) | Momentary Button

hmm, so the OSC spec is a bit weird in this regard since the "value" is part of the typetag, but I believe that the main point of T / F is not to have an OSC method which can only ever receive a "true" or "false" message, but can take one of either - the spec is mostly about sending messages and I think that they just wanted to optimize the size of messages with many booleans (else it'd look like /foo/bar ,BBBBBBBB true true false true true false true false instead of /foo/bar ,TTFTTFTF).

So, my point is that, I would find it more logical to treat a T or F value in the typetag as a boolean type, for which we know the value but which can entirely be flipped from one state to another. Hence a toggle or checkbox seems more logical.

Apart for this, I'm in agreement with your message - in addition we plan to do the following whether applicable with libossia apps:

• extended_type is to be used to show a file selector when relevant
• units can provide more specificities to ui widgets: for instance, some units (gain, frequency) are logarithmic, and as such it makes sense to provide logarithmic sliders. Likewise for all the color units.

[bltzr]

So, my point is that, I would find it more logical to treat a T or F value in the typetag as a boolean type, for which we know the value but which can entirely be flipped from one state to another. Hence a toggle or checkbox seems more logical.

I agree with @jcelerier on that !

And I also agree on all the rest of your proposals @mrRay and @dlublin

[bltzr]

Some apps that work with images express OSC methods with the type tag string "ff" (two floating-point values) as a single UI item capable of specifying a two-dimensional point input.

that's also the case for sound spatialization, so my impression is that a 2D widget is a quite common case - though, that brings us into vectors and all that, so maybe that'll be opening yet another can of worms...

Also, some other thoughts:

Concerning the bool case, that's precisely why I opened #48 and after those discussions, I think we should recommend support for the 3 listed cases:

• integer with 0/1 MIN/MAX
• integer with 0/1 VALS
• OSC-typetage T or F What do you think ?

Concerning the slider/menu choice, we've been discussing this off-issue @mrRay and I think your point is right, that a menu is most often more appropriate than a slider for a small range of values, but this is not universal, and sometimes it's better to have a plain slider. Though, I don't see any other way (in the current proposal) to hint at that than with the VALS attribute. Anyway, we don't have to legislate on that, and we can just mention that both options can be appropriate depending on cases.

[mrRay]

"So, my point is that, I would find it more logical to treat a T or F value in the typetag as a boolean type, for which we know the value but which can entirely be flipped from one state to another. Hence a toggle or checkbox seems more logical."

if we were dealing with boolean data types then i would agree with you: a toggle or a checkbox is a more logical choice than a momentary button. however, we aren't dealing with boolean data types.

we are dealing with the OSC types T and F, which are two completely different valueless data types that have been confusingly named after the boolean states "true" and "false". a boolean data type, by definition, must have two possible values. OSC "T" and a OSC "F" do not fit this definition at all precisely because they have no value. as far as i can tell- and somebody should correct me on this if i'm mistaken- the OSC spec does not have a boolean data type.

if you aren't comfortable using N and I as alternate states for a toggle- and you shouldn't be, i'm pretty sure the wider OSC community favors using "I" to communicate valueless events like "bang"- then you shouldn't be comfortable using T and F to represent a toggle, either.

"and I think that they just wanted to optimize the size of messages with many booleans (else it'd look like /foo/bar ,BBBBBBBB true true false true true false true false instead of /foo/bar ,TTFTTFTF)."

motivations aside, the fact remains that T and F are explicitly defined as different data types in the OSC spec, and the TYPE attribute in this proposal uses OSC data types to express the type tag string expected of OSC messages sent to that address. OSC is a strongly-typed language (one of the many reasons declaring T and F as different types is a questionable) and this proposal reflects that- frankly, i do not think we should be trying to carve out exceptions and special behaviors for some data types just because they're misleadingly named.

i'm not suggesting this proposal drop support for T and F, nor that it should ignore T and F- i'm only suggesting that this proposal should treat T and F the same way the OSC spec treats them: as totally separate valueless data types.

on a practical level, how many apps actually require the use of T and F? i don't recall seeing it outside of experiments/academic circles (student patches, one-off projects that don't need to be compatible with other stuff, etc). @jcelerier, you mentioned in another thread that lots of OSC apps require the use of T and F- would you mind listing some of those for us?

"and I think your point is right, that a menu is most often more appropriate than a slider for a small range of values, but this is not universal, and sometimes it's better to have a plain slider."

i'm sorry, but i don't understand what change you're suggesting here; would you mind re-reading the comment that this is in response to, and then telling me specifically what changes you'd like to see made? i'm slightly confused because sliders are already listed first, and pop-up menus are only listed as a footnote...

[bltzr]

i'm sorry, but i don't understand what change you're suggesting here;

I'm the one that has to be sorry, because my comment was particularly unskillful, so let's try that again, hopefully better this time :

I think that the proposal of a menu / pop-up for small-range numbers is interesting and even sometimes much more relevant than a slider (as you pointed me to in a private discussion), so I'm just wondering how we can express this preference in the frame of the current proposal. (I'm not sure there's an answer to this currently, or if it's desirable that there be one)

Is that any better?

[bltzr]

concerning the boolean issue, I think we all agree the current specs are a bit weird on this point Also, the optimisation example you mention, @jcelerier, - even if I can imagine might have been a motivation behind the current specs - is really a edge case that doesn't show up so often, does it?

[jcelerier]

we are dealing with the OSC types T and F, which are two completely different valueless data types that have been confusingly named after the boolean states "true" and "false".

I really think that the expectations of the OSC spec authors was to have T/F represent a boolean. That's what they refer to in this paper : https://pdfs.semanticscholar.org/e499/3f0b3aa1bda8127a9ed653250803a3fb8901.pdf (which is if I'm not mistaken the first mention of T/F) :

uOSC packs boolean data types using the ‘T’ and ‘F’ typetags

and in the 1.1, as a way to communicate bits : http://cnmat.org/files/attachments/Nime09OSCfinal.pdf - if the exact same set of bits had to be communicated at each message it would not really be useful :

The True, False, Null and Impulse/bang types are useful for efficiently communicating bit strings, empty arguments and events, a very common scenario in robotic control, in gestural interfaces and bit twiddling for hardware development.

on a practical level, how many apps actually require the use of T and F?

As far as I know, in term of "big" app, there's at least Renoise, Ventuz, Wizard of OSC, ZynAddSubFX, Ardour, which support or have an API that use T/F. Besides, those types are referenced in the very first OSC tutorial for pure data which is I suppose the first step for a lot of people (at least it was for the students I taught :p) : http://booki.flossmanuals.net/pure-data/network-data/osc - note how the same address pattern is used to send a message with T and another with F. There is also the case of environments & libraries that will automatically convert C / C++ / C# / Java / ... bools to T/F, e.g. OpenFrameworks does this if you use ofxOSC, as well as the OscPack library. Other environments such as SuperCollider are considering doing this too.

Here are some docs where T/F are matched to a single boolean type :

• LabVIEW : http://www.ni.com/white-paper/14398/en/
• Ventuz: https://www.ventuz.com/support/help/V4_05/Remoting4Protocol.html

OSC is a strongly-typed language

hmm... where are you getting this from the spec ? AFAIK there is nothing in any of the OSC official or almost-official documents that binds types to particular address patterns - it's only the format of messages to these address patterns that are fixed (that is, for a given message, the arguments must match the message's type tag), but it's perfectly correct to send, e.g. /foo 1.234 followed by /foo "hello world" "goodbye world" - even if it's a fairly nasty thing to do. I read it and re-read it but there are only so many places where the word "type" is mentioned. Note that I'm of course of the opinion that such freedom should be quelled ! but should T/F be sacrificed to this ? I don't think so.

[bltzr]

just a candid question: is it such a burden to support both int[0-1] and T/F ?

[jcelerier]

@bltzr the question is, what happens when you want to show an UI for something that tells you that it's T or F (and, more fundamentally, should you allow to send a "F" to something that states that it is a "T" and conversely). If something says that it's an int... well... it will show an int slider that goes from 0 to 1 or a combo box with 0 / 1 ?

[mrRay]

"I think that the proposal of a menu / pop-up for small-range numbers is interesting and even sometimes much more relevant than a slider (as you pointed me to in a private discussion), so I'm just wondering how we can express this preference in the frame of the current proposal."

hah- that's ironic, as a result of the same conversation i explicitly listed sliders first- as you pointed out, it's a relatively common pattern and in my situations a generic pop-up button isn't as usable. i listed pop-up buttons as a footnote because i figured it's a minority use-case.

i'll try putting it in the table- if it looks awkward because the asterisk/explanation we'll play it by ear...

[mrRay]

"the question is, what happens when you want to show an UI for something that tells you that it's T or F (and, more fundamentally, should you allow to send a "F" to something that states that it is a "T" and conversely). If something says that it's an int... well... it will show an int slider that goes from 0 to 1 or a combo box with 0 / 1 ?"

yep, that's one of the things i've been thinking about. i think this debate is centered around two lines of text from the respective proposals:

• the OSC spec explicitly declares T and F as different data types.

• the OSCQuery proposal's TYPE attribute uses an OSC type tag string to describe the data types expected by the OSC method.

hypothetical situation: you have an OSC method with a declared type tag string of "T".

• what's the VALUE of this OSC method? technically, a valueless data type can't have a value- its value is determined solely by its data type...i'm assuming it'd just return "T" because that's probably what people expect...

• how would you express that this OSC method can also receive OSC messages containing the data type "F"? like it or not, the spec defines "T" and "F" as different types, and the query proposal states that the TYPE attribute dictates which data types the method expects to receive. if you want to be in compliance with everybody you'd have to OVERLOAD the method, right? why do we have to overload the method if T and F are the same data type?

• let's say you send an OSC "F" to this method. has its VALUE changed? how can the value of a valueless type change? what do you even return in a situation like this? T and F have been defined as different data types, so if you return "F" then you're returning a value of the wrong type, even if its value is technically correct...

sorry, but i'd like to ponder this one a little bit more!

[mrRay]

i think there are a number of nasty logical inconsistencies with this decision, and neither dave nor myself think that this is the right thing to do from a design standpoint...but you make a good point that practical concerns are not to be taken lightly, and this means supporting other apps that may not share this point of view.

so: what's the best way to go about this? OVERLOADS are fine, right?

{
    "FULL_PATH": "/asdf",
    "TYPE": "F",
    "OVERLOADS": [
        {
            "FULL_PATH": "/asdf",
            "TYPE": "T"
        }
    ]
}

[bltzr]

concerning the bool issue, a few opinions of mine:

• may I suggest that we propose a EXTENDED_TYPE "bool", so that helps cater with the different ways this is dealt with (either int[0-1] and T/F, which are both surprisingly cumbersome ways to deal with such a simple matter in the 21st century, but, well... that's the specs we have to deal with)

what happens when you want to show an UI for something that tells you that it's T or F

I'd say we'd have a toggle of some kind

(and, more fundamentally, should you allow to send a "F" to something that states that it is a "T" and conversely)

well, what would be the point of having this on top of N and I (which are already so close semantically to each other) if that wasn't to deal with boolean states ? So, I would go for a: yes.

If something says that it's an int... well... it will show an int slider that goes from 0 to 1 or a combo box with 0 / 1 ?

I think the convention proposed by @mrRay makes perfect sense: if an int as a RANGE or VALS {0, 1}, then we can reasonably consider it's a bool (and then display a toggle)

As for the OVERLOADs, I understand your concern, @mrRay and maybe my position is a sloppy one, but I wouldn't bother with (or at least enforce) the use of OVERLOADS for this. Since T is really here to go with a F IMHO, then we can just go along with the specs, and consider a T node will accept to receive a F. That's really the fault of the way the specs are designed, but I think we should just accept this failure and make things as simple as possible to cater with them. Mileages may vary on this, though, I guess.

[mrRay]

"As for the OVERLOADs, I understand your concern, @mrRay and maybe my position is a sloppy one, but I wouldn't bother with (or at least enforce) the use of OVERLOADS for this."

sure- in for a penny, in for a pound...

[adrianfreed]

Congratulations on this interesting and civil discussion of an interesting evolving use of OSC!

OSC is primarily an encoding, a structuring or "punctuation" for data. Because it came relatively early (I used precursors to OSC in 1986) and from experimental contexts it has always been rather lightweight/open (loose) on interpretations of the data. This has allowed the user community to explore different things - at the price occasionally of ambiguous and contradictory interpretations.

If I remember correctly the TF stuff came about when we learned that people were use OSC to control strings of solenoids for amusement part rides and it was ugly to bit-stuff long strings of Booleans into fixed length integer types. At CNMAT we were doing work with micro controllers (and eventually Arduino flavored ones) where you want to twiddle bits with devices with anything from a few to hundreds of pins.

From a high level application perspective we might want to talk about HIGH/LOW, ON/OFF, GO/STOP, DEFINED/UNDEFINED 1/0 etc. There is an old tradition in computer languages to represent these two-state situations with a built-in boolean type. I have become less enamored with this over the years. Frequently I find I need to express the fact that a value is not known yet (the startup state problem). In electronics we frequently manage a third state (HIGH/LOW/TRISTATE). On Arduinos the API started out with pins being INPUT or OUTPUT. Now we have OUTPUT, INPUT, INPUT with PULLUP and INPUT with PULLDOWN, Weak and Strong Pullup etc. etc.

At CNMAT we developed many interesting musical user interfaces that represented activities probabilistically. There are many flavors of logic outside Boolean logic that are helpful here real-valued-logic, fuzzy-logic etc. We can reasonably use a floating point 0-1 for this and in some situations it is useful to combine this with TF.

In rich or complex applications we developed a practice around CNMAT of using large OSC bundles to describe these high level interpretations of data represented in the small number of primitive types encodable by OSC. It's a sort of lightweight/stateless/data type system. If you want to do this properly you would probably using XML but I have always been afraid in real-time situations of the complexity of the schema documents being separate from the data. To illustrate my personal pedantic approach to all this where button toggles were involved I paste below the text representation of an OSC bundle used for the Arduino Explora in our Max/MSP o.io library: /Manufacturer "Arduino" /Device "Esplora" /Serialnumber 1099 /Sequence/Number 1133 /acceleration/x 0.0117188 /acceleration/y 0.0878906 /acceleration/z 0.28125 /photoresistor 0.954057 /slider/horizontal 0.969697 /connector/white/right 0.926686 /connector/orange/left true /connector/orange/right false /temperature/fahrenheit 69. /temperature/celsius 20. /microphone/loudness 0. /led/rgb 0 0 0 /joystick/horizontal -0. /joystick/vertical 0.0175781 /connector/white/left 0.498534 /joystick/button/down false /joystick/button/up true /joystick/forward/button/down false /joystick/forward/button/up true /joystick/left/button/down false /joystick/left/button/up true /joystick/right/button/down false /joystick/right/button/up true /joystick/backward/button/down false /joystick/backward/button/up true /diamond/backward/button/down false /diamond/backward/button/up true /diamond/left/button/down false /diamond/left/button/up true /diamond/right/button/down false /diamond/right/button/up true /diamond/forward/button/down false /diamond/forward/button/up true

(in this text representation true false is the encoded TF stuff) Why is the button state represented twice with up and down? This approach allows for musical keyboards during the time the key travels between the top and button conductor and with regular switches before you have decided how to debounce them... This approach is also handy to represent soft buttons on devices like the Quneo which sense pressure because you can add a further parameter such as: /diamond/forward/button/pressure 0.2

In this same spirit of self-documenting OSC messaging it would be useful to explicitly indicate that this "pressure" was bound to the unit interval: /diamond/forward/button/range/pressure 0.0 1.0

p.s. I often field heckles on efficiency at this point - we worked through the details long ago of caching and hashing OSC messages- something that is very developed, mostly transparent and now in most browser/server interaction protocols. What I am always trying to optimize for is developer/user time and I like that the documentation for an OSC message is the message.

[joreg]

helo everyone, thanks for bringing rabbitcontrol into the discussion. i would like to quickly line out what it is and how we handle the question of UI widgets.

up front:

• the protocol is binary, which makes it hard for debugging but hopefully easier to implement for microcontrollers
• it is transport-agnostic, our prototypes are using websocket, since our main client will be web/mobile

rabbit's idea is very similar to OSC+OSCQuery but not identical. we're focusing mostly on the usecase of rapid UI creation from exposed parameters. a host/server application can expose parameters describing it in 4 aspects: 1) value: datatype (float32, float64,... vector2,... range,..string, uri, rgba) + type specific options like min, max, default,... 2) widget (optional) 3) layout (optional): think different auto-layout options 4) style (optional): thinks css

we're quite confident with 1) already, 2) seems to fit the topic of this question, 3) and 4) are not yet covered by our specification. it is our hope that with one reference-client we can already cover a lot of usecases where people are not so worried about the looks of their control UI but simply need a fast way to control their parameters. individual clients with specialized fancy UIs are obviously possible to create.

a client that receives a parameter without a widget specified, will resort to displaying the default widget for the specified datatype. but datatypes like float32 could be controlled via different widgets (slider, knob, numberbox), therefore, if the optional widget property is set, the client should display the specified widget.

ad 2) as you can see in our spex: https://github.com/rabbitControl/RCP/blob/master/RCPWidget.md the main idea for a widget is to have one of the specified types (see: Widget type table) plus a number of options. some general widget options (enabled, label-visible,..), some specific to the type of widget (e.g. textbox: multiline, wordwrap,..). for any option that is not set, the client knows a default.

to my understanding the definition of such a widget-type-table + options-per-widget could potentially be shared between rabbit and oscquery...

[mrRay]

"thanks for your feedback, @adrianfreed , that helped taking the final decision on this boolean case, as documented here."

no, it definitely didn't. don't get me wrong, i appreciate the context and find it interesting- but it's simply not accurate to say that this had any impact on the decision at all, which we made before we were even aware of this comment.

it's important to be clear about these things: we agreed to compromise under the assumption that the use of T/F as two different values for the same data type is, while uncommon, not completely unheard of, and that its absence would make this proposal less useful.

"so, now that this looooong (but necessary) digression on bools has come to its end, I'd suggest that we now try and wrap up the initial topic of this issue: UI widgets."

it's already wrapped up! i added the list of conventions to the repos days ago, and i've been keeping the issue open because i know it takes some people a few days to catch up and i thought they might find freed's post interesting (i sure did).

"I'm also wondering if we shouldn't add a WIDGET attribute"

we already discussed this- no, we shouldn't.