search

mavoweb / sort

Sorting for Mavos!
0 stars 1 forks source link

Brainstorming #1

Open LeaVerou opened 7 years ago

LeaVerou commented 7 years ago

Sorting could be as simple as an mv-sort property. Its value could be hardcoded, or set via an expression (so you'd need to use mutation observers to monitor changes — Mavo has a helper for that). Both hardcoded and dynamic sorting has use cases. If set via an expression, it's trivial to make a sorting widget:

<select property="sort" mv-storage="none">
...
</select>

<div property="person" mv-multiple mv-sort="[sort]">

It may be useful to introduce a meta function that returns all property names within a scope, to make it possible to auto-populate such widgets. The mv-sort attribute should accept multiple fields, and there should be some kind of designator for descending instead of ascending sort.

Here are two possible syntaxes with their pros and cons:

One could argue for optional commas, but the SVG WG tried to do that in SVG and it proved a terrible idea.

Some implementation challenges to consider:

Dsan10s commented 7 years ago

Hi Lea, thanks for all the well thought out comments! Here are some of my thoughts on the things you brought up:

I agree a simple extra mv-sort attribute is a good way to make this work. Just to bring up further examples for consideration, another option is the way Angular does it:

<div ng-repeat="person in persons | orderBy:'-age'">

The equivalent Mavo code I suppose would be:

<div mv-multiple="persons | orderBy: '-age'">

However, I think having an mv-sort attribute is better for our use case, as I believe the syntax will be easier to understand by our users. We just have to be sure that we define behavior for when mv-sort is put on an element that has no mv-multiple attribute (do nothing? tell the users in some way that mv-multiple is expected?).

With respect to your proposed syntaxes: I prefer the space separated syntax (trying not too speak too biased since I come from the perspective of a programmer). Although the first syntax may be more readable, it is true that comma separated values don't normally appear in html, and I believe that this will throw our user base off more than be of help.

With respect to the implementation challenges to consider:

LeaVerou commented 7 years ago

As I have mentioned before, we cannot use a value for mv-multiple that is an expression, because the value of mv-multiple is already taken for the property name. The Mavo equivalent of Angular’s ng-repeat="foo" is mv-multiple mv-value="foo", so the orderBy (if we had it) would go there.

Furthermore, I'm not convinced sorting needs to be a new language construct. We can (and probably should) just define a sort(array, property1, property2, ...) function that takes an array and returns a sorted one based on the properties provided. That way, we've tackled sorting for computed lists with very minimal effort.

New language constructs are harder to learn and cannot usually be guessed, so I'm usually in favor of them only when they significantly improve readability. For example, look at spreadsheets, probably the most successful end-user programming paradigm today: Most things are implemented via helper functions, so all a user needs to do is lookup the right function for what they want to do, instead of having to learn a new language construct. However, for certain things, having to use functions is tedious. Concatenation is a good example: A concatenate() function exists, but is overly verbose for such a common task, which is why we have the ampersand operator, both in spreadsheets and in Mavo.

Good point about mv-sort on a non-collection! That is something to think about, for now it's ok to fail silently or just print a warning to the console.

I prefer the space separated syntax (trying not to speak too biased since I come from the perspective of a programmer). Although the first syntax may be more readable, it is true that comma separated values don't normally appear in html, and I believe that this will throw our user base off more than be of help.

@karger, thoughts?

  • I think it's like you said, when displaying data in a sorted order, it should just be presentational. In other words on an mv-multiple element that has mv-sort, we should not allow editing on this element. Rather, to edit the collection, you must perform those operations (add/remove/reorder with drag and drop) on a separate mv-multiple element without mv-sort attached to it. If there is no mv-multiple element with mv-sort attached to it that the user defined, perhaps when editing this element, we can display the data in its original order in a Mavo-created element directly next to the mv-multiple mv-sort element (or perhaps in a modal overlay, we can discuss this more if you agree with this strategy), that the user can perform operations on to affect the underlying data. (If you would like me to further explain this bullet point please let me know)
  • Having said my first bullet point, to answer your second, I don't think drag and drop should be enabled on the mv-multiple mv-sort element (but is doable on the mavo-created element if we decide to work with that implementation)
  • To your third bullet point, I think my first bullet point also solves this issue, since we're editing the data in an element separate from the element used to display the data in sorted order, so we won't notice any jumping when performing edits on this separate element.

So your list can be sortable or editable but not both?! Why? That makes no sense to an author or user. Yes, there are challenges that we need to solve, but IMO disabling editing is a cop-out. Worst case, we just disable drag and drop, which seems to be the trickiest one and doesn't make much sense (but should still be possible to drag & drop between lists, if there's a relevant mv-accepts). In general, crippling the user experience to make our life easier as programmers is an antipattern.

I'm not sure if I understand the last bullet point? Are you saying we need to figure out what to do if the underlying data does not have the same structure for every element in the collection?

Not at all. Let me try again: There is no element in your HTML that represents Collection objects. The mv-multiple element represents collection items. This distinction is rarely relevant however, but it has made certain cases tricky in the past (e.g. the mv-mode attribute, since it applies to both collections and collection items, so it's hard to disambiguate).

Dsan10s commented 7 years ago

To the mv-multiple as an expression comment, I do remember you saying that you can use it as shorthand for the property name. That's why in that example I used persons, because I would want the name of the collection to be persons. Not sure if my understanding of property vs mv-value is correct, but in this case I would want the collection in mv-storage to be called persons. I know the value of mv-multiple is already taken for the property name, but if we were to follow that example, couldn't we tweak the parsing of mv-multiples value such that it parsed out the first keyword in the expression and used that as the property name (the keyword being persons in this case, anything before the orderBy).

I also agree with your comments about not needing sorting as a separate language construct, so perhaps I shouldn't spend too much time trying to defend an example I don't support in the first place.

In your proposed sort, we'd also have to specify for each property whether we want that property in increasing or decreasing order correct?

So your list can be sortable or editable but not both?! Why? That makes no sense to an author or user. Yes, there are challenges that we need to solve, but IMO disabling editing is a cop-out. Worst case, we just disable drag and drop, which seems to be the trickiest one and doesn't make much sense (but should still be possible to drag & drop between lists, if there's a relevant mv-accepts). In general, crippling the user experience to make our life easier as programmers is an antipattern.

To this point, I don't think I'm proposing that lists can't be both sortable and editable, and when I proposed disabling editing, I only proposed disabling editing on the sorted data, and having the actual edits take place on the unsorted version of the data. Although, after some thinking, I think there might be a way to make this work.

One idea is when clicking edit, a user can switch between editing the sorted data and the unsorted underlying data (perhaps by using tabs?).

In the sorted 'tab' (I'm going to use the term 'tab' for now as that's the first thing I think of in terms of how this would be displayed to the user), the users can perform drag and drops to reorder how the data is displayed in that specific element. This give the user control over how they want sorted data displayed without affecting the order of the underlying data. For example, say a user wants 2 specific elements at the top of a collection, and every other element afterwards sorted by age, and they want to make these changes via drag and drop. The downside of allowing this is that in order to store this new order of the data, we'd have to store another custom-ordered version of the data in mv-storage (unless we're going to accept this order as the new order of the original data, which I'm not sure is a good idea, especially if we end up deciding that we will allow multiple mv-multiples for the same collection to be used on the same page). As for performing edits in the sorted tab, right now I just see the data remaining in the same order as long as the user is focused on it, and when unfocusing the element moves to its new place in the sort (if the edits cause the element to reorder). Perhaps an animation where the element moves to it's new spot could help with letting the user know why their element disappeared and where it went.

In the unsorted tab, performing edits works in exactly the same way that they work now, so I won't go in depth here.

My only concern is how to make it intuitive to the user the difference between making edits on the way the data is displayed (the sorted version), and how the data is stored (the unsorted version), especially since Mavo has had a great emphasis thus far on strongly connecting the way the data is displayed with how it is stored (what you see is how it is in mv-storage).

Ah I think I understand what you mean by your last bullet point, are you worried about the case where the user puts in a value for mv-sort that changes for each element? In that case I believe the sort should fail, and display the data in its original order (perhaps also notify the user in some way? Do we have a standardized way of alerting the user of Mavo errors right now?)

LeaVerou commented 7 years ago

To the mv-multiple as an expression comment, I do remember you saying that you can use it as shorthand for the property name. That's why in that example I used persons, because I would want the name of the collection to be persons. Not sure if my understanding of property vs mv-value is correct, but in this case I would want the collection in mv-storage to be called persons. I know the value of mv-multiple is already taken for the property name, but if we were to follow that example, couldn't we tweak the parsing of mv-multiples value such that it parsed out the first keyword in the expression and used that as the property name (the keyword being persons in this case, anything before the orderBy).

I see. Yes, it would be possible, though I'm not sure whether it would be desirable. Is there any argument to support doing it in the property name? "Angular does it" is not one. Angular originally tried to get novices to use it, failed, then pivoted and became an MVC framework for web developers. Since we don't want that fate for Mavo, we should be careful about taking its syntax as an example to follow. Yes, it's successful, but not in our target group. We should certainly look at it and how it solved the same problems, because there are a lot of smart people working on it, but we shouldn't necessarily follow those solutions without evaluating them on a case-by-case basis first.

I also agree with your comments about not needing sorting as a separate language construct, so perhaps I shouldn't spend too much time trying to defend an example I don't support in the first place.

👍 (I wrote the paragraph above before seeing this)

In your proposed sort, we'd also have to specify for each property whether we want that property in increasing or decreasing order correct?

Yes, and it's open to discussion how that could work and what the default should be. We could either use a +/- prefix, an optional asc/desc keyword, or something else.

I only proposed disabling editing on the sorted data, and having the actual edits take place on the unsorted version of the data.

I think we need to separate what we mean by "editing" here. There is absolutely no reason to disable or change the editing of property values for example. It's only collection editing (adding, deleting, rearranging items) that brings some challenges.

In the sorted 'tab' (I'm going to use the term 'tab' for now as that's the first thing I think of in terms of how this would be displayed to the user), the users can perform drag and drops to reorder how the data is displayed in that specific element. This give the user control over how they want sorted data displayed without affecting the order of the underlying data. For example, say a user wants 2 specific elements at the top of a collection, and every other element afterwards sorted by age, and they want to make these changes via drag and drop. The downside of allowing this is that in order to store this new order of the data, we'd have to store another custom-ordered version of the data in mv-storage (unless we're going to accept this order as the new order of the original data, which I'm not sure is a good idea, especially if we end up deciding that we will allow multiple mv-multiples for the same collection to be used on the same page). As for performing edits in the sorted tab, right now I just see the data remaining in the same order as long as the user is focused on it, and when unfocusing the element moves to its new place in the sort (if the edits cause the element to reorder). Perhaps an animation where the element moves to it's new spot could help with letting the user know why their element disappeared and where it went.

You raise a very good point here: that users or authors may want to override the sort for specific items, akin to "pinned" topics on a discussion forum. I'm not sure this is something that should be exposed to end users by default (any addition to the UI for end users comes with a ton of baggage, so we should be very conservative about it), but it should be possible for authors to put that together with the syntax we pick.

And I think it already is: If the author wants such a thing, they could have a property (e.g. a property="pinned" checkbox) and put it first in mv-sort. So the problem of editing a collection with overrides is basically the same problem as editing any sorted collection.

As for the challenges in that, we've so far only discussed drag & drop. What about adding items? Where do they go? What about editing properties that are involved in the sort? When does the item move so that the experience is not jarring? These are challenges we need to think about, we can't just disable adding items or editing properties.

But as far as drag & drop goes, I'm ok with disabling it. It's nonsensical in a sorted collection. I would suggest not just hiding it, but adding a disabled attribute to it, and a tooltip that explains why it's disabled.

Ah I think I understand what you mean by your last bullet point, are you worried about the case where the user puts in a value for mv-sort that changes for each element? In that case I believe the sort should fail, and display the data in its original order (perhaps also notify the user in some way? Do we have a standardized way of alerting the user of Mavo errors right now?)

Yes. We do, but we probably don't want to show an error to the end user for something like this, we can just display it in the console. There is also the Mavo debugger, but it is currently broken. I want to add a small helper function for warnings like this that displays them in the Mavo debugger if it's loaded, or just via console.warn() if not.

Dsan10s commented 7 years ago

I think we need to separate what we mean by "editing" here. There is absolutely no reason to disable or change the editing of property values for example. It's only collection editing (adding, deleting, rearranging items) that brings some challenges.

Right sorry, I used the term "edit" pretty loosely here. Yes I think there's no reason to disable editing of property values, but collection editing is what would pose issues.

As for the challenges in that, we've so far only discussed drag & drop. What about adding items? Where do they go? What about editing properties that are involved in the sort? When does the item move so that the experience is not jarring? These are challenges we need to think about, we can't just disable adding items or editing properties.

I envision one potential flow for adding items as follows:

  1. User clicks "Add " button ( is whatever item they are adding)
  2. New item appears at the bottom of the collection, with fields ready for editing, perhaps auto-focus the first editable field.
  3. User makes edits as normal, but the item only reorders itself when the user unfocuses from the item (clicks away, tabbing fields doesn't count)
  4. When the user does lose focus on the item after editing it, if the current state of its properties would cause it to be reordered in this sorted list, the item should animate to its new spot (I mentioned this in my last reply)

For editing properties involved in the sort, I also think the item should stay in its spot until the user unfocuses the element, at that point that's a trigger to reevaluate the state of if the list should be sorted, and have the item animate to its new spot.

Another option that I thought of was having a resort button visible in edit mode. While making edits, the data stays put (does not automatically resort itself). After having made whatever changes the user desires, the user can click this button to see the new sorted data. Also, saving and exiting edit mode, we should resort the data by default using this implementation.

But as far as drag & drop goes, I'm ok with disabling it. It's nonsensical in a sorted collection. I would suggest not just hiding it, but adding a disabled attribute to it, and a tooltip that explains why it's disabled.

I agree, there's not really an expected behavior that we can abide by with respected to sorted collection drag and drop, so disabling it seems like the most appropriate course of action. The additional tooltip information and disabled attribute will definitely help hit the point home to our users as well.

Yes. We do, but we probably don't want to show an error to the end user for something like this, we can just display it in the console. There is also the Mavo debugger, but it is currently broken. I want to add a small helper function for warnings like this that displays them in the Mavo debugger if it's loaded, or just via console.warn() if not.

This sounds good to me.

Another thought I'm going to throw into the mix is that our syntax should also support sorting by nested properties (e.g. mv-sort="+property1.nested1")

Also you mentioned providing a sort function earlier. Is this in addition to providing mv-sort, or are we thinking of providing one or the other? I just want to make sure I fully understand the purpose and usage of the function.

LeaVerou commented 7 years ago

I envision one potential flow for adding items as follows:

That makes sense. However, I just realized something: The properties have initial values anyway, either via mv-default or empty. That would place the new item somewhere in the sort order anyway. Its place in the underlying data is already well defined, based on which add button was clicked. So all we need to do for the new item is to smoothly scroll to whatever its sorted position is.

For editing properties involved in the sort, I also think the item should stay in its spot until the user unfocuses the element, at that point that's a trigger to reevaluate the state of if the list should be sorted, and have the item animate to its new spot.

Unfocuses which element? The property editor? Yes, as a minimum it shouldn't move with every keystroke. In any case, we can't just abruptly move it, that will be very jarring. Animation can help to show what's happening, but we still need to think what to animate.

Another option that I thought of was having a resort button visible in edit mode. While making edits, the data stays put (does not automatically resort itself). After having made whatever changes the user desires, the user can click this button to see the new sorted data.

That doesn't seem like a good idea, for the following reasons:

  1. As I mentioned above, we need to be very conservative with UI additions and only resort to them when there is no other reasonable solution. UI additions come with extra learning cost for end users, extra customization cost for authors, extra development cost for us because they need to work well with a very large variety of designs.
  2. This is not externally consistent: Users are not expecting to manually trigger sorting, it's something that typically happens on its own.
  3. It takes a burden off of us and puts it on end users. That's typically not a good plan, we should be striving for the opposite.

Another thought I'm going to throw into the mix is that our syntax should also support sorting by nested properties (e.g. mv-sort="+property1.nested1")

Of course. However, remember that there is a well-defined algorithm for resolving property references from anywhere in the Mavo app without the need for the dot notation except for disambiguation. The same should work here, which probably means that the code should be abstracted away from Mavo.Node#relativizeData() into a separate method that can be used outside of expression stuff.

Also you mentioned providing a sort function earlier. Is this in addition to providing mv-sort, or are we thinking of providing one or the other? I just want to make sure I fully understand the purpose and usage of the function.

In addition, as it solves different use cases. For example, imagine you have a collection of people and you want to show all names somewhere, alphabetically. If all you had was mv-sort, you'd need to create a pointless collection with mv-value just so you could get the sorted names from it and pass them through join().

It may be a good idea to start with sort() since it's much easier to implement because it only deals with data, no UI implications.

karger commented 7 years ago

I'd like to consider use cases before getting to syntax. What kinds of things do we see on the web today?

  1. sites with data sorted by some rule, not under control of users
  2. sites that let users pick from a small set of sorting options (specific properties)

Are there any other categories?

I can't think of any sites that let users both edit data and sort data---can you? So one easy out is for us not to offer this functionality (at least for now).

If we do want to offer both, perhaps a simplification would be that sorting is disabled whenever editing is enabled, and vice versa. In particular if you've sorted the data, then activate editing, the data stays in its currently sorted order but doesn't re-sort while editing is happening. When the user exists edit mode the data re-sorts with its new data. This also means you wouldn't be able to apply sorting to a collection with auto-edit/auto-save activated.

karger commented 7 years ago

Another approach that simplifies many things (but comes with its own issues) is to consider sorting as a data manipulation action instead of as a functional dependency. That is, instead of saying "the collection is a reactive function of the sort specification, like other spreadsheet function", we say "sorting the collection is a data manipulation action, like dragging a single item out of order." This fixes some of the problems observed above. For example, if I have a "sorted" collection and drag an item, that's fine---I've changed the order to a different order. If I want it to go back to being sorted, I have to invoke the sort again. Similarly, if I add an item, it stays where I added it, because that's the order I specified.

Not that this is a compelling example, but this is how app managment seems to work on my (android) phone. I can get a list of apps, and whenever I want I can sort it A-Z. However, if I add a new app, it shows up at the end---after the Zs. If I want to get a sorted list again, I have to reinvoke the sort.

We could, of course, offer an "auto-sort" override that the author can use to specify that the collection should be kept sorted always (drag and drop disabled, adds jump to the right place, etc.).

One potential issue here is that if we literally think of sorting as a data manipulation then when we save the data we'll need to save it in the newly sorted order. Which would really mess up the version control diff functions on github, as they don't do well with re-ordering. I suppose we could think of this as a kind of "soft" data manipulation that doesn't get saved. Alternatively we could save the state of the sort (e.g. what is the chosen sort property) with the data in original order instead of changing the actual data order. But this could get in trouble if e.g. someone sorts and then starts dragging items---the order no longer has a compact representation.

The idea that sorting is a manipulation can be extended to filtering, although in that case we certainly don't want to save the result of the manipulation since it would cause data loss. But the concept that inserting an item into filtered data set leaves it visible there even if it doesn't fit the filter, since the filter "finished previous to the insert" and is no longer enforcing.

this could all also coexist with a sort function that is reactive and instantly updating and so on.

Dsan10s commented 7 years ago

I think one thing we should also decide is if we want multiple mv-multiples to be allowed to be used on the same page for the same collection (use case: separate sort/filter conditions, which would allow for grouping without needing to move data between collections). If so, then having sort be a data manipulation action would not work, as we would have to store multiple versions of the data in various sorted orders.

LeaVerou commented 7 years ago

I think one thing we should also decide is if we want multiple mv-multiples to be allowed to be used on the same page for the same collection (use case: separate sort/filter conditions, which would allow for grouping without needing to move data between collections). If so, then having sort be a data manipulation action would not work, as we would have to store multiple versions of the data in various sorted orders.

Not sure I follow. If multiple collections are used for the same data, one is the canonical one that is saved and the rest are via mv-value, with different property names. We don't want to be saving multiple versions of the data with different orders, that sounds like a maintenance nightmare.

karger commented 7 years ago

I can't remember if I already emailed this, but if we're thinking about sorting we should also think about grouping, and in particular what syntax would allow someone to specify that e.g. they want to sort their cities by country but also want a header for each country (containing at min the name of the country, but perhaps you want to format other info into the head as well).

LeaVerou commented 7 years ago

I can't remember if I already emailed this, but if we're thinking about sorting we should also think about grouping, and in particular what syntax would allow someone to specify that e.g. they want to sort their cities by country but also want a header for each country (containing at min the name of the country, but perhaps you want to format other info into the head as well).

Not sure about modifying the user’s template. Perhaps we can make this available via expressions…

Dsan10s commented 7 years ago

Hi, just wanted to bump this conversation so we can flesh out any final details so development can start.

It's sounding like we were somewhat agreeing on having an mv-sort item not be editable? (In mv-storage, collection is unsorted, displayed as sorted to user, user can make changes to each item, but cannot make collection changes like reordering) Is this still true? If so, unless there's other discussion points, perhaps development can start?

LeaVerou commented 7 years ago

It's sounding like we were somewhat agreeing on having an mv-sort item not be editable?

I don’t see any such consensus. @karger you want to clarify your views a bit? I recall when we discussed this f2f you said you were not suggesting to disable editing while the data is sorted.

Dsan10s commented 7 years ago

I believe I misspoke, when I said mv-sort item, I meant the the mv-sort collection. So individual item editing still allowed, but resorting not allowed.

It might get confusing, so when talking about editing let's try and specify if we're talking about collection or item editing.

LeaVerou commented 7 years ago

The only thing I see consensus on is that drag & drop should be disabled when sorting. I don't see consensus on:

Several solutions have been proposed that include:

But no consensus. In any case, sometimes it's easier to experiment with ideas when they are more concrete, so if you want to start coding and accept that you may have to completely change the behavior, then that's fine by me (but do wait for @karger to opine as well).

Dsan10s commented 7 years ago

So in that case, I'll at least start on an implementation where drag and drop is not allowed when sorting, and we can figure out adding items, deleting items, and property editing behavior as we go.

Does this sound good?

On Mon, Aug 7, 2017 at 12:44 AM, Lea Verou notifications@github.com wrote:

The only thing I see consensus on is that drag & drop should be disabled when sorting. I don't see consensus on:

  • Adding items
  • Deleting items (why disable that?!)
  • Editing properties (if they participate in the sort, resorting as you type may be jarring)

Several solutions have been proposed that include:

  • Disabling editing altogether
  • Certain ways to handle the issues, e.g. to smoothly scroll to new/resorted items
  • Not updating the sort while editing

But no consensus. In any case, sometimes it's easier to experiment with ideas when they are more concrete, so if you want to start coding and accept that you may have to completely change the behavior, then that's fine by me (but do wait for @karger https://github.com/karger to opine as well).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mavoweb/sort/issues/1#issuecomment-320592683, or mute the thread https://github.com/notifications/unsubscribe-auth/ADdHTFABNUFG4COT-wnaeMj-ddygjNJgks5sVsBJgaJpZM4OBnmt .

-- All the best,

Daniel Sanchez

Dsan10s commented 6 years ago

@LeaVerou 's proposed improvements to the current state of sorting:

as a first pass, you could try resorting on the mv-change event of the properties you're sorting by, IFF they are on read mode and on mv-done (which is fired when we exit edit mode) in the future, we could try resorting during edit too, if the user is idle or something, but that's less important