"highlight" vs "filter"

[benloh]

The current "Filters" panel only fades the filtered nodes and edges.

Need to add a different type of filtering system that temporarily removes the node or edge from rendering without affecting the core data.

Issues

• Want to be able to do both kinds of filtering still.
• Not sure if both kinds of filtering need to be active at the same time?
• What is the use model for filtering? How would you work with such a filter?

---

Highlight Filtering

“I want to highlight things that meet my criteria but see them in situ in the whole network vis.”

The current filtering should be renamed highlight.

Highlight To Do

• [x] Rename current "Filters" panel to "Highlight"
• [x] Highlight and Filter are mutually exclusive

Filter

“I want to filter for things that meet a specific set of criteria and remove everything else, with the attendant force-direction re-layout that accompanies removal of nodes/edges”

Filters To Do

• [x] Add new "Filters" panel
• [x] A removed node should also remove any associated edges
• [x] Reduce frequency of FDATA, D3DATA updates
• [x] Add degrees as a stored calculated value
• [X] Export filtered data Converted to issue #170
• [X] Add form entry for degrees
• [x] Show filtered nodes/edges in tables? Tables should still show unfiltered data (perhaps with indicator of removal)
• [ ] Rewrite Transparency hack to use a proper Filter definition. Skip for now. Too complicated. Save for later if necessary.

[benloh]

@kalanicraig @jdanish It would be very helpful if you could write out a user scenario for how one might use a "remove" filter as part of an investigation. In particular, in trying to figure out the right UI for it:

1. Can you provide examples of what you might be filtering? And the sequence of steps that you would use in conjunction with it? What do you do after you set the filter? Are you wanting to remove other combinations of nodes/edges? (In other words, is the current limited filter form adequate?)

2. How would the use of a "remove" filter work in conjunction with a "fade" filter (the current filter)? Would you want both active at the same time or is only using one type of filtering at a time adequate?

3. In your use of the existing filter, do you find that the filter area eats up too much of the screen real estate? Do you still have enough room for graphs? Do we need to revisit how the form entries are constructed to better optimize the screen?

Understanding the rationale for the filtering will help us make decisions about best to implement.

For example, you mentioned removing "Mark Antony" from the "Cleopatra" graph. What kinds of investigations would you do after you remove "Mark Antony"? What would you look at? Click on nodes/edges? Add a fade filter on top of the remove filter? Add nodes/edges while the remove filter is active? Is this just a temporary state? Does it need to be saved? Do you need a history for the types of filters you had created? Do you need to be able to share a URL that can restore the filter?

Also, do you have a better name for the filters? "remove" and "fade" seemed the most obvious to me, but "remove" can be a little deceptive since we're not actually removing it. "hide" doesn't quite work because it doesn't convey the sense in which the force effects are also cancelled out.

[kalanicraig]

Easy stuff first:

The nomenclature I’d use is “filter” vs “highlight” . So “I want to filter for things that meet a specific set of criteria and remove everything else, with the attendant force-direction re-layout that accompanies removal of nodes/edges” or “I want to highlight things that meet my criteria but see them in situ in the whole network vis.”

I would vote not to offer filtering and highlighting at the same time. It’s useful but other programs that are more robust will do it, and do it better (if not as prettily).

I think the filter display works because it’s a toggle drop-down (the way the nodes and edges works). We might need a short statement of what’s currently filtered so that it’s clear the network is filtered when the filter toggle is closed (e.g. filter drop down shows you the menu settings, but then there’s a line by the key that reads “NETWORK VISUALIZATION CURRENTLY FILTERED: node degree between 2 and 56”)

There are lots of reasons to highlight (finding specific keywords in nodes, node types, node names). Not all of these highlight requests will apply to filters (though the existing UI might lead us to apply them anyway): Centrality. We might want to remove all nodes from the network with a specific degree measurement. If you remove the highest degree-central nodes, you can see how the rest of the network breaks into subnetworks (or doesn’t) and who clusters where. If you remove the lowest degree-central nodes (nodes with zero or 1 degree), you can see the network’s main structure more clearly. Node or edge type filtering. A network comprised of only nodes labeled people, with no nodes labeled organizations or groups, so that you could see when people communicate directly vs when there is a non-person intermediary, and how much of the network is predicated on those direct connections A network comprised of only nodes with specific edge-type relationships. For instance you might hide any node with out a “positive relationship” edge, or hide all nodes with “negative relationship” edges, so that you can see the network nodes that participated in a particular kind of interaction. I think the second half of the first question is: do we need to do any of these things at the same time (e.g. a node type of person with a relationship type of “positive”). Ideally, but there are other tools that do that, so if the answer is that we need to choose either/or, then we do simple well and save the complex for exports.

Which leads to a question that’s not here: can we export a filtered or highlighted network state in static form for viewing?

Hope this helps!

—Kalani

On Nov 8, 2021, at 5:09 PM, benloh @.***> wrote:

@kalanicraig https://github.com/kalanicraig @jdanish https://github.com/jdanish It would be very helpful if you could write out a user scenario for how one might use a "remove" filter as part of an investigation. In particular, in trying to figure out the right UI for it:

Can you provide examples of what you might be filtering? And the sequence of steps that you would use in conjunction with it? What do you do after you set the filter? Are you wanting to remove other combinations of nodes/edges? (In other words, is the current limited filter form adequate?)

How would the use of a "remove" filter work in conjunction with a "fade" filter (the current filter)? Would you want both active at the same time or is only using one type of filtering at a time adequate?

In your use of the existing filter, do you find that the filter area eats up too much of the screen real estate? Do you still have enough room for graphs? Do we need to revisit how the form entries are constructed to better optimize the screen?

Understanding the rationale for the filtering will help us make decisions about best to implement.

For example, you mentioned removing "Mark Antony" from the "Cleopatra" graph. What kinds of investigations would you do after you remove "Mark Antony"? What would you look at? Click on nodes/edges? Add a fade filter on top of the remove filter? Add nodes/edges while the remove filter is active? Is this just a temporary state? Does it need to be saved? Do you need a history for the types of filters you had created? Do you need to be able to share a URL that can restore the filter?

Also, do you have a better name for the filters? "remove" and "fade" seemed the most obvious to me, but "remove" can be a little deceptive since we're not actually removing it. "hide" doesn't quite work because it doesn't convey the sense in which the force effects are also cancelled out.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-963619147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NFAOXNJ6MTCZAEKPBDULBDAZANCNFSM5HTVLZ7Q.

[jdanish]

Just noting that there is a filter display that appears - Kalani just hadn’t seen it.

Also, one limit with the current filtering is that you can’t multi-select. So you can say “contains A” or “contains B” but not “contains A or B” which would be nice, but would complicate the interface a bit. If it is doable I can pitch some interface options.

Let us know if we can further clarify!

Joshua

---

Joshua A. Danish (he, him, his) (https://www.mypronouns.org/ https://www.mypronouns.org/)

@.***

http://www.joshuadanish.com

On Nov 9, 2021, at 4:57 PM, Kalani Craig @.***> wrote:

Easy stuff first:

The nomenclature I’d use is “filter” vs “highlight” . So “I want to filter for things that meet a specific set of criteria and remove everything else, with the attendant force-direction re-layout that accompanies removal of nodes/edges” or “I want to highlight things that meet my criteria but see them in situ in the whole network vis.”

I would vote not to offer filtering and highlighting at the same time. It’s useful but other programs that are more robust will do it, and do it better (if not as prettily).

I think the filter display works because it’s a toggle drop-down (the way the nodes and edges works). We might need a short statement of what’s currently filtered so that it’s clear the network is filtered when the filter toggle is closed (e.g. filter drop down shows you the menu settings, but then there’s a line by the key that reads “NETWORK VISUALIZATION CURRENTLY FILTERED: node degree between 2 and 56”)

There are lots of reasons to highlight (finding specific keywords in nodes, node types, node names). Not all of these highlight requests will apply to filters (though the existing UI might lead us to apply them anyway): Centrality. We might want to remove all nodes from the network with a specific degree measurement. If you remove the highest degree-central nodes, you can see how the rest of the network breaks into subnetworks (or doesn’t) and who clusters where. If you remove the lowest degree-central nodes (nodes with zero or 1 degree), you can see the network’s main structure more clearly. Node or edge type filtering. A network comprised of only nodes labeled people, with no nodes labeled organizations or groups, so that you could see when people communicate directly vs when there is a non-person intermediary, and how much of the network is predicated on those direct connections A network comprised of only nodes with specific edge-type relationships. For instance you might hide any node with out a “positive relationship” edge, or hide all nodes with “negative relationship” edges, so that you can see the network nodes that participated in a particular kind of interaction. I think the second half of the first question is: do we need to do any of these things at the same time (e.g. a node type of person with a relationship type of “positive”). Ideally, but there are other tools that do that, so if the answer is that we need to choose either/or, then we do simple well and save the complex for exports.

Which leads to a question that’s not here: can we export a filtered or highlighted network state in static form for viewing?

Hope this helps!

—Kalani

On Nov 8, 2021, at 5:09 PM, benloh @.***> wrote:

@kalanicraig https://github.com/kalanicraig @jdanish https://github.com/jdanish It would be very helpful if you could write out a user scenario for how one might use a "remove" filter as part of an investigation. In particular, in trying to figure out the right UI for it:

Can you provide examples of what you might be filtering? And the sequence of steps that you would use in conjunction with it? What do you do after you set the filter? Are you wanting to remove other combinations of nodes/edges? (In other words, is the current limited filter form adequate?)

How would the use of a "remove" filter work in conjunction with a "fade" filter (the current filter)? Would you want both active at the same time or is only using one type of filtering at a time adequate?

In your use of the existing filter, do you find that the filter area eats up too much of the screen real estate? Do you still have enough room for graphs? Do we need to revisit how the form entries are constructed to better optimize the screen?

Understanding the rationale for the filtering will help us make decisions about best to implement.

For example, you mentioned removing "Mark Antony" from the "Cleopatra" graph. What kinds of investigations would you do after you remove "Mark Antony"? What would you look at? Click on nodes/edges? Add a fade filter on top of the remove filter? Add nodes/edges while the remove filter is active? Is this just a temporary state? Does it need to be saved? Do you need a history for the types of filters you had created? Do you need to be able to share a URL that can restore the filter?

Also, do you have a better name for the filters? "remove" and "fade" seemed the most obvious to me, but "remove" can be a little deceptive since we're not actually removing it. "hide" doesn't quite work because it doesn't convey the sense in which the force effects are also cancelled out.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-963619147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NFAOXNJ6MTCZAEKPBDULBDAZANCNFSM5HTVLZ7Q.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-964583255, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKGCKWOTIN3MIPZBHSH2TTULGKNTANCNFSM5HTVLZ7Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[benloh]

Thanks! This is really helpful.

So teasing this apart...

network is filtered when the filter toggle is closed

If you add a filter, and then click on the "Graph" tab, there is a short summary of the currenly active filters that displays.

degrees

It sounds like we need to add the ability to filter/highlight by degrees.

export filtered/highlighted data

I believe it should be relatively easy to do this once we get exports working.

"or" filtering

I think this would significantly complicate the interface, at least as we currently have it. We'd probably want to re-implement it as a line-by-line UI, ala the Finder's Search feature: e.g. you first select which dimension you want to search, and then can add subsequent dimensions. It's more powerful, but I find the interface a little more difficult to use (e.g. you have to remember which dimensions you can use and do an extra selection). On the other hand, the UI would be much more compact.

We can probably do this if this type of filtering is important. That was one of the reasons I was asking for scenarios. The "nodes labeled people, but no nodes labeled organizations" would require this kind of interface.

Let me know if you think we should pursue a complete rewrite.

[benloh]

One more question about filtering:

For "highlighting" it makes sense that you would set "Label contains Theodora" in the form to highlight particular elements. So implicitly the search is "show me only things that match xyz"

But "filtering" is kind of the opposite: when you set a filter you're saying "remove everything that matches xyz".

• e.g. "remove label contains Theodora" from the graph,
• e.g. "remove nodes with degrees > 10"

I would think in general that with filtering you're mostly removing a few select items rather than having to carefully set filters that keep everything but a few. E.g. if filtering were "show" rather than "remove", in order to remove Theodora, you'd have to set "label does not contain Theodora" in the form, which seems harder to get your head around?

We can make this explicit with a bit of help text on each panel. But I can see it leading to some confusion.

Unless you think we should treat both the same way? e.g. either both are "show" or both are "remove"?

[kalanicraig]

Ah. OK, I see “keep my selections” option would work best (so “highlight Nodes with Theodora in the notes field” or “keep nodes with degree range between 1 and 52 and filter everything else out.”)

How would that work in terms of the rewrite question?

On Nov 9, 2021, at 7:59 PM, benloh @.***> wrote:

 One more question about filtering:

For "highlighting" it makes sense that you would set "Label contains Theodora" in the form to highlight particular elements. So implicitly the search is "show me only things that match xyz"

But "filtering" is kind of the opposite: when you set a filter you're saying "remove everything that matches xyz".

e.g. "remove label contains Theodora" from the graph, e.g. "remove nodes with degrees > 10" I would think in general that with filtering you're mostly removing a few select items rather than having to carefully set filters that keep everything but a few. E.g. if filtering were "show" rather than "remove", in order to remove Theodora, you'd have to set "label does not contain Theodora" in the form, which seems harder to get your head around?

We can make this explicit with a bit of help text on each panel. But I can see it leading to some confusion.

Unless you think we should treat both the same way? e.g. either both are "show" or both are "remove"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[jdanish]

I'm all for the move to a finder-style line-by-line filter, and think we need that eventually. However, I defer to Kalani on whether that fits her timeline / budget on this project.

Also random thought - I wonder if while re-organizing we should / can change the name of the repo to not be 2018? Maybe just version 1.5 or something? Or not putting it in the repo name?

[benloh]

@kalanicraig I think the challenge is helping people understand what's going on and how that behavior might change as you move from the "Highlight" tab to the "Filter" tab. Probably the best thing to do right now is take a first pass and see if it's clear/confusing and iterate.

A finder style line by line filter is probably a pretty big rewrite. What we can do is try to get through the three main project goals first and then come back and see if we have time.

Also, @jdanish renaming the repo is already on the Project Board: https://github.com/netcreateorg/netcreate-2018/projects/3, as #161. This ought to be something you can do. I think renaming is relatively easy. The hard part is chasing down all the links and references.

[jdanish]

Waiting on the rewrite makes sense to me.

I think if the help text on top of the panel says clearly: "Hide / fade the following nodes or edges:" that'll help.

Another option would be to make a switch so that the default is "show" but the user can switch it to "hide" or "fade" depending on whether they are in filter / highlight mode, and then it'll be obvious what they are doing and they can pick the logic that makes the most sense to them at the time.

The first is cleaner, the second is more flexible. I'll defer to Kalani's take...

[benloh]

Help text is definitely necessary. I'll see if I can easily add a checkbox (and it doesn't add confusion).

[jdanish]

I think a dropdown might be easier to interpret so that the sentence reads as hide / show rather than [x] hide or [ ] hide but … cool.

On Nov 10, 2021, at 12:15 PM, benloh @.***> wrote:

Help text is definitely necessary. I'll see if I can easily add a checkbox (and it doesn't add confusion).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-965560276, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKGCKTIOQ3VQP22JG4VGMDULKSCTANCNFSM5HTVLZ7Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[kalanicraig]

Agree that a full finder-like interface would take time, but I do like the drop-down in sentence context. It matches how we’ve chosen to label edge types as fully articulated sentences.

On Nov 10, 2021, at 12:16 PM, Joshua Danish @.***> wrote:

I think a dropdown might be easier to interpret so that the sentence reads as hide / show rather than [x] hide or [ ] hide but … cool.

On Nov 10, 2021, at 12:15 PM, benloh @.***> wrote:

Help text is definitely necessary. I'll see if I can easily add a checkbox (and it doesn't add confusion).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-965560276, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKGCKTIOQ3VQP22JG4VGMDULKSCTANCNFSM5HTVLZ7Q. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/netcreateorg/netcreate-2018/issues/167#issuecomment-965561481, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACKL4NAZLJ2I5FYQXGS6GGLULKSHRANCNFSM5HTVLZ7Q.

[benloh]

Degrees already works. The type just needed to be set to "number" (instead of "hidden") in the template.

[jdanish]

Confirmed. I forget when but I think you told me and my templates are all correct now.

[benloh]

Marking this closed. Re-open if needed.