Closed gogonzo closed 1 year ago
Please consider following:
F
, how the AGE should be affected:
I will be using this ticket to track FilterPanel redesign for sprint 62. Updating SP to 21 to match Kendis.
So rather than thinking about general filtering I wonder if we should be understanding what types of filters are most useful and making the filter panel easier to use for those cases (for example some datasets/variables are much more important to filter on) - if as an app users you are not aware of all the details of all your datasets it can be daunting to set the filters you want.
So you could imagine a "subgroup filter" which allows users/app developers to specify a set of subjects based on some filters in say ADSL which filters the rest of the data accordingly - each of these subgroup filters is named and then in the filter panel you can easily turn on/off a subgroup filter
You could imagine a paramcd filter, an endpoint filter, a visit filter and maybe others which provide other intuitive ways of filtering the data - you then can of course have a completely general filter so more obscure cases can still be done. Certain modules (in say goshawk) which require say a paramcd filter could require an app developer to include one in their filters argument etc.
When creating/editing/viewing details of these filters a modal is used and say for a paramcd filter the information/UI could be tailored to that type of filter and in the filter panel you see (maybe with a mouse over) some details and the different filter types could be coloured and easy to turn on and off.
You could even have a mark on the filter which donates whether the filter is "turned on" for all modules or only the current one sort of handling whether the filter is (currently) global or not.
You could then imagine the (teal) reporter being able to do smarter things with the filters (i.e. defining a set of filters at the beginning of the report and then saying in the cards "Output for Subgroup XXX, for details see filter list"
The above of course would also need to work for MAE data (i.e. select a set of genes/experiments) and also require some analysis to work out what filters would most help the user base and is a big step away from here is a list of columns of your data to filter and so may not be feasible without a huge amount of effort.
I'd also hope there's opportunity for specialism of the filter panel as say if you have helios data how you handle filtering may be different (and that also opens up app developers/module developers to create their own filter panel if what we provide isn't sufficient)
You could imagine a reset to default option which resets the filter panel to exactly how the app developer created it
One important feature request that I hear from e.g. dashdis folks sometime is to allow for predefined filters. So would it be possible for teal to support a simplified filter panel as alternative option, so that the user does not need to select stuff from dropdown menus etc. but can just click on simple on/off buttons, to select efficacy population etc.
KEYWORDS: SPECIALISM, SIMPLIFY, SHARED-ENGINE Summing up we need to have a FIlter Panel factory (design pattern) which will provide different panels for specific scenarios. Some of these panels will be highly simplified (@danielinteractive suggestion). From the code maintenance perspective it will be great to have as much as possible shared engine for all type of filter panel.
From above document I conclude that we need couple of things which require consistent API and UI for app-developer (and user) to set them up. I can distinguish so far independent filter-settings which can be combined together to achieve specific effect:
teal
users expectation is to have simple filter-flags which de facto are single TRUE/FALSE choices, but this can be easily extended with the &
operator and also doesn't prevent us to extend this to multiple-conditions (more below).chevron
there is a need to set fixed-local-filter-label so that this filter can't be removed or modified.Above requirements are made on the users requests, but I can see the need for more if we want to discuss this together with data-merge refactor. Considering filter-spec
being specified only once per dataset it might be wort to move this functionality into filter-panel, which can be extended by:
Above can be achieved with consistent API and new UI which can extend filter-panel to data-panel where user can add filters, create a labels/variables. Please have a look on the wireframe below, key notes:
API CALL
# I want to start the app with filters on: ADSL.gender{M, F} and ADRS.avisit{baseline, week 1}
filters = list(
ADSL = list(
gender = list(c("M", "F"), keep_na = TRUE)
),
ADRS = list(
avisit = list("baseline", "week 1")
)
)
UI element
# I want to limit filter variables to: adsl.{gender, age, sex, race, country, arm} and adrs.{avisit, paramcd, aval}
filters = list(
ADSL = list(
...,
filterable(c("gender", "age", "sex", "race", "country", "arm"))
)
)
# I want to add module-filter: adrs.paramcd{besrspi, invet}
filters = list(
ADRS = list(
avisit = list(...),
paramcd = list(
c("basrspi", "invet"),
modules = c("KM Plot", "Demographic table"),
custom_variable = list()
)
)
As described in the summary, initial expectation was about having filter-flag but this can be easily extended to multiple conditions/choices.
# I want to add custom-filter (including filter-flag):
- adult-male: adsl.gender == m & adsl.age >= 18
- adult-woman: adsl.gender == f & adsl.age >= 18
- minor
filters = list(
ADSL = list(
genderage = custom_filter(
label = "Gender and age category",
`adult-male` = gender == "m" & age >= 18,
`adult-woman` = gender == "f" & age >= 18,
`minor` = age < 18
),
adult = custom_filter(
label = "Adulthood",
`is adult` = age >= 18
)
)
)
Above only intitializes the app with the specific filters, but doesn't provide the set of labels possible to select from. It means we need a consistent way to specify a filterable as in (1). This can be illustrated with
filter = list(
"ITT" = list(), # default selection
"AP" = list(), # default selectio
filterable(read_citrix_labels("path/to.yaml")) # possible labels
)
# I want gender filter to be fixed (not changeable):
filters = list(
ADSL = list(
gender = list(c("M", "F"), fixed = TRUE)
)
)
# I want gender filter to be fixed (not removable):
filters = list(
ADSL = list(
gender = list(c("M", "F"), multiple = FALSE)
)
)
UI is obvious here radioButton/selectizeInput - illustration not provided.
# I want gender filter to be fixed (not removable):
filters = list(
ADSL = list(
gender = list(c("M", "F"), permanent = TRUE)
)
)
Another things to consider to include in API:
Not related to API directly is to:
make a filter-state cards reactive to changes in the data. So the counts will be updated when data changes.
Make a proper UI design for all scenarios to make filter-panel visually attractive
WIP: I'll continue in this post with some UML focused on backend.
Key implementation questions:
-
I'm not a fan of having herarchical-filter which seems too compicated to me. Why should we change the counts in the specific order (filter1 > filter2 > filter3 > ...) while we can update filter-counts everytime when something changes. Counts can be updated in filter1 when filter4 is changes end so on. This could be confusing for the other users than these supporting hierarchy. If it's only about seeing counts I think we might need some other tools to do this, maybe "count tree module"?
@gogonzo it's not just the counts - more importantly it's the ranges
If you have 2 biomarkers X and Y with the range of values of X in [0, 1] and the range of Y in [100, 200] then if you want to filter on only X values > 0.5 then at the moment you filter only on biomarker X but then the range filter still shows you a range of [0, 200] which is very unhelpful - this is the reason why goshawk has its own filtering in the encoding panel I believe @npaszty ?)
Maybe a special "filter" option (hierarchical or I prefer Tableau's name context) to allow a couple of linked variables to be filtered together to handle ranges nicely? Without everything being in a complex hierarchy? This concept seems to be called "context filters" https://help.tableau.com/current/pro/desktop/en-us/filtering_context.htm which I think I prefer (and the general behaviour of filters changing is also called "cascading filters" in other places)
In some sense the "custom filter" is a "filter-group" and it's almost as if we could let people take existing filters and group them into one filter which can be turned on/off rather than having to get people to specify filters in code and us to parse it?
Any thoughts on how to get the filter information back to ADSL for filtering for subgroups? It sounds like that is quite an important piece of functionality ( @kumamiao ?) - even if it's not implemented now we should make sure it could be implemented in future?
re: hierarchical filter, I think the issue is that currently in our filter panel, the filtering statistics (counts, ranges, etc.) cannot reflect the last filtered data, but only the original unfiltered data, which could be confusing and make it hard to fulfill some needs (i.e., accurate AVAL range based on certain filtered PARAM in ADLB). The concept of the cascading/context filters sound good, will also take a look at some Spotfire examples.
Any thoughts on how to get the filter information back to ADSL for filtering for subgroups? It sounds like that is quite an important piece of functionality ( @kumamiao ?) - even if it's not implemented now we should make sure it could be implemented in future?
User request is reasonable given the analysis scenarios (i.e., subgroups who have a certain AE [based on ADAE], subgroups who have abnormal glucose [based on ADLB], etc.) and is marked to be urgent, although I am also debating on the generalization vs. clinical use for this feature. My initial thought is when users apply filters for non-parent data, there will be a checkbox next to the data name, for users to choose whether to create a subgroup. If so, a new flag will be created in the parent data (user can fill in flag name and filter label via a pop-up window) if the the primary keys in the parent data also exist in the filtered non-parent data. Happy to hear other possibilities as well.
Another thought re: the subgroup, I agree with Pawel as this subgroup creation (with relationship to other non-parent data) can complicate the filter panel. What if there is a way to create a separate module just for the purpose of the subgroup creation/data exploration, in that case we can leave the filter panel to be cleaner, and more generalizable to wider adoption beyond clinical trials.
@nikolas-burkoff
the goshawk screening and baseline data constrain filtering is very specific. it allows users to filter out subjects who have baseline values, for example, between a specifically set range. this can't be done effectively with current filter design. if paramcd was filtered to let's say ALT then it would be very difficult if not impossible to filter base because the slider would include the range of all biomarkers. so along the lines of what was noted above but with a much broader range.
no paramcd filtering
filtered to one paramcd and no change in base range.
a much more simple concept is if you filter on gender "F" for a demographics subset then you don't want to see data in the filters that belong to "M". SAS viewtable creates hierachical filtering and to me that's what you would expect from a data filtering.
had follow up discussion on dynamically building UI elements into the encoding panel. this could be done using a couple of additional arguments to the modules. one argument to indicate if dynamic UI element code should execute (logical) and the other being a list of lists instruction. list element would be type of ui and values. so list(radio, c("Ocular AESIs = aesi", "Ocular Serious AEs")), etc.
this would need to be implemented via a renderUI in the server function along with a code block to do the filtering of the reactive data. found this article on dynamic UI elements in shiny. not exact match but concept indicates this would work.
with that said a little bird told me... the proper way to filter data is to enable filter label based filtering in the filter panel. For example if the filter label ITT stands for the filter (subset) operation ADSL$ITTFL == "Y" and AESER stands for ADAE$SERIOUSAE == "Y" then one could have a select box where the user can select different filter labels, e.g. ITT and AESER and the resulting subset expression would be ADSL$ITTFL == "Y" & ADAE$SERIOUSAE == "Y" this would match the current SPA thinking of subsetting data
@kumamiao and I discussed how the slider should behave when ranges recalculated during hierarchical filtering activity.
For example:
For this scenario, we think it makes sense to reset the filter slider range. We also suggest that this could always the default behavior for ranges/slider related for recalculating ranges.
Linking https://github.com/insightsengineering/teal.slice/issues/102 We want to expose FilterStates object from teal.slice so that user can extend it in their own package.
@kumamiao and I discussed how the slider should behave when ranges recalculated during hierarchical filtering activity.
For example:
- We have demographic data with AGE overall ranges from 1-50
- User filters AGE and updates the filter slider range to 10(min)-20(max)
- User makes other filter changes which affects the AGE overall ranges to 30-50.
- Ranges recalculated, but what would happen to slider on step 2 since the min/max is now out of range?
For this scenario, we think it makes sense to reset the filter slider range. We also suggest that this could always the default behavior for ranges/slider related for recalculating ranges.
This would lead to recalculation of each active FilterState. Because each FilterState (selected) value will be updated it will cause data filtering n-times. It means that if we have 5 filters, changing one FS can trigger recalculation of the module 5 times. To sequential retriggering we can apply some caching comparison mechanism, but nothing is for free (cache will affect size of the app). This is why I suggested to update counts only (density chart in case of slider).
I think there's been talk recently @mhallal1 @pawelru about having the option for much more locked down filter panels as certain use cases may require this
I think there's been talk recently @mhallal1 @pawelru about having the option for much more locked down filter panels as certain use cases may require this
Let's don't bring that up yet. This discussion is still very much up in the air. Don't want to introduce some another level of complexity now
Note from collaboration:
Currently fiilter panel works in the way depicted on the diagram below. Filters are added by the user or through the API by add_filter_variable
. FilterState
is created from the selected column of a data.frame and populated to the ReactiveQueue
. Based on the changes in ReactiveQueue
different filter-call is evaluated (and thus different filtered data). All FilterState
in the ReactiveQueue
are shared by all modules and imediately applied to them all.
New proposition is slighty different. ReactivseQueue
where active filters are kept needs a mapping interface to match the slot (module) with it's filters. The mapping system enables that modules will be able to use different set of filters. Because of the new mapping system UX/UI, API need necessary changes which will affect significant code refactor (but not rewriting from scratch)
Before going further, it's important to distinguish few terms:
Filter types:
Comming soon...
I'd like to present the UX prototype (first iteration). The prototype focuses on the interaction with the filter cards, filter manager, changing labels, and adding a new filter. Please see the wireframe with entire prototype
Please see interactive wireframe. Enabled filters are displayed in the filter-panel as "non-interactive" cards presenting summary of applied labeled filters. Clicking "gear" icon (or card itself) will show the interactive control of the filter. One can change the selected values and assign the label to the filter. By default labels are following a selection, so that:
$varname: $min-$max
$varname: $choice[1], ..., $choice[n]
cut on some character limits.
When user assigns the value to the filter it no longer follows the selection.
Icon x
on the filter-card removes the filter from the slot (module).Default | Interactive |
---|---|
![]() |
![]() |
In the module one can apply or unapply filters. By clicking "plus" on the top of the filter-panel menu should show up where one can select/deselect labeled filters for this module. One can also disable filters for the current module by clicking switch on the bottom of the dropdown.
Default | Enable/disable filters |
---|---|
![]() |
![]() |
Opened by clicking "gear" icon on the top of the filter-panel. New modal will pop up
If you look on the list in "Enable/disable filters " (2), you might ask "Ok, how to apply filter outside of the list". To do this, app user has to first add a labeled filter from the filterable (column). There are two dropdowns, first to select a dataset and second to select a filter. In the second dropdown one can select multiple options:
Selecting two values automatically create a group. Please see the wireframe
Default | Select filters to add | Save choice |
---|---|---|
![]() |
![]() |
![]() |
After clicking save filter is saved to the list of labeled filters and then can be applied to the model. Notes:
Similar to the "Filter cards" (1), here one can also select values of the labeled filter, change the label. But also can change other properties of the filter like:
multiple
in the selectizeInput
)Similar to (3.2.1) but on a group level. One can change label and selected values the labeled group and additional properties of labeled components.
In progress. To have a control over the slot/filters map. Possibly drag and drop or a checkbox matrix
@gogonzo looks very comprehensive! but also quite complex. Does this proposal include a an optional "low-complexity" filter version, that the app developer can activate?
@danielinteractive many things can be controlled by the app developer when initializing FP. I'm happy to continue discussions especially regarding MAE objects
When setting which filters can be applied to which modules, I would consider an option to not allow a given filter to apply to certain modules. Let's say I as an app developer do not want users to be able to apply the Age 20-60 filter for the demographic table module. Then I would like a way to specify that so users are prevented from applying the Age 20-60 filter to the demographic table.
Thanks @asbates , good point. Result of such configuration could look like this in the filter manager. Where some filters will be disabled to apply them in the module
I'm moving discussions to the document here Let's update this issue if we conclude on something because comments here can make this messy.
@donyunardi I'm happy to close this one ;)
Please provide the balsamic