Re-design with crossfilter?

[will-moore]

Having had a look at the crossfilter docs, some thoughts on what adoption of crossfilter might give us:

Current issues:

• Would be nice to get rid of eval() for filter functions
• Loading a whole table of data: only use 1 column at a time. Need to load table again to filter by another column.
• Have to load the table data for filter AND again if we want to show in table.
• Don't handle multiple tables attached to parent Project / Screen.
• Don't handle text columns for OMERO.tables
• Don't handle numerical values for Map annotations

Crossfilter provides lots of nice features, but how would it fit into parade?

• Assume we don't need current 'parade filter' eval() functionality, and we simply use 'dataprovider' to load a table (all, or selected columns), which is what crossfilter expects.
• Init crossfilter with a table of data. Then allow the user to choose dimensions to filter by. Less-than, greater-than, equal or range for numerical columns, equal/contains for string columns.
• May want to group-by Dataset, Tags or various table columns.
• After crossfilter init (and create dimension etc) is it possible to load additional data for filtering or display? Not as far as I can see. Might need to re-init crossfilter and re-create dimensions and filters.
• Might want to plot data with https://dc-js.github.io/dc.js/ which seems to give a lot of nice features out of the box. How does user choose what to plot?
• Don't know how well dc.js works with React.

cc @chris-allan @emilroz

[emilroz]

@will-moore we probably need a longer discussion on when and how to use the crossfilter. It's a great tool but has few limits. E.g. looking at dimensions: https://github.com/square/crossfilter/wiki/API-Reference#dimension: Creating more than 8 dimensions, and more than 16 dimensions, introduces additional overhead. More than 32 dimensions at once is not currently supported, but dimensions may be disposed of using dimension.dispose to make room for new dimensions.... Do we actually need more than 32 dimensions? Probably not, but that's a thing to take into consideration.

For dc.js we would have to build a user interface to actually do the plotting. We could in principle remove existing plot and add a dc.js plot instead. I have not tried, and do not know if possible, to connect dc.js to existing crossfilter yet. The benefit of doing it would be of course the fact that the plot selection would also filter the data on the grid and in the table. Happy to discuss further.

[will-moore]

Been working with crossfilter on a prototype at https://github.com/will-moore/parade-crossfilter and it's looking good in many ways. Still a few questions to work out. Would be good to give a demo and discuss sometime.

[will-moore]

Demo video of the "parade-crossfilter" app is at https://youtu.be/FyjGhZxx6es Feedback appreciated!