Implement interactive reporting dashboard

eboileau commented 1 year ago

The app part 1 mostly relies on pre-processed data (call to summarize-rpbp-profile-construction), such that it quickly does the plotting and minor data wrangling.

For app-specific related problems/improvements, I will open separate issues, where relevant. This still needs to be integrated.

Here is a non-exhaustive list of things to do/consider

[X] We need two apps, one for the pre-processing (part 1), and one for the ORF prediction output (part 2, WIP).
[x] #136
[X] Remove old reporting scripts, and replace them with scripts to generate input data (WIP).
[X] Harmonize nomenclature across outputs to facilitate handling.
[X] Update documentation, in particular for QC and downstream analysis.

still WIP, but this should be done soon....

[x] The conda environment sets the HOST variable to some installation path, see e.g. https://github.com/conda/conda/issues/7031 or https://github.com/plotly/dash/issues/1480

eboileau commented 1 year ago

For the app part 1:

[X] How to set default value for HoverData in funnel_fig based on selected/available samples?
[X] ylabel for selected read lengths in read_length_metagene_bars_fig is out
[X] all_metagene_fig: maybe we should swap the windows for the end profiles?
[X] There are missing profiles for selected lengths (TODO: open another issue)

I need to have a closer look at this...

[X] Group bars by samples if replicates/conditions

If traces have arrays, then the axis type is automatically inferred to be multicategory. I guess for a DataFrame with MultiIndex this might work, but this is not a major issue, we can address that later.

[X] Metagene profiles (at least global profiles): try using ORF profiles instead? That would allow to subset by ORF type.

Long term.

[X] change color palettes to colorblind-friendly palettes?

eboileau commented 1 year ago

For the app part 2:

[X] Can we implement hover for Circos plot?

We can, but the output is not synchronizing well, e.g. hover position for the lower circle appears far away, and this seem to depend on the position of the figure in the page... same problem appears on the circos Dash gallery for the histogram. I suggest we leave it out, the overall diagram is more important than the actual size of the bars.

[X] Tooltip in main table does not wrap.

I had to reformat the output for markdown. This works now, but if the transcript names are too long, we might have the same problem, we can adjust after testing on e.g. human data.

[X] Custom fields for BED files (IGV) are currently not supported. List as long-term (low priority).
[X] Is it possible to subset a table -> export as csv?

This has taken some time to figure out... that's because we're paginating the table in the back end. To get the whole filtered table, we'd have to either let the front end handle pagination (no), or implement a callback with the same filtering, and making sure it does not fire unless the download button is triggered (yes).

[X] Implement search by ORF ID on the main table.

I've actually implement filtering on the whole table, but the functionality is limited by the search syntax and the column types, but I think this is doing the job until we can think of something more flexible/robust.

[X] It could be a nice feature to add links to genes/transcripts, but we need to think how to do that in a "annotation-independent" way, e.g. don't link to Ensembl is the starting GTF is not Ensembl. Using gene name? Link to which resource?

Long term.

[X] I noticed strange behaviour in multi-column sorting the main table: needs testing.
[x] IGV does not load... when testing with c-elegans, this was not a problem, but in general this is... we already use indexed FASTA... the GTF track has no index, but I doubt this is the bottleneck...
[X] Circos plot ordering: double-check if the order of both histogram and genome matters. In addition, chroms are shown in the same order they were defined, alphanumerically (this depends on "chrNameLength.txt"). Shall we customize ordering?

Leave for now, we can add this to a minor/patch release in the future

[x] Add extra column in ORF table when standardized ORFs are present.

eboileau commented 1 year ago

[ ] Sometimes we get Callback error updating stacked_frames_fig.figure, but this is not consistently reproducible....

Traceback (most recent call last):
  File "/beegfs/prj/rpbp-dev/bullseye/rpbp_cmdstanpy/rp-bp/src/rpbp/analysis/profile_construction/dashboard/rpbp_profile_construction_dashboard.py", line 1123, in stacked_frames
    fig = px.bar(
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/express/_chart_types.py", line 373, in bar
    return make_figure(
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/express/_core.py", line 1988, in make_figure
    apply_default_cascade(args)
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/express/_core.py", line 969, in apply_default_cascade
    args["pattern_shape_sequence"] = [
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/express/_core.py", line 970, in <listcomp>
    bar.marker.pattern.shape for bar in args["template"].data.bar
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/graph_objs/bar/marker/_pattern.py", line 252, in shape
    return self["shape"]
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4733, in __getitem__
    elif self._props is not None and prop in self._props:
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4428, in _props
    return self.parent._get_child_props(self)
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4442, in _get_child_props
    if self._props is None:
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4428, in _props
    return self.parent._get_child_props(self)
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4442, in _get_child_props
    if self._props is None:
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4428, in _props
    return self.parent._get_child_props(self)
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 4462, in _get_child_props
    child_ind = BaseFigure._index_is(children, child)
  File "/beegfs/prj/rpbp-dev/bullseye/envs/rpbp-cmdstan/lib/python3.10/site-packages/plotly/basedatatypes.py", line 3960, in _index_is
    raise ValueError("Invalid value")
ValueError: Invalid value

eboileau commented 1 year ago

I really don't know where the above error if coming from... it's not consistently reproducible... I guess we leave it for now, and keep an eye on this...

dieterich-lab / rp-bp

Implement interactive reporting dashboard #131