Metrics for analyzing weight data

[kdahlquist]

Our task for the next week is to brainstorm a list of comparisons or metrics that we want to do with the model data to help us understand the results of the modeling. What we came up with in the meeting already is listed here. Please add to this list by commenting on this issue. We will prioritize and assign tasks at next week's meeting.

• What is the relationship of the MSE to the within-strain ANOVA p value for each gene.
  • does p < 0.05 give a better fit?
  • does p > 0.05 give a better fit?
  • is there no relationship?
• Are genes with no inputs modeled worse or better or no relationship?
  • use in-degree, out-degree, total-degree tables for this
• do genes with decreases in expression have a worse fit?

Note that the above MSE stuff has been moved to issue #326

• Descriptive statistics of optimized weights:
  • sum
  • mean
  • max/min
  • median
• Compare weight values for edges that are the same between networks

Tasks below moved to #331

• Do something with the distance matrix, @maggie-oneil generated one already for the dHAP4 network
  • can this be made computationally instead of by hand
  • what happens if we attach weight values to it

[kdahlquist]

Reposting @maggie-oneil 's comment here:

Distance-Matrix_From-Unweighted-Adjacency-15-genes_28-edges_GJ-dHAP4-fam_strains-added_Sigmoid_Estimation.xlsx

Here's the distance matrix. Might need to be looked at again to confirm correct values

[bklein7]

In preparation for the LMU URS, I began analyzing edge weights based on the modeling results from the six database-derived networks. This work aims to satisfy the points listed above: generating descriptive statistics of optimized weights, comparing weight values for edges that are the same between networks. The spreadsheet that I have been working with thus far can be found here: https://github.com/kdahlquist/DahlquistLab/blob/master/data/15-gene_networks_analysis/Regulatory-Relationships_Six-Networks_BK20170206.xlsx.

@kdahlquist and I discussed the following ways to further analyze this data:

• Generate a heat map comparing the weight values for edges that are shared between networks. Also consider methods for clustering similar regulatory relationships.
• Generate distributions of weighted values for each network, and assess how the values that GRNsight grays out compare to the distributions.
• Look at the sums of the weighted values for all the edges associated with singular regulators.
• Assess how the weights from each network are normalized in GRNsight and consider normalizing them collectively in the spreadsheet.

[bklein7]

Based on the commentary provided during last week's meeting, I expanded the regulatory weight analysis discussed above. Further statistics assessing the properties of weights within each network have been added to this spreadsheet: https://github.com/kdahlquist/DahlquistLab/blob/master/data/15-gene_networks_analysis/Regulatory-Relationships_Six-Networks_BK20170206.xlsx.

In addition, an analysis of the distribution of normalized weight values in each network using histograms can be found here: https://github.com/kdahlquist/DahlquistLab/blob/master/documents/Weight-Distributions_Six-Networks_BK2017213.pptx.

[kdahlquist]

This issue has been renamed to just be about the analysis of the weights that @bklein7 has been doing and the other ideas have been moved off to separate issues: #326, #328, #329, #330, #331.

Notes from Thursday's meeting:

• think about how to make the spreadsheet usable for others to that other people can plug in data and not have to redo equations or formatting.
  • for example, the number activating and percent are in same cell, separate to separate cells
  • rename networks db1, db2, rand1, etc.
• make spreadsheet easier to use for SPSS
• in terms of the histograms, figure out how to fix the bin width; may want to have 10 bins instead of 20 bins
• @bengfitzpatrick suggested that what we may really want is to plot the cumulative instead of the histogram
• maybe for now, get rid of the grey bars and use only red and blue. I'll make a note on the GRNsight issue that the user may want to change the threshold for coloring an edge grey (which is what this is based on).

[bklein7]

Based on the feedback provided during last week's meeting, the weights analysis spreadsheet has been updated: https://github.com/kdahlquist/DahlquistLab/blob/master/data/15-gene_networks_analysis/Regulatory-Relationships_Six-Networks_BK20170206.xlsx. The new version automates the statistical analysis and color coding of regulatory weights. However, it is currently only designed to support data from six different networks. This can be expanded moving forward. Further, once a data analysis protocol for these weights has been established, a second tab can be formatted to facilitate importing data into SPSS (if necessary).

Further, the weight distribution analyses were updated with improved histograms and cumulative plots: https://github.com/kdahlquist/DahlquistLab/blob/master/documents/Weight-Distributions_Six-Networks_BK2017213.pptx.

[kdahlquist]

Notes from the 2/23 meeting:

• For the weights analysis spreadsheet, 6 networks is fine for now. 10 might be a good round number, but it's not worth messing around with for now.
• For the histograms, please increase the bin size to 20 now that it's on a scale of 0-100. This will clean up the "hockey player" look to it (thanks to @bengfitzpatrick for that analogy).
• For the cumulative plots, make them a curve instead of bar chart and make them "proportional" (? can't read my handwriting! but I think you probably remember what this was)--something to do with kernel density

[bklein7]

Based on the feedback provided during last week's meeting, the weight distributions analysis has been updated: https://github.com/kdahlquist/DahlquistLab/blob/master/documents/Weight-Distributions_Six-Networks_BK2017213.pptx. This new version includes histograms with a bin size of 20 and cumulative percentage line graphs.

There are some issues with the new method I found in SPSS to plot cumulative percentage line graphs. Moving forward, it may be best to manually generate cumulative distribution functions for each set of weights and plot those. I investigated methods for doing so in SPSS but could not find a clear answer (in a reasonable amount of time). Thus, I will talk to @bengfitzpatrick further about this during the upcoming data analysis team meeting.

[kdahlquist]

To recap on this one:

• @bklein7 has made a generalizable Excel spreadsheet and computed descriptive stats for db1-6
• @Nwilli31 has used it for rand7, 12, 15, 16, 24, 31 (3 best and 3 worst)
• at some point we may want to complete all 30 random networks (but lower priority)
• We have degree distribution for unweighted networks for db1-6, but we need the randoms, at least the 3 best/3 worst
• @bklein7 has made weighted degree distribution charts and cumulative plots for db1-6, but needs to do random best/worst
• @bklein7 needs to document step-by-step instructions for how to do the weighted degree distribution charts and cumulative plots in SPSS for reproducibility

[kdahlquist]

We also need instructions for how to do the heat map in SPSS.

[bklein7]

Weighted degree distribution charts and cumulative plots for db5, RAND7, RAND12, RAND15, RAND16, RAND24, RAND31 ("parent" network + 3 best/worst random networks) have been compiled for comparison: https://github.com/kdahlquist/DahlquistLab/blob/master/documents/six-random-networks_weight-distribution-figures_BK20170405.pptx.

I will write instructions for creating the degree distribution charts, cumulative plots, and weights heat map. The latter was made in Excel using conditional formatting, so I could likely add a second tab to the generalized Excel spreadsheet (for analyzing weights) that could produce a heat map instead. What would be the best format for documenting the weighted degree distribution chart and cumulative plot protocols?

[kdahlquist]

Use the OWW wiki for documentation. You could either include it in your notebook or create a new protocol page under the DahlquistLab site. See one of the other protocols for how to format. http://www.openwetware.org/wiki/Dahlquist:Protocols

[kdahlquist]

• [x] write up protocol for creating weighted degree distribution and cumulative plots in SPSS
• [x] create tab in weights analysis spreadsheet for automation of heat map creation

[bklein7]

An updated version of the weights analysis spreadsheet was uploaded to the Dahlquist Lab repository: https://github.com/kdahlquist/DahlquistLab/blob/master/data/Spring2017/15-gene_networks_analysis/Regulatory-Relationships_Six-Networks_BK20170313.xlsx. This version automates normalization of input edge weights to the singular maximum weight value (all networks) in the "Normalized Values" tab. This data is then used to generate a heat map akin to the one presented at the 2017 LMU Research Symposium, which can be viewed in the "Heat Map" tab.

[bklein7]

The protocol for creating histograms and cumulative plots in SPSS to show the distribution of weighted regulatory weights in a network has been written: http://www.openwetware.org/wiki/Generating_Distribution_Charts_and_Cumulative_Plots_for_GRNmap_Weight_Values_in_SPSS. The link to this protocol has also been added to the 15-gene_networks_analysis folder's ReadMe file on the Dahlquist Lab repository: https://github.com/kdahlquist/DahlquistLab/tree/master/data/Spring2017/15-gene_networks_analysis.

[kdahlquist]

This looks like it has been completed to the state required at the end of last semester. We are going to end up re-doing some analyses with model runs on cleaned up GRNmap code. We can refer back here as needed when we need to run the analyses again.