add code to build dff stats table

[dougollerenshaw]

• adds a function to data_access.processing called add_dff_stats_to_specimen_table that takes the session object as the input and returns a cell_specimen_table with statistics on dff_traces appended (stats are provided by pd.describe() and include mean, std, 25%, 50%,75%, min, max)
• adds a script called log_dff_stats.py that will take an ophys_experiment_id as an input, call add_dff_stats_to_specimen_table, then log the resulting table to a mongo collection called ['ophys_data']['dff_summary']
• adds a script called deploy_log_dff_stats.py that will iterate over the filtered_ophys_experiment_table and create one cluster job per experiment to execute log_dff_stats.py
• adds a function to database.py called log_cell_dff_data that handles writing of a single dff_stats record to mongo
• adds a function to database.py called get_cell_dff_data that facilitates searching mongo for the dff_stats

[dougollerenshaw]

Here is the syntax for retrieving data for all cells from the database:

import visual_behavior.database as db
dff_summary = db.get_cell_dff_data()

That command takes ~1.5 seconds to execute. Here is a 5 row sample of the resulting datafame:

dff_summary.sample(5)

Note that the _id column is the random hash that mongo has assigned to the entry to ensure uniqueness.

Data from a specific experiment can be queried as follows:

import visual_behavior.database as db
dff_summary = db.get_cell_dff_data({'ophys_experiment_id':994056251})

Records can also be queried numerically using the $gt or $lt operators for greater-than or less-than comparisons, respectively. For example, to find all cells with standard deviations greater than 1:

import visual_behavior.database as db
dff_summary = db.get_cell_dff_data({"std": {"$gt": 1}})

[dougollerenshaw]

Also note that I have created a column called 'previous_cell_specimen_ids'. This is intended to track changes in cell_specimen_id if/when cell-matching is re-run.

[DowntonCrabby]

I would either make the column names more explicate for things like "id", 'x", "y" or include descriptions of the columns in the documentation docstrings.

[alexpiet]

What does the "count" column contain?

[dougollerenshaw]

@DowntonCrabby See additional documentation.