This PR contains some refactors of the snakemake pipeline so that it works better when multiple sessions of recordings are analyzed together. Specifically, it contains the following the major changes
Instead of using a long relative time coordinate for pyphotometry data, the data is now extracted using trial and event time coordinate. It reduced the issues created by downsampling the relative time (depending on the averaging window, the downsampled coordinate may not be the same for different trials). Now raw data can be downsampled very easily using coarsen without the need to realign the relative time coordinates.
Example dataset format:
Trial specific information in df_condition is stored in its own variable with the trial_nb coordinate. This should avoid the need to duplicate those information for each time point. As we are now using trial_nb as one of the coordinate of the photometry data, when we convert the xarray dataset to dataframe, the variables from df_condition will be automatically expanded to each time point, making data plotting very easy.
trialexp/process/pycontrol/event_filters.py contains the function to extract timestamp of different events for aligning the photometry data
A new notebook is added to notebooks/workflow/multi_session_analysis.ipynb to show how to perform multi-session analysis on the pipeline output.
There is also some moving around of imports. So better test it on the existing pipeline first
Change in dependencies:
I have to bump the Python version to 3.9 because of new xarray requirement. Probably also need to reinstall snakehelper because there is some update upstream.
TODO
Directly averging df/f from multiple animals together probably won't work because they have different baseline of photometry signal. Probably need to use the z-score instead.
This PR contains some refactors of the snakemake pipeline so that it works better when multiple sessions of recordings are analyzed together. Specifically, it contains the following the major changes
coarsenwithout the need to realign the relative time coordinates.Example dataset format:
Trial specific information in
df_conditionis stored in its own variable with thetrial_nbcoordinate. This should avoid the need to duplicate those information for each time point. As we are now usingtrial_nbas one of the coordinate of the photometry data, when we convert the xarray dataset to dataframe, the variables fromdf_conditionwill be automatically expanded to each time point, making data plotting very easy.trialexp/process/pycontrol/event_filters.pycontains the function to extract timestamp of different events for aligning the photometry dataA new notebook is added to
notebooks/workflow/multi_session_analysis.ipynbto show how to perform multi-session analysis on the pipeline output.There is also some moving around of imports. So better test it on the existing pipeline first
Change in dependencies: I have to bump the Python version to 3.9 because of new xarray requirement. Probably also need to reinstall snakehelper because there is some update upstream.
TODO
Directly averging df/f from multiple animals together probably won't work because they have different baseline of photometry signal. Probably need to use the z-score instead.