denoise SSF experiments

djkapner commented 3 years ago

Copied below are ophys sessions for the SSF project that cover 160 experiments.

Tasks:

[x] Run these experiments through the deep denoiser and put the resulting movies on isilon somewhere useful. Probably /allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ since there is plenty of space there.
[x] Start a sqlite db with the metadata and these videos for the SSF project.

Validation:

[x] deep denoised hdf5 videos exist for these experiments.
[x] Document in this ticket the slurm script used to do this.
[x] Link to the sqlitedb or the script made to create it in this ticket.

djkapner commented 3 years ago

2-photon sessions, collected on DeepScope, each consists of 6 imaging planes

mouse: 409828; column 1; volume: 1; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_794202009
mouse: 409828; column 1; volume: 2; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_794875503
mouse: 409828; column 1; volume: 3; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_794964451
mouse: 409828; column 1; volume: 4; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_795660198
mouse: 409828; column 1; volume: 5; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_795771997
mouse: 409828; column 2; volume: 1; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_774328450
mouse: 409828; column 2; volume: 2; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_785941763
mouse: 409828; column 2; volume: 3; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_786071018
mouse: 409828; column 2; volume: 4; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_790940857
mouse: 409828; column 2; volume: 5; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_791865124
mouse: 409828; column 3; volume: 1; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_783110306
mouse: 409828; column 3; volume: 2; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_783251261
mouse: 409828; column 3; volume: 3; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_786879416
mouse: 409828; column 3; volume: 4; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_792629775
mouse: 409828; column 3; volume: 5; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_792686600
mouse: 409828; column 4; volume: 1; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_783878040
mouse: 409828; column 4; volume: 2; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_784057573
mouse: 409828; column 4; volume: 3; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_790009715
mouse: 409828; column 4; volume: 4; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_790027620
mouse: 409828; column 4; volume: 5; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_793340788
mouse: 409828; column 5; volume: 1; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_785378984
mouse: 409828; column 5; volume: 2; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_788220278
mouse: 409828; column 5; volume: 3; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_790585092
mouse: 409828; column 5; volume: 4; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_793463644
mouse: 409828; column 5; volume: 5; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_794083001

3-photon sessions, each is a single plane

mouse: 409828; column 1; volume: 6; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806855912
mouse: 409828; column 1; volume: 7; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_803610979
mouse: 409828; column 1; volume: 8; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_803948333
mouse: 409828; column 1; volume: 9; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_803948442
mouse: 409828; column 1; volume: 10; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_803948540
mouse: 409828; column 1; volume: 11; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806847679
mouse: 409828; column 1; volume: 12; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806787026
mouse: 409828; column 1; volume: 13; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806787714
mouse: 409828; column 1; volume: 14; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806848230
mouse: 409828; column 1; volume: 15; lims: \\allen\programs\braintv\production/neuralcoding\prod55\specimen_734689833\ophys_session_806855673

the 160 experiment IDs are:

[794298220, 794298209, 794298176, 794298165, 794298198, 794298187, 795011996, 795011974, 795012008, 795012030, 795012019, 795011985, 795018590, 795018579, 795018557, 795018534, 795018568, 795018546, 795897823, 795897812, 795897800, 795897778, 795897766, 795897789, 795901906, 795901884, 795901895, 795901873, 795901861, 795901850, 774392931, 774393041, 774393007, 774392982, 774392958, 774392903, 786334337, 786334394, 786334383, 786334372, 786334361, 786334350, 786400885, 786400874, 786400863, 786400852, 786400841, 786400830, 790986890, 790986879, 790986867, 790986856, 790986845, 790986834, 791920156, 791920214, 791920202, 791920191, 791920180, 791920168, 783605259, 783605247, 783605236, 783605225, 783605213, 783605202, 783623275, 783623264, 783623286, 783623253, 783623313, 783623302, 787473523, 787473478, 787473512, 787473500, 787473489, 787473465, 792765582, 792765569, 792765558, 792765546, 792765535, 792765524, 792757272, 792757295, 792757283, 792757260, 792757249, 792757238, 784388310, 784388343, 784388332, 784388299, 784388288, 784388321, 784370431, 784370419, 784370407, 784370396, 784370385, 784370374, 790122271, 790122304, 790122260, 790122316, 790122293, 790122282, 790118079, 790118138, 790118116, 790118105, 790118092, 790118127, 793618875, 793618849, 793618831, 793618818, 793618796, 793618773, 785569459, 785569447, 785569470, 785569436, 785569412, 785569423, 788422870, 788422859, 788422848, 788422836, 788422825, 788422814, 790624009, 790623985, 790623966, 790623944, 790623924, 790624031, 794123762, 794123807, 794123818, 794123785, 794123773, 794123796, 794288353, 794288320, 794288331, 794288307, 794288342, 794288368, 806862946, 803616717, 803959659, 803958257, 803965468, 806854201, 806847865, 806844147, 806856553, 806928824]

djkapner commented 3 years ago

57 of these had already been denoised over the summer. For the rest of them, this slurm script:

#!/bin/bash
#SBATCH --job-name=ophys-deepint-inference
#SBATCH --mail-type=NONE
#SBATCH --ntasks=24
#SBATCH --mem=140gb
#SBATCH --time=48:00:00
#SBATCH --output=/allen/aibs/informatics/danielk/deepinterpolation/logs/inference_%A-%a.log
#SBATCH --partition braintv
#SBATCH --array=0-102

image=docker://alleninstitutepika/deepinterpolation:main
readarray -t injs < /allen/aibs/informatics/danielk/deepinterpolation/ssf_experiments.txt
input_json=${injs[${SLURM_ARRAY_TASK_ID}]}

export TMPDIR=/scratch/fast/${SLURM_JOB_ID}
echo "$TMPDIR"
export SINGULARITY_CACHEDIR=/allen/ai/hpc/singularity/cache/danielk

SINGULARITY_TMPDIR=${TMPDIR} singularity run \
    --bind /allen:/allen,${TMPDIR}:/tmp \
    ${image} python -m \
        deepinterpolation.cli.inference \
        --input_json ${input_json}

where /allen/aibs/informatics/danielk/deepinterpolation/ssf_experiments.txt is a list of input jsons like:

/allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ophys_experiment_795011996/inference_input.json
/allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ophys_experiment_795012008/inference_input.json
/allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ophys_experiment_795897800/inference_input.json
/allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ophys_experiment_795901895/inference_input.json
...

and one of those input jsons looks like below (this could improve with further refinements to the CLI for deepinterpolation):

{
  "generator_params": {
    "name": "OphysGenerator",
    "start_frame": 0, 
    "train_path": "/allen/programs/braintv/production/neuralcoding/prod55/specimen_734689833/ophys_session_785941763/ophys_experiment_786334372/processed/motion_corrected_video.h5"
  },
  "inference_params": {
    "model_source": {
      "local_path": "/allen/aibs/informatics/danielk/deepinterpolation/output/ophys_experiment_958741232/processed/2021_04_06_14_15_transfer_train__mean_squared_error_transfer_model.h5"
    },
    "output_file": "/allen/programs/braintv/workgroups/nc-ophys/danielk/deepinterpolation/experiments/ophys_experiment_786334372/denoised.h5"
  }
}

djkapner commented 2 years ago

The denoised movies for these 160 experiments and preview webms exist and are collected in this database file:

/allen/aibs/informatics/segmentation_eval_dbs/ssf_mouse_id_409828.db

djkapner commented 2 years ago

The webm previews show a side-by-side before/after denoising. I reused the code for making motion correction previews in an un-merged branch of ophys_etl:

#!/bin/bash
#SBATCH --job-name=deep_webm
#SBATCH --mail-type=NONE
#SBATCH --ntasks=4
#SBATCH --mem=80gb
#SBATCH --time=3:00:00
#SBATCH --output=/allen/aibs/informatics/danielk/deepinterpolation/logs/deep_webm_%A-%a.log
#SBATCH --partition braintv
#SBATCH --array=4-5

image=docker://alleninstitutepika/ophys_etl_pipelines:19a1fa639df29106ecbee1e1abbd31eb803f4eab
readarray -t injs < /allen/aibs/informatics/danielk/deepinterpolation/preview_input_jsons.txt
input_json=${injs[${SLURM_ARRAY_TASK_ID}]}

TMPDIR=/scratch/fast/${SLURM_JOB_ID}

SINGULARITY_TMPDIR=${TMPDIR} singularity run \
    --bind /allen:/allen,${TMPDIR}:/tmp \
    ${image} \
    /envs/suite2p/bin/python -m ophys_etl.qc.registration_qc \
      --input_json ${input_json} \
      --log_level INFO \
      --no_downsample False \
      --preview_frame_bin_seconds 2.0

AllenInstitute / ophys_etl_pipelines

denoise SSF experiments #323