bulk tracking (interactive)

[allenleetc]

There is a bug in the "external" capability/api of DeepTracker.track() when multiple movies/trx/frames are specified. See user report below

As in "Production tracking" this workflow does bulk tracking but with the tracking monitor available. Maybe add doc for this one, consider maintaining only one of these etc.

i think we found bug, but not where/how to fix it. each ssh cmd is sending the first movie, but the correct trxfile, trkfilenames. ssh -o "StrictHostKeyChecking no" login1.int.janelia.org 'source /etc/profile; /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/repo_snapshot.sh /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle > /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337.aptsnapshot; bsub -n 1 -gpu "num=1" -q gpu_any -o /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337_mov1.log -R"affinity[core(1)]" singularity exec --nv -B /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b -B /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/deepnet -B /groups/branson/home/robiea/Projects_data/Labeler_APT -B /groups/branson/home/robiea/Projects_data/Labeler_APT/Austin_labelerprojects_expandedbehaviors/GT  /misc/local/singularity/branson_cuda10_mayank.simg bash -c ". /opt/venv/bin/activate && python /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/deepnet/APT_interface.py -name 20190418T174652 -view 1 -cache /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b -err_file /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337_mov1.err -model_files /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/deepnet-60000 -type mdn /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/20190418T174652_20190418T174811.lbl track -mov /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_GMR_SS00168_CsChr_RigD_20150909T111218/movie.ufmf -out /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_GMR_SS00168_CsChr_RigD_20150909T111218/apt_mdn_04182019_test.trk -trx /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_GMR_SS00168_CsChr_RigD_20150909T111218/registered_trx.mat" </dev/null &'
Process job (view 1, mov 1) spawned, jobid=58334663.

Tracking job (movie 1, job 1) spawned:
ssh: /misc/local/matlab-2018a/bin/glnxa64/libcrypto.so.1.0.0: no version information available (required by ssh)
ssh: /misc/local/matlab-2018a/bin/glnxa64/libcrypto.so.1.0.0: no version information available (required by ssh)
Job <58334663> is submitted to queue <gpu_any>.

ssh -o "StrictHostKeyChecking no" login1.int.janelia.org 'source /etc/profile; /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/repo_snapshot.sh /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle > /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337.aptsnapshot; bsub -n 1 -gpu "num=1" -q gpu_any -o /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337_mov2.log -R"affinity[core(1)]" singularity exec --nv -B /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b -B /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/deepnet -B /groups/branson/home/robiea/Projects_data/Labeler_APT -B /groups/branson/home/robiea/Projects_data/Labeler_APT/Austin_labelerprojects_expandedbehaviors/GT  /misc/local/singularity/branson_cuda10_mayank.simg bash -c ". /opt/venv/bin/activate && python /groups/branson/home/robiea/Code_versioned/Labeler/APT_bundle/deepnet/APT_interface.py -name 20190418T174652 -view 1 -cache /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b -err_file /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/trk/movie_trn20190418T174652_iter60000_20190424T174337_mov2.err -model_files /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/mdn/view_0/20190418T174652/deepnet-60000 -type mdn /groups/branson/home/robiea/.apt/tpcebdd850_c6ec_4421_b543_202c7c01402b/multitarget_bubble_expandedbehavior_20180425_allGT_MK/20190418T174652_20190418T174811.lbl track -mov /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_GMR_SS00168_CsChr_RigD_20150909T111218/movie.ufmf -out /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_JHS_K_85321_CsChr_RigD_20150909T163219/apt_mdn_04182019_test.trk -trx /groups/branson/home/robiea/Projects_data/Labeler_APT/cx_JHS_K_85321_CsChr_RigD_20150909T163219/registered_trx.mat" </dev/null &'
Process job (view 1, mov 2) spawned, jobid=58334664.

[arobie]

I think these two features are very helpful and should both be possible if only one is kept. from tracker.track - track a list of movies - just track the whole movie short and sweet from trackListFile - just track the mov,frm,tgt combos you need tracker

[allenleetc]

see https://github.com/kristinbranson/APT/issues/263 for ongoing