Facemap hanging on time computers

[cbimbo]

Automatic facemap is hanging when automatically run.

[Timothysit]

Hi, I made sure the timeline computers pulled the latest code last night (2022-10-03), and I modified the facemap code so they only ran one video (just to have a minimal example of whether it is being called properly, and test what is causing the hanging with some print statements), overall I don't see the hanging any more: either facemap ran successfully or it returned an error in which case the matlab instance is still open but not running any code.

More specifically (I may open separate issues to address these)

time1 computer: restarted, I don't know why time2 computer: processed a single movie without issue time3 computer: seems like it encountered a video-causing error that I haven't seen before, the video opens fine using VLC so I will have to look into why facemap cannot process it:

The error is:

\zaru.cortexlab.net\Subjects\AV020\2022-08-04\7\2022-08-04_7_AV020_eyeCam.mj2 \zaru.cortexlab.net\Subjects\AV020\2022-08-04\7\2022-08-04_7_AV020_frontCam.mj2 \zaru.cortexlab.net\Subjects\AV020\2022-08-04\7\2022-08-04_7_AV020_sideCam.mj2 eyeCam not processed yet, will run facemap on it now Computing subsampled mean... img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. Computed subsampled mean at 3.64s Computing subsampled SVD... motSVD True movSVD False img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. img load failed, replacing with prev.. Traceback (most recent call last): File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1830, in main(subject = 'all',expDate = 'last100') File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1817, in main recompute_facemap=recompute_facemap) File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1532, in batch_process_facemap run_facemap_mod([[video_fpath]], proc=proc) File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 700, in run_facemap_mod sbin=sbin, rois=rois, fullSVD=fullSVD) File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 362, in compute_SVD usv = utils.svdecon(lilbin.T, k=ncb) File "C:\Users\Experiment.conda\envs\facemap\lib\site-packages\facemap\utils.py", line 378, in svdecon Sv, U = eigsh(COV, k = k, v0=v0) File "C:\Users\Experiment.conda\envs\facemap\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 1690, in eigsh params.iterate() File "C:\Users\Experiment.conda\envs\facemap\lib\site-packages\scipy\sparse\linalg\eigen\arpack\arpack.py", line 572, in iterate raise ArpackError(self.info, infodict=self.iterate_infodict) scipy.sparse.linalg.eigen.arpack.arpack.ArpackError: ARPACK error 3: No shifts could be applied during a cycle of the Implicitly restarted Arnoldi iteration. One possibility is to increase the size of NCV relative to NEV. 4f6e: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x801f: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x4267db59: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x4263dc88: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x425ddde6: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x4254dfdb: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x42476449: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x42157667: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x41e884a7: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x41a51b74: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x413fb934: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x40a7ecfe: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x3fc437df: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x3e6ea8fd: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x3c6e5c15: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x396e1a13: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x34ed822f: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x2e2c0d95: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x240a30e4: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x14d915f4: partial file [mov,mp4,m4a,3gp,3g2,mj2 @ 000002121066a380] stream 0, offset 0x801f: partial file

time4 computer:

there are two errors here: (1) another computer marked a video as "processed" after seeing the facemap output file before this computer had the chance, and so it couldn't find the "processing" file, this I can fix by asking the original computer to just not look for it after seeing the "processed" txt file. (2) There's this "Failed to add docroot into connector static content", which this post here suggest it's some firewall/antivirus issue, but I don't think it affected how facemap runs, and if nothing runs downstream of the facemap code then it doesn't affect the autorunovernight process. Perhaps it is caused by having PyCharm open at the same time (that's the only difference I noticed between time4 and the other timeline computers)

Full error message: 3 Candidate videos to look over \zaru.cortexlab.net\Subjects\AV020\2022-08-04\8\2022-08-04_8_AV020_eyeCam.mj2 \zaru.cortexlab.net\Subjects\AV020\2022-08-04\8\2022-08-04_8_AV020_frontCam.mj2 \zaru.cortexlab.net\Subjects\AV020\2022-08-04\8\2022-08-04_8_AV020_sideCam.mj2 frontCam not processed yet, will run facemap on it now Computing subsampled mean... Computed subsampled mean at 3.92s Computing subsampled SVD... motSVD True movSVD False Computed subsampled SVD at 1613.58s Computing projection for motSVD... Computed motSVD projection at 1900.58s run time 1913.39s Traceback (most recent call last): File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1830, in main(subject = 'all',expDate = 'last100') File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1817, in main recompute_facemap=recompute_facemap) File "C:\Users\Experiment\Documents\Github\PinkRigs\Analysis\+vidproc_facemap\run_facemap.py", line 1540, in batch_process_facemap os.rename(processing_facemap_txt_file, processed_txt_file_name) FileNotFoundError: [WinError 2] The system cannot find the file specified: '\\zaru.cortexlab.net\Subjects\AV020\2022-08-04\8\2022-10-03-20-00-46_frontCam_processing.txt' -> '\\zaru.cortexlab.net\Subjects\AV020\2022-08-04\8\2022-10-03-20-32-40_frontCam_processed.txt' [mov,mp4,m4a,3gp,3g2,mj2 @ 000002080d3d2600] moov atom not found [mov,mp4,m4a,3gp,3g2,mj2 @ 000002080c2c3540] error reading header

Exception "java.lang.ClassNotFoundException: com/intellij/codeInsight/editorActions/FoldingData"while constructing DataFlavor for: application/x-java-jvm-local-objectref; class=com.intellij.codeInsight.editorActions.FoldingData Exception "java.lang.ClassNotFoundException: com/intellij/codeInsight/editorActions/FoldingData"while constructing DataFlavor for: application/x-java-jvm-local-objectref; class=com.intellij.codeInsight.editorActions.FoldingData Failed to add docroot into connector static content