Full_Demo assumes existing summary.npy file

[Sihao]

When I run job.show_summary_plots() it assumes the summary.npy file exists in my job directory even after running job.run_init_pass() which was commented out.

[alihaydaroglu]

Sorry, the run_init_pass shouldn't have been commented out.

job.run_init_pass should create a summary.npy file in the job directory. Does it not show up after you run this?

[Sihao]

It says it created the file, but nothing is there

[Sihao]

This doesn't seem to actually save anything

[alihaydaroglu]

At the end of that function, it should save? It's possible you're running into an error. Could you share the log.txt file in the exp directory?

[Sihao]

[2023-07-18 14:28:01][00] Loading job directory for test_20230718 in ./
[2023-07-18 14:28:01][02]       Created dir ./s3d-test_20230718/registered_data
[2023-07-18 14:28:01][02]       Created dir ./s3d-test_20230718/summary
[2023-07-18 14:28:01][02]       Created dir ./s3d-test_20230718/iters
[2023-07-18 14:28:01][01]    Loading default params
[2023-07-18 14:28:01][02]       Updating param fs
[2023-07-18 14:28:01][02]       Updating param planes
[2023-07-18 14:28:01][02]       Updating param tau
[2023-07-18 14:28:01][01]    Updated main params file
[2023-07-18 14:29:52][01]    Saved a copy of params at ./s3d-test_20230718/registered_data
[2023-07-18 14:29:52][01]    Updated main params file
[2023-07-18 14:41:06][01]    Saved a copy of params at ./s3d-test_20230718/summary
[2023-07-18 14:41:06][01]    Updated main params file
[2023-07-18 14:41:06][00] Launching initial pass
[2023-07-18 14:41:06][00] Saving summary to ./s3d-test_20230718/summary/summary.npy
[2023-07-18 15:43:11][01]    Saved a copy of params at ./s3d-test_20230718/registered_data
[2023-07-18 15:43:11][01]    Updated main params file
[2023-07-18 15:43:21][01]    Saved a copy of params at ./s3d-test_20230718/summary
[2023-07-18 15:43:21][01]    Updated main params file
[2023-07-18 15:43:21][00] Launching initial pass
[2023-07-18 15:43:21][00] Saving summary to ./s3d-test_20230718/summary/summary.npy
[2023-07-18 15:56:52][01]    Found and loaded params from ./s3d-test_20230718/params.npy
[2023-07-18 15:56:52][01]    Saved a copy of params at ./s3d-test_20230718/summary
[2023-07-18 15:56:52][01]    Updated main params file
[2023-07-18 15:56:52][00] Launching initial pass
[2023-07-18 15:56:52][00] Saving summary to ./s3d-test_20230718/summary/summary.npy

Sorry, I hadn't noticed it outputted an error:

Traceback (most recent call last):
  File "/home/slu/pycharm-2022.2/plugins/python/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/slu/pycharm-2022.2/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/slu/PycharmProjects/s2p-lbm/notebooks/Full_Demo.py", line 75, in <module>
    job.run_init_pass()
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/job.py", line 252, in run_init_pass
    init_pass.run_init_pass(self)
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/init_pass.py", line 107, in run_init_pass
    assert nt > params['init_n_frames'], 'not enough frames in loaded tifs'
AssertionError: not enough frames in loaded tifs
python-BaseException

[alihaydaroglu]

OK, I guess you have a different number of frames per tif than I expected. Can you set that parameter to None? I'll update the defaults so this is the case for all datasets

[alihaydaroglu]

Just to clarify, this should do it:

job.params['init_n_frames'] = None
job.save_params()
job.run_init_pass()

[alihaydaroglu]

Setting init_n_frames to None uses all of the frames available in the number of initialization files specified in n_init_files, so assuming your files are a few 100s of frames (and not thousands) that should run OK. job.save_params() will update the saved params.npy file so if you load this job again you can retrieve the correct params. Most functions (e.g. job.calculate_corr_map) automatically save the params file, but I haven't added that to the initialization function.

[Sihao]

I get this:

Traceback (most recent call last):
  File "/home/slu/pycharm-2022.2/plugins/python/helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/slu/pycharm-2022.2/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/slu/PycharmProjects/s2p-lbm/notebooks/Full_Demo.py", line 75, in <module>
    job.run_init_pass()
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/job.py", line 252, in run_init_pass
    init_pass.run_init_pass(self)
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/init_pass.py", line 129, in run_init_pass
    __, __, cross_coeff = utils.calculate_crosstalk_coeff(im3d,
  File "/home/slu/PycharmProjects/s2p-lbm/suite2p/suite2p/suite3d/utils.py", line 135, in calculate_crosstalk_coeff
    axsins2.set_xticks([m_opt], rotation=0)
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/site-packages/matplotlib/axes/_base.py", line 74, in wrapper
    return get_method(self)(*args, **kwargs)
  File "/home/slu/anaconda3/envs/s2p-lbm/lib/python3.8/site-packages/matplotlib/axis.py", line 2072, in set_ticks
    raise ValueError('labels argument cannot be None when '
ValueError: labels argument cannot be None when kwargs are passed
python-BaseException

[Sihao]

Same error when I put a small number in (5)

[alihaydaroglu]

Ah this is a different issue. Can you pull the latest version with git pull --recurse-submodules?

I thought I fixed this error, this line which is causing the issue on your version is different in the latest version: https://github.com/alihaydaroglu/suite2p/blob/ac72290790a19c560b95f9ae4566ca66c1750264/suite2p/suite3d/utils.py#L137

[Sihao]

Ran the git command and still the same issue, though my local utils.py still seems to be different from the one in your main branch

[alihaydaroglu]

fixed, there was a submodule updating issue