issue in preprocessing step

[Mariofe25]

Hi,

I'm trying to implement this tool in my voltage imaging data analysis pipeline. After following the intallation steps i get the following error when I input the preprocessing command:

(cellmincer) C:\Users\mariofer>cellmincer preprocess -i noisy.tif -o ./noisy --manifest manifest_config.yaml --config preprocess_config.yaml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\mariofer\AppData\Local\anaconda3\envs\cellmincer\Scripts\cellmincer.exe\__main__.py", line 4, in <module>
ModuleNotFoundError: No module named 'cellmincer.cli'

Maybe a problem with the installation? I checked the ./env/cellmincer/Lib/site-packages/cellmincer and is empty. Should it not have the source code?

[bricewang]

Thank you for raising this issue! I agree that there appears to be a problem with installation--have you tried this?

python -m pip install git+https://github.com/broadinstitute/CellMincer.git

[Mariofe25]

I have tried and I get the same error. I have also tried using the command pip install -e git+https://github.com/broadinstitute/CellMincer.git#egg=CellMincer and got the following errors

(cellmincer) C:\cellmincer_test>cellmincer preprocess -i Zoom9-32x_00002_62mW.tif -o Zoom9-32x_00002_62mW --manifest manifest_config.yaml --config preprocess_config.yaml

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "\\?\C:\ProgramData\anaconda3\envs\cellmincer\Scripts\cellmincer-script.py", line 33, in <module>
    sys.exit(load_entry_point('cellmincer', 'console_scripts', 'cellmincer')())
  File "c:\src\cellmincer\cellmincer\cli\base_cli.py", line 94, in main
    cli_dict = generate_cli_dictionary()
  File "c:\src\cellmincer\cellmincer\cli\base_cli.py", line 56, in generate_cli_dictionary
    module_cli = importlib.import_module('.'.join(module_cli_str_list))
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "c:\src\cellmincer\cellmincer\train\cli.py", line 9, in <module>
    from cellmincer.train.main import Train
  File "c:\src\cellmincer\cellmincer\train\main.py", line 7, in <module>
    from lightning.pytorch import Trainer
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\__init__.py", line 19, in <module>
    from lightning.fabric.fabric import Fabric  # noqa: E402
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\fabric\__init__.py", line 30, in <module>
    from lightning.fabric.fabric import Fabric  # noqa: E402
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\fabric\fabric.py", line 46, in <module>
    from lightning.fabric.loggers import Logger
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\fabric\loggers\__init__.py", line 15, in <module>
    from lightning.fabric.loggers.tensorboard import TensorBoardLogger  # noqa: F401
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\fabric\loggers\tensorboard.py", line 31, in <module>
    from lightning.fabric.wrappers import _unwrap_objects
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\lightning\fabric\wrappers.py", line 38, in <module>
    from torch._dynamo import OptimizedModule
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\__init__.py", line 64, in <module>
    torch.manual_seed = disable(torch.manual_seed)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\decorators.py", line 50, in disable
    return DisableContext()(fn)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\eval_frame.py", line 410, in __call__
    (filename is None or trace_rules.check(fn))
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 3378, in check
    return check_verbose(obj, is_inlined_call).skipped
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 3361, in check_verbose
    rule = torch._dynamo.trace_rules.lookup_inner(
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 3442, in lookup_inner
    rule = get_torch_obj_rule_map().get(obj, None)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 2782, in get_torch_obj_rule_map
    obj = load_object(k)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 2811, in load_object
    val = _load_obj_from_str(x[0])
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\_dynamo\trace_rules.py", line 2795, in _load_obj_from_str
    return getattr(importlib.import_module(module), obj_name)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\nested\_internal\nested_tensor.py", line 417, in <module>
    values=torch.randn(3, 3, device="meta"),
C:\ProgramData\anaconda3\envs\cellmincer\Lib\site-packages\torch\nested\_internal\nested_tensor.py:417: UserWarning: Failed to initialize NumPy: _ARRAY_API not found (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:84.)
  values=torch.randn(3, 3, device="meta"),
INFO:root:Command:
cellmincer preprocess -i Zoom9-32x_00002_62mW.tif -o Zoom9-32x_00002_62mW --manifest manifest_config.yaml --config preprocess_config.yaml
cellmincer:preprocess:10:51:53: Command:
cellmincer preprocess -i Zoom9-32x_00002_62mW.tif -o Zoom9-32x_00002_62mW --manifest manifest_config.yaml --config preprocess_config.yaml
INFO:root:Dejittering movie...
cellmincer:preprocess:10:52:01: Dejittering movie...
Traceback (most recent call last):
  File "\\?\C:\ProgramData\anaconda3\envs\cellmincer\Scripts\cellmincer-script.py", line 33, in <module>
    sys.exit(load_entry_point('cellmincer', 'console_scripts', 'cellmincer')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\src\cellmincer\cellmincer\cli\base_cli.py", line 104, in main
    cli_dict[args.tool].run(args)
  File "c:\src\cellmincer\cellmincer\preprocess\cli.py", line 76, in run
    config=config).run()
                   ^^^^^
  File "c:\src\cellmincer\cellmincer\preprocess\main.py", line 63, in run
    movie_txy = self.dejitter(movie_txy)
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\src\cellmincer\cellmincer\preprocess\main.py", line 108, in dejitter
    np.min(self.ws_base.movie_txy[self.dejitter_config['ignore_first_n_frames']:, :, :]))
                                  ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'ignore_first_n_frames'

[bricewang]

We pushed a hotfix that should address the KeyError stemming from a config versioning issue; please try the updated preprocess config yaml.