BUG: Runtime error when using animate_fisheye()

CassandraAuri commented 1 year ago

Describe the issue:

When using this simpe code snippet where rego is from \asilib\asi\rego.py you get a runetime error from anaconda multiprocessing

RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Reproduce the code example:

import asilib

import rego  # just copy of rego.py
from datetime import datetime, timedelta
time_range = (datetime(2021, 3, 18, 8, 0),
              datetime(2021, 3, 18, 8, 30))
asi = rego.rego('GILL', time_range=time_range)
# And run either one of these two commands
asi.animate_fisheye()  # animate fisheye images

Error message:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
    prepare(preparation_data)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
  File "C:\Users\1101w\anaconda3\lib\runpy.py", line 265, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\1101w\anaconda3\lib\runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\1101w\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "c:\Users\1101w\Clone\Programming-8\Summer 2023\helptest.py", line 9, in <module>
    asi.animate_fisheye()  # animate fisheye images
  File "C:\Users\1101w\anaconda3\lib\site-packages\asilib\imager.py", line 251, in animate_fisheye
    for _ in movie_generator:
  File "C:\Users\1101w\anaconda3\lib\site-packages\asilib\imager.py", line 382, in animate_fisheye_gen
    for i, (image_time, image) in _progressbar:
  File "C:\Users\1101w\anaconda3\lib\site-packages\asilib\utils.py", line 134, in progressbar
    for i, item in enumerate(iterator):
  File "C:\Users\1101w\anaconda3\lib\site-packages\asilib\imager.py", line 1232, in __iter__
    for file_times, file_images in self.iter_files():
  File "C:\Users\1101w\anaconda3\lib\site-packages\asilib\imager.py", line 1276, in iter_files
    times, images = self._data['loader'](path)
  File "c:\Users\1101w\Clone\Programming-8\Summer 2023\regotest.py", line 326, in _load_rego_pgm
    images, meta, problematic_file_list = rego_imager_readfile.read(str(path))
  File "C:\Users\1101w\anaconda3\lib\site-packages\rego_imager_readfile\_rego.py", line 35, in read
    pool = Pool(processes=workers)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\pool.py", line 212, in __init__
    self._repopulate_pool()
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\context.py", line 326, in _Popen
    return Popen(process_obj)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 45, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\1101w\anaconda3\lib\multiprocessing\spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Runtime information:

0.14.3 3.8.3 (default, Jul 2 2020, 17:30:36) [MSC v.1916 64 bit (AMD64)] Windows-10-10.0.19041-SP0

Context for the issue:

When testing TREX I was trying to figure out how imager.py was fetching the infromation (as it couldnt find the TREx file) fixing this would help me implement TREx into imager.py and then eventually a way to figure out a way to implement TREx into asilib.animate_fisheye_generator()

mshumko commented 1 year ago

Hi @CassandraAuri,

I am able to reproduce this error. I also confirmed that this error affects the themis() imager instance.

As mentioned in this recent issue, it is a common bug with the Windows multiprocessing module. The permanent solution necessitates the removal of the multiprocessing module from themis-imager-readfile and rego-imager-readfile which @dchaddock mentioned is unlikely. This leaves asilib with two choices:

Refactor a copy of the themis-imager-readfile and rego-imager-readfile source code in asilib. This is not difficult, but then asilib's loaders will be unsupported by the Calgary developers. Given that the THEMIS and REGO data format is stable, this should be OK.
Instruct Windows users to use asilib inside of a if __name__ == '__main__': block as I show in the example below. This is an easy fix if you're writing scripts using asilib, but won't work if you put your asilib analysis code into our own module and import it.

The example below, slightly modified, works on my Windows 10 machine.

  from datetime import datetime, timedelta

  import asilib
  from asilib.asi import rego

  if __name__ == '__main__':
      time_range = (datetime(2021, 3, 18, 8, 0),
                  datetime(2021, 3, 18, 8, 5))
      asi = rego.rego('GILL', time_range=time_range)
      # And run either one of these two commands
      asi.animate_fisheye()  # animate fisheye images

I'm leaning towards option 1, but I am open to ideas or suggestions.

CassandraAuri commented 1 year ago

So I talked to darren and name == 'main' will just have to be the solution since everything they have runs like that.

dchaddock commented 1 year ago

Hi all - I am going through Cassandra's code right now working on figuring out the best way to address the issue. Updates should be available in the next few days.

dchaddock commented 1 year ago

With the latest release of our readfile libraries this morning, this issue has been resolved. You can do the following do get up-to-date.

pip install --upgrade trex-imager-readfile themis-imager-readfile rego-imager-readfile

mshumko / asilib