[BUG] freeze on import of aeon.classification.convolution_based

user706 commented 1 week ago

Describe the bug

Hi, when doing

from aeon.classification.convolution_based import Arsenal

it takes forever.

It was working a few days back. In fact in the old venv it still works. (Edit: might be that the 1st loading always takes long. [Not quite sure how long it should take...])
But when trying a new venv (see below -> reproduce) it takes forever. Why?

my system

Ubuntu 24.04.1 LTS (according to cat /etc/os-release)
Python 3.12.3 (according to python --version after starting the venv (see below))

reproduce (or does it work on your side?)

# in the shell (I'm using bash on linux)
python3 -m venv pyvenv
. pyvenv/bin/activate
pip install --upgrade pip
pip install -U aeon
python -c "print('loading'); from aeon.classification.convolution_based import Arsenal; print('done')" # takes forever

Any ideas?

details

stopping the freeze by hitting Ctrl-C, results in

KeyboardInterrupt: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/__init__.py", line 10, in <module>
    from aeon.classification.convolution_based._arsenal import Arsenal
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/_arsenal.py", line 20, in <module>
    from aeon.transformations.collection.convolution_based import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/__init__.py", line 33, in <module>
    from aeon.transformations.collection.channel_selection import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/channel_selection/__init__.py", line 16, in <module>
    from aeon.transformations.collection.channel_selection._channel_scorer import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/channel_selection/_channel_scorer.py", line 7, in <module>
    from aeon.classification.convolution_based._rocket_classifier import RocketClassifier
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/_rocket_classifier.py", line 16, in <module>
    from aeon.transformations.collection.convolution_based import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/convolution_based/__init__.py", line 14, in <module>
    from ._multirocket import MultiRocket
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/convolution_based/_multirocket.py", line 556, in <module>
    @njit(
     ^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/decorators.py", line 232, in wrapper
    disp.compile(sig)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 912, in compile
    self._cache.save_overload(sig, cres)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/caching.py", line 652, in save_overload
    self._save_overload(sig, data)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/caching.py", line 661, in _save_overload
    data = self._impl.reduce(data)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/caching.py", line 389, in reduce
    return cres._reduce()
           ^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler.py", line 199, in _reduce
    libdata = self.library.serialize_using_object_code()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 921, in serialize_using_object_code
    data = (self._get_compiled_object(),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 629, in _get_compiled_object
    raise RuntimeError("no compiled object yet for %s" % (self,))
RuntimeError: no compiled object yet for <Library '_transform_multi' at 0x77c57e6ddd00>

or

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/__init__.py", line 10, in <module>
    from aeon.classification.convolution_based._arsenal import Arsenal
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/_arsenal.py", line 20, in <module>
    from aeon.transformations.collection.convolution_based import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/__init__.py", line 33, in <module>
    from aeon.transformations.collection.channel_selection import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/channel_selection/__init__.py", line 16, in <module>
    from aeon.transformations.collection.channel_selection._channel_scorer import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/channel_selection/_channel_scorer.py", line 7, in <module>
    from aeon.classification.convolution_based._rocket_classifier import RocketClassifier
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/classification/convolution_based/_rocket_classifier.py", line 16, in <module>
    from aeon.transformations.collection.convolution_based import (
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/convolution_based/__init__.py", line 14, in <module>
    from ._multirocket import MultiRocket
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/aeon/transformations/collection/convolution_based/_multirocket.py", line 556, in <module>
    @njit(
     ^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/decorators.py", line 232, in wrapper
    disp.compile(sig)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 905, in compile
    cres = self._compiler.compile(args, return_type)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 80, in compile
    status, retval = self._compile_cached(args, return_type)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 94, in _compile_cached
    retval = self._compile_core(args, return_type)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/dispatcher.py", line 107, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler.py", line 744, in compile_extra
    return pipeline.compile_extra(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler.py", line 438, in compile_extra
    return self._compile_bytecode()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler.py", line 506, in _compile_bytecode
    return self._compile_core()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler.py", line 472, in _compile_core
    pm.run(self.state)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 356, in run
    self._runPass(idx, pass_inst, state)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass
    mutated |= check(pss.run_pass, internal_state)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/compiler_machinery.py", line 273, in check
    mangled = func(compiler_state)
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/typed_passes.py", line 468, in run_pass
    lower.lower()
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/lowering.py", line 187, in lower
    self.lower_normal_function(self.fndesc)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/lowering.py", line 226, in lower_normal_function
    entry_block_tail = self.lower_function_body()
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/lowering.py", line 256, in lower_function_body
    self.lower_block(block)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/lowering.py", line 270, in lower_block
    self.lower_inst(inst)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/parfors/parfor_lowering.py", line 51, in lower_inst
    _lower_parfor_parallel(self, inst)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/parfors/parfor_lowering.py", line 67, in _lower_parfor_parallel
    return _lower_parfor_parallel_std(lowerer, parfor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/parfors/parfor_lowering.py", line 381, in _lower_parfor_parallel_std
    call_parallel_gufunc(
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/parfors/parfor_lowering.py", line 1672, in call_parallel_gufunc
    info = build_gufunc_wrapper(llvm_func, cres, sin, sout,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/np/ufunc/parallel.py", line 271, in build_gufunc_wrapper
    info = build_gufunc_kernel(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/np/ufunc/parallel.py", line 152, in build_gufunc_kernel
    wrapperlib.add_linking_library(info.library)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 720, in add_linking_library
    library._ensure_finalized()
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 567, in _ensure_finalized
    self.finalize()
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 762, in finalize
    self._optimize_final_module()
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/numba/core/codegen.py", line 682, in _optimize_final_module
    self._codegen._mpm_full.run(self._final_module)
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/llvmlite/binding/passmanagers.py", line 698, in run
    return ffi.lib.LLVMPY_RunPassManager(self, module)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/dude/proj/test_aeon/pyvenv/lib/python3.12/site-packages/llvmlite/binding/ffi.py", line 192, in __call__
    return self._cfn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Steps/Code to reproduce the bug

# in the shell (I'm using bash on linux)
python3 -m venv pyvenv
. pyvenv/bin/activate
pip install --upgrade pip
pip install -U aeon
python -c "print('loading'); from aeon.classification.convolution_based import Arsenal; print('done')" # takes forever

Expected results

should finish import quickly

Actual results

takes forever

Versions

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'show_versions' from 'aeon.utils' (/home/dude/aeon_test/pyvenv/lib/python3.12/site-packages/aeon/utils/__init__.py)

Any ideas? thanks.

user706 commented 1 week ago

Ok, incredible, seems like 1st loading really takes that long: 4 minutes 19 seconds on a rather fast laptop

# in the shell (I'm using bash on linux)
python3 -m venv pyvenv
. pyvenv/bin/activate
pip install --upgrade pip
pip install -U aeon

#1st loading
time python -c "print('~~loading (1st)'); from aeon.classification.convolution_based import Arsenal; print('~~done (1st)')"

echo -e "\n\n\n"

#next loading
time python -c "print('~~loading (next)'); from aeon.classification.convolution_based import Arsenal; print('~~done (next)')"

results in

~~loading (1st)
/home/dude/aeon_test/pyvenv/lib/python3.12/site-packages/aeon/base/__init__.py:24: FutureWarning: The aeon package will soon be releasing v1.0.0 with the removal of legacy modules and interfaces such as BaseTransformer and BaseForecaster. This will contain breaking changes. See aeon-toolkit.org for more information. Set aeon.AEON_DEPRECATION_WARNING or the AEON_DEPRECATION_WARNING environmental variable to 'False' to disable this warning.
  warnings.warn(
~~done (1st)

real    4m19,439s
user    4m12,878s
sys 0m10,163s

~~loading (next)
/home/dude/aeon_test/pyvenv/lib/python3.12/site-packages/aeon/base/__init__.py:24: FutureWarning: The aeon package will soon be releasing v1.0.0 with the removal of legacy modules and interfaces such as BaseTransformer and BaseForecaster. This will contain breaking changes. See aeon-toolkit.org for more information. Set aeon.AEON_DEPRECATION_WARNING or the AEON_DEPRECATION_WARNING environmental variable to 'False' to disable this warning.
  warnings.warn(
~~done (next)

real    0m1,821s
user    0m1,975s
sys 0m3,774s

Why is that actually? Is that numba or something else compiling something in the back?? thanks

TonyBagnall commented 1 week ago

thanks for that, we will take a look

user706 commented 1 week ago

As shown in the 2nd comment: might be that a 1st loading always takes long.

I am very interested in the speed of stuff like Arsenal classification (during normal operation, not first loading)

How fast is it? What tricks are employed (numba or otherwise).
In case you know the algorithmic details: how involved is it? Could this be straightforwardly coded in a compiled language like C++, for someone that knows C++ well? ...

Thanks.

TonyBagnall commented 1 week ago

hi, regarding the first question, I've just done a fresh install on windows through pycharm and it took about about 2 or 3 minutes, mostly scipy. I just tried with scikit-learn and it was a bit quicker, but still 1-2 mins and I think we have more core dependencies. Maybe your internet connection was flakey? I think it is just how it is and not really a bug, we cant really reduce our core dependencies any more. However, happy to hear any ideas on how to speed up the first build.

Regarding our implementation, its all done in numba and is we think as fast as it can be except for some possible minor tweaks.

Regarding implementing Arsenal and other Rocket classifiers in C++, I would say it would be quite easy for someone who knows C++ well, its basically lots of loops :)

user706 commented 6 days ago

hi, regarding the first question, I've just done a fresh install on windows through pycharm and it took about about 2 or 3 minutes, mostly scipy. I just tried with scikit-learn and it was a bit quicker, but still 1-2 mins and I think we have more core dependencies. Maybe your internet connection was flakey?

Hang on a second. My issue was not the downloading (via pip install), but that the very 1st call to...

#1st loading
python -c "print('~~loading (1st)'); from aeon.classification.convolution_based import Arsenal; print('~~done (1st)')"

... takes more than 4 minutes. (Probably because numba then kicks in and generates code, correct?)

But from then on... the ensuent calls (repeating the above) are fast.

Question: How long does that 1st call take for you, just after having finished pip install?

Thanks.

PS: I have no problem with it, if the 1st call takes long to generate or compile code; if it results in ensuent calls and actual classifications... being superfast. I just want to verify that this is normal behaviour.

MatthewMiddlehurst commented 5 days ago

I would assume numba compilation is the main culprit here, but not 100%. Is this only Arsenal or do you see this with other code you run? Best test would probably be something from deep_learning.

I have noticed this, I think it is numba caching or some sort of import caching. We could probably add a warning to the webpage documentation at least, I don't see the numba function compilation being easily resolvable though, if it is even feasible.

I'll see if I can do some runs at a later point, very annoying since it has to be fresh 🙂.

user706 commented 5 days ago

I would assume numba compilation is the main culprit here, but not 100%. Is this only Arsenal or do you see this with other code you run?

Just the import is suffcient to cause a 4 minute lag (when run the 1st time, in fresh venv):

from aeon.classification.convolution_based import Arsenal

(I'm only using Arsenal)

By the way... the uncool thing with numba is... it starts to get complicated when trying to do pip install aeon on ARM platforms such as raspberry pi: I need to separately compile LLVM in the correct version etc.

TonyBagnall commented 5 days ago

hi, regarding the first question, I've just done a fresh install on windows through pycharm and it took about about 2 or 3 minutes, mostly scipy. I just tried with scikit-learn and it was a bit quicker, but still 1-2 mins and I think we have more core dependencies. Maybe your internet connection was flakey?

Hang on a second. My issue was not the downloading (via pip install), but that the very 1st call to...

ah sorry, misunderstood.

Question: How long does that 1st call take for you, just after having finished pip install?

just done it, and it took about 2 minutes on a windows desk top, then is instant after that. As Matthew said, it will be the numba compilation. It has made me think of something though, it is possible it is compiling more functions than it needs. By modularising numba functions it means single runs like this may actually be doing more work than required. Efficient C programming and software engineering do not always go hand in hand :) We will discuss and see if we can sensibly measure this and maybe isolate more code.

PS: I have no problem with it, if the 1st call takes long to generate or compile code; if it results in ensuent calls and actual classifications... being superfast. I just want to verify that this is normal behaviour.

I think it is normal behaviour, although 4 mins is very slow for an import.

TonyBagnall commented 5 days ago

By the way... the uncool thing with numba is... it starts to get complicated when trying to do pip install aeon on ARM platforms such as raspberry pi: I need to separately compile LLVM in the correct version etc.

ouch, didnt know that, I have never used ARM platforms myself and am not good with such things, so cant really help. They claim to support ARM, maybe raise an issue over there?

https://numba.pydata.org/numba-doc/dev/user/installing.html

but if there is anything we can do our end to facilitate let us know

MatthewMiddlehurst commented 4 days ago

Probably related #1852. Unlikely to be resolved in the short term either case.

baraline commented 4 days ago

Adding a warming on the install page of the docs/(readme) with some short explanation might help new users to understand what is going on, I'll try to find some time to add it

aeon-toolkit / aeon