problem with scikit-fda

[arezooorooji]

Describe the bug Hi there, I hope you are doing well. I was trying to import skfda. but, I encountered a problem which is as follows:

_FileNotFoundError: [Errno 2] No such file or directory: 'C:\Users\Arezoo\AppData\Local\Programs\Python\Python310\lib\site-packages\dcor\pycache\_fast_dcov_mergesort._generate_distance_covariance_sqr_mergesort_generic_impl.locals._distance_covariance_sqr_mergesort_genericimpl-212.py310.1.nbc.tmp.21925382eb1f4350'

I appreciate it if you could guide me to fix it.

To Reproduce

import skfda
...

Expected behavior I wanna use the scikit-fda package.

Version information

• OS: Windows
• Python version: Python 3.10.7
• numpy version=1.23.3
• scipy version=1.9.1
• matplotlib version=3.6.0

Additional context Add any other context about the problem here.

[vnmabus]

I think this problem is because Numba uses very long names for its cache files, but Windows has a small length limit for the whole path. When the cache files are generated inside a folder too deep in the hierarchy, the path is longer than the limit.

Please try defining a different, more shallow, folder for Numba cache files using an environment variable, as explained in https://github.com/vnmabus/dcor/issues/31#issuecomment-943233307.

[vnmabus]

As you requested me further help, I will post this here in case it helps others too:

Description of the error:

• Numba is a (transitive) dependency of scikit-fda, used to compile numerical Python code to achieve more efficiency.

• In order to speed up the functions that use Numba, these can be compiled the first time they are defined (that is why the first time you use the library it takes longer to import), and the compiled code is stored (cached) in a file in disk for subsequent uses.

• By default, Numba caches these functions in the directory in the pycache directory of the package which defines them, and use very long names for the files (concatenating the full path of the compiled function and additional data). In your case, the directory is "C:\Users\Arezoo\AppData\Local\Programs\Python\Python310\lib\site-packages\dcor\pycache" (because dcor is the package that defines them) and the name of the file is "_fast_dcov_mergesort._generate_distance_covariance_sqr_mergesort_generic_impl.locals.distance_covariance_sqr_mergesort_generic_impl-212.py310.1.nbc.tmp.21925382eb1f4350...".

• Windows has a limit of 260 for the full path (directory + "/" + file).

• In your case the full path is "C:\Users\Arezoo\AppData\Local\Programs\Python\Python310\lib\site-packages\dcor\pycache_fast_dcov_mergesort._generate_distance_covariance_sqr_mergesort_generic_impl.locals.distance_covariance_sqr_mergesort_generic_impl-212.py310.1.nbc.tmp.21925382eb1f4350..."

• It seems that in some versions of Python in Windows, the path is truncated to 256 bytes before reading it (https://bugs.python.org/issue45777).

• As your path is longer than that, it is truncated and then Numba fails to open it.

Solution:

• Numba can use a different directory for the cache files if you have the NUMBA_CACHE_DIR environment variable set to that directory.

• If you set NUMBA_CACHE_DIR to a shallower directory, such as "C:\Users\Arezoo\numba_cache", you will reduce the length of the path (in this case, by 59 characters). This is probably enough for the path to be shorter than 256 characters and thus work.

Alternative solution:

• Change to another version of Python without that particular bug. Note that Windows could still prevent the usage of paths longer than a particular limit (260) depending on your configuration, but at least the error in that case should be more clear (OSError: [Errno 36] File name too long). If your version of Windows is updated, you can remove this restriction (https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=registry#enable-long-paths-in-windows-10-version-1607-and-later).

I hope this is detailed enough for you to solve this problem and be able to work with scikit-fda. If that is not the case, don't hesitate to ask again for more help.