FileNotFoundError when attempting to download MIT1003 database

[CarolinGhali]

Hello,

I've encountered a FileNotFoundError issue while trying to download the MIT1003 dataset using pysaliency. Below is the error traceback:

FileNotFoundError                Traceback (most recent call last)
File ~\anaconda3\Lib\site-packages\scipy\io\matlab\_mio.py:39, in _open_file(file_like, appendmat, mode)
     38 try:
---> 39     return open(file_like, mode), True
     40 except OSError as e:
     41     # Probably "not found"

FileNotFoundError: [Errno 2] No such file or directory: '..\\Local\\Temp\\tmpwnyng9c2\\extracted\\i05june05_static_street_boston_p1010764.jpeg_CNG.mat'

This error occurs during the extraction and loading process of the MIT1003 dataset. It seems like the script is unable to find or access the .mat file after downloading and extracting the dataset.

I am using pysaliency version 0.2.21 on an Anaconda environment with Python. I have also ensured that all dependencies are up to date and have attempted to run the code on different MATLAB versions (2023b and 2020a) to check for compatibility issues, but the error persists.

Could you please provide guidance on how to resolve this issue, or is there a known workaround to successfully download and access the MIT1003 dataset through pysaliency?

Thank you for your assistance.

[AshtrayGu]

same problem

[ahnchive]

I just checked the download is working with pysaliency 0.2.22, matlab 2023a on Ubuntu machine! I had to install matlab engine as per this: https://www.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html

[matthias-k]

Thank you all for your comments and my apologies for not reaching out earlier.

I just tried to download the MIT1003 dataset locally and for me (debian testing, Matlab R2022a) it works smoothly. Part of the MIT1003 import process is running MATLAB to extract the fixation data from the provided gaze traces. The error message looks a bit like this part didn't succeed. Errors in MATLAB should be detected by my code but unfortunately it's nontrivial to run matlab in a failsafe way and maybe I missed something. Could you check and maybe paste the output also before the error message? Is matlab starting at all? It should print something like Processing scanpath i/15045 with i iterating from 0 to 15044. The full matlab call should take easily 10 minutes.

@ahnchive the matlab engine for python should not be necessary, because I simply call a matlab script from python's subprocess library, but there is no direct communication involved.

[alikecat]

The same problem. Moreover, I found that the MATLAB Command Window displayed the error: Function or variable extract_all_fixations is not recognized. It seems that this is the reason why the correct file was not output.

[alikecat]

I believe I might have identified the true cause of this issue.

In line 118 of mit.py, the method run_matlab_cmd attempts to invoke Matlab. After that, Matlab takes a bit of time for startup and loading. However, during this period, the Python code does not wait for the startup and loading process of Matlab but continues to run further.

Eventually, before Matlab is fully started, the files in the temporary directory are cleared. This results in Matlab being unable to find the extract_all_fixations method that should originally be in the temporary directory, causing a FileNotFoundError in Python and simultaneously a "Function or variable extract_all_fixations is not recognized" error in Matlab.

I attempted to add a time.sleep after line 118 of mit.py and found that it could solve this problem, but this is not an appropriate solution.

@matthias-k , I sincerely hope that you or other experts could notice and solve this.

[alikecat]

I have discovered the underlying cause and the corresponding solution.

An inappropriate Matlab startup parameter exists in line 333 of pysaliency/utils/__init__.py. The startup parameter -r initiates an interactive workflow, while -nosplash merely hides the visual window during startup loading.

In some instances, the Python script does not wait for the conclusion of this interactive workflow, consequently resulting in a series of FileNotFoundError.

Therefore, it is merely necessary to remove the parameters -r and -nosplash, and add a new startup parameter -batch. This will enable Matlab to start in a non-interactive mode, including direct startup without the desktop, not showing the startup screen, and disabling changes to presets and disabling the toolbox cache.

I sincerely hope that this suggestion will be noticed promptly and the modification can be made without delay here to resolve the problem.

[matthias-k]

@alikecat thanks for the comment! I'm implementing your suggestion right now. I'm not sure that it explains the problem because the code should always wait for matlab to complete, but it is a good idea anyway. When I started implementing the matlab models, I only found -r as the recommended way to run scripts, and I also never liked it. -batch seems to be the much better solution.

[matthias-k]

Another question for everybody who experienced this problem: Which OS are you running? And maybe also add the matlab version. Could it be that the problem always happens in windows?

[matthias-k]

I'm not sure that it explains the problem because the code should always wait for matlab to complete,

ah, maybe matlab behaves differently under windows. We'll see soon

[matthias-k]

I just merged a PR which (among other things) switches matlab calls to use the -batch option. It's in the current dev branch, could people give it a try?