RuntimeError: Cannot find cache directory at cache_path='$HOME/.cache' when using evaluate

[mayalinetsky-kryon]

Hi, I've followed the instructions in the README, and when I'm running the evaluate command from the usage example I get the following error:

  File "C:\Program Files\JetBrains\PyCharm 2023.2\plugins\python\helpers\pydev\pydevconsole.py", line 364, in runcode
    coro = func()
  File "<input>", line 1, in <module>
  File "C:\git\algo-research\gai-qa\gai_qa_venv\lib\site-packages\aac_metrics\functional\evaluate.py", line 102, in evaluate
    candidates = preprocess_mono_sents(
  File "C:\git\algo-research\gai-qa\gai_qa_venv\lib\site-packages\aac_metrics\utils\tokenization.py", line 195, in preprocess_mono_sents
    tok_sents = ptb_tokenize_batch(
  File "C:\git\algo-research\gai-qa\gai_qa_venv\lib\site-packages\aac_metrics\utils\tokenization.py", line 80, in ptb_tokenize_batch
    raise RuntimeError(f"Cannot find cache directory at {cache_path=}.")
RuntimeError: Cannot find cache directory at cache_path='$HOME/.cache'.```

I'm working on Windows, using PyCharm IDE.

[Labbeti]

Hi ! The package hasn't been tested on Windows, so I guess that's the problem since paths are handled differently. Could you tell me what the result of the aac-metrics-download command is? It should indicate where the models (like SPICE) have been downloaded.

[mayalinetsky-kryon]

I don't remember if the first time had this exception, but this time I get this:

(base) (gai_qa_venv) PS C:\git\algo-research\gai-qa> aac-metrics-download
[2023-08-02 09:09:36,778][aac_metrics.download][INFO] - Stanford model file 'stanford_nlp' is already downloaded.
[2023-08-02 09:09:36,778][aac_metrics.download][INFO] - Meteor file 'meteor' is already downloaded.
[2023-08-02 09:09:36,778][aac_metrics.download][INFO] - Meteor file 'meteor_data' is already downloaded.
[2023-08-02 09:09:36,778][aac_metrics.download][INFO] - Downloading JAR sources for SPICE metric into '$HOME/.cache\aac-metrics\spice'...
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\git\algo-research\gai-qa\gai_qa_venv\Scripts\aac-metrics-download.exe\__main__.py", line 7, in <module>
  File "C:\git\algo-research\gai-qa\gai_qa_venv\lib\site-packages\aac_metrics\download.py", line 217, in _main_download
    download(
  File "C:\git\algo-research\gai-qa\gai_qa_venv\lib\site-packages\aac_metrics\download.py", line 143, in download
    subprocess.check_call(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\subprocess.py", line 1420, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
(base) (gai_qa_venv) PS C:\git\algo-research\gai-qa> ```

It just writes '$HOME' and not the actual value.
Then I use the app "Everything" to search for the path `aac-metrics\spice` and the path is literally `C:\git\algo-research\$HOME\.cache\aac-metrics\spice`...

[Labbeti]

I see the problem now! The original bash install script is called in my code for SPICE, and I don't think it would work with Windows. Unfortunately, I don't have the time to rewrite and test all the bash scripts to make it work by now. The two current solutions would be to use a Linux-based distribution instead of Windows or to manually install all the files required for SPICE, METEOR and the PTB tokenizer, but I'm not sure if that would be enough.

[mayalinetsky-kryon]

I was interested only in METEOR. Do you think I should install it manually? (if so, how?) Or should I look for a different package that supports windows?

[Labbeti]

According to the aac-metrics-download output, METEOR is already installed on your system, but SPICE is not. To run METEOR only, you can try the following example:

from aac_metrics.functional import meteor
scores, _ = meteor(["a man speaks"], [["man speaking"]])
print(scores["meteor"].item()) # 0.37281534596864435

Otherwise, I think the official METEOR repo is here: https://github.com/cmu-mtlab/meteor, and other implementations of METEOR in the NLTK or HuggingFace/evaluate packages exist, but I don't know if they can work on Windows.

[Labbeti]

This issue is fixed in v0.4.5 since the path are now handled with "~" instead of "$HOME" variable.