Lock file does not exist

[Zahrun]

 ~  UnixSync  Applications  aTrain_core transcribe "/media/data2/samples/Feedback.wav" --model large-v3 --language en --speaker_detection --device GPU --compute_type int8  
torchvision is not available - cannot save figures
Running aTrain_core
Created directory at /home/aroun/Documents/aTrain/transcriptions/2024-06-19 17-18-06 Feedb
.gitattributes: 100%|██████████████████████████████████████████████████| 1.52k/1.52k [00:00<00:00, 4.46MB/s]
config.json: 100%|█████████████████████████████████████████████████████| 2.39k/2.39k [00:00<00:00, 12.6MB/s]
README.md: 100%|███████████████████████████████████████████████████████| 2.05k/2.05k [00:00<00:00, 7.09MB/s]
tokenizer.json: 100%|██████████████████████████████████████████████████| 2.48M/2.48M [00:01<00:00, 1.83MB/s]
preprocessor_config.json: 100%|█████████████████████████████████████████████| 340/340 [00:00<00:00, 777kB/s]
model.bin: 100%|███████████████████████████████████████████████████████| 3.09G/3.09G [23:03<00:00, 2.23MB/s]
Fetching 7 files:  86%|██████████████████████████████████████████████▎       | 6/7 [23:04<03:50, 230.78s/it]
The following error has occured: [Errno 2] No such file or directory: '/home/aroun/Documents/aTrain/models/large-v3/.huggingface/.gitignore.lock'
Traceback: Traceback (most recent call last):
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/aTrain_core/cli.py", line 99, in cli
    transcribe(args.audiofile, file_id, args.model, args.language, args.speaker_detection, args.num_speakers, args.device, args.compute_type, timestamp)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/aTrain_core/transcribe.py", line 101, in transcribe
    model_path = get_model(model)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/aTrain_core/load_resources.py", line 48, in get_model
    snapshot_download(repo_id=model_info["repo_id"], revision=model_info["revision"], local_dir=model_path, local_dir_use_symlinks=False)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 294, in snapshot_download
    thread_map(
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/tqdm/std.py", line 1181, in __iter__
    for obj in iterable:
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
    return fut.result(timeout)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 268, in _inner_hf_hub_download
    return hf_hub_download(
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1202, in hf_hub_download
    return _hf_hub_download_to_local_dir(
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1406, in _hf_hub_download_to_local_dir
    paths = get_local_download_paths(local_dir=local_dir, filename=filename)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/_local_folder.py", line 138, in get_local_download_paths
    metadata_path = _huggingface_dir(local_dir) / "download" / f"{sanitized_filename}.metadata"
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/_local_folder.py", line 223, in _huggingface_dir
    with WeakFileLock(gitignore_lock):
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/huggingface_hub/utils/_fixes.py", line 84, in WeakFileLock
    lock.acquire()
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/filelock/_api.py", line 321, in acquire
    self._acquire()
  File "/home/aroun/UnixSync/Applications/atrain_core_venv/lib/python3.10/site-packages/filelock/_unix.py", line 42, in _acquire
    fd = os.open(self.lock_file, open_flags, self._context.mode)
FileNotFoundError: [Errno 2] No such file or directory: '/home/aroun/Documents/aTrain/models/large-v3/.huggingface/.gitignore.lock'

Why even is it defaulting to "~/Documents/aTrain/"? That folder is meant to store Documents, as the name suggests. aTrain data should probably be kept either in the venv installation folder or in ~/.local/share/aTrain

[Zahrun]

I suggest following the XDG_Base_Directory freedesktop specification https://wiki.archlinux.org/title/XDG_Base_Directory

[JuergenFleiss]

Hi, will have to have a look at that; did not throw an error on our two linux test machines.

As for the reasoning, the models used to be in the venv, resulting in +10GB download sizes for the gui installers, in turn resulting in user complains.

With atrain_core (and the upcoming new gui version to be based on it) we introduce downloading and removing models on the user side; in the process of doing so, we moved them to the same folder as the transcriptions, mainly having gui users in mind that then can find atrain downloads and transcriptions in the same place.

But we will have a look at the link you posted and rethink the model location.

[Zahrun]

Can I manually specify a different directory? On my system, ~/Documents is mapped to a different type of device, so I'm thinking that it might be the source of the issue.

[Zahrun]

You may be interested to have a short read on the FHS https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard The file-hierarchy man page has a section on the home directory https://www.freedesktop.org/software/systemd/man/latest/file-hierarchy.html#Home%20Directory

[JuergenFleiss]

Can I manually specify a different directory? On my system, ~/Documents is mapped to a different type of device, so I'm thinking that it might be the source of the issue.

I think it should be sufficient to change the definition of model_path in get_model() -- line 46 -- and delete_model() -- line 54 -- in load_resources.py, as transcribe.py gets it from there.

[JuergenFleiss]

@Zahrun can you try installing the branch Frontend_Connection should fix both the numpy issue as well as the model location in linux, but i dont have a linux testsystem at hand right now

[Zahrun]

That created a ~ directory under the current execution directory. So in my case that created /home/aroun/UnixSync/Applications/~/.local/share/aTrain/models/large-v3/model.bin instead of going to the absolute ~. The lock file issue seems to have disappeared. Now I successfully transcribe on CPU.

Can’t diarize though as the path for that model has also changed FileNotFoundError: [Errno 2] No such file or directory: '/home/aroun/Documents/aTrain/models/diarize/config.yaml'

For GPU, I had "Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory" although the file exists at "/home/aroun/UnixSync/Applications/atrain_core_venv/lib64/python3.10/site-packages/nvidia/cudnn/lib/libcudnn_ops_infer.so.8". Solved that with sudo apt install nvidia-cudnn Now it does not complain but I get nothing transcribed

torchvision is not available - cannot save figures
Running aTrain_core
Preparing transcription
Created directory at /home/aroun/Documents/aTrain/transcriptions/2024-06-20 00-27-11 Feedb
Time taken from start to init model 5.540017127990723
Time taken from start until getting the transcription segments and info via transcription_model.transcribe 15.601551055908203
Transcription segments of faster whisper 0
Total steps without diarization: 0
Progress 0/0
Transcribing with Whisper: 100%|███████████████████▉| 113.8986875/113.9 [00:08<00:00, 13.95 audio seconds/s]
Finishing up
Thank you for using aTrain 
If you use aTrain in a scientific publication, please cite our paper:
'Take the aTrain. Introducing an interface for the Accessible Transcription of Interviews'
available under: https://www.sciencedirect.com/science/article/pii/S2214635024000066

I have CUDA 11.5.1 installed, is it too old?

[Zahrun]

This will direct to the correct directory.

model_path = os.path.join("~/.local/share/aTrain", "models", model)
model_path = os.path.expanduser(model_path)

https://docs.python.org/3/library/os.path.html#os.path.expanduser

For diarize, the path can be changed at https://github.com/JuergenFleiss/atrain_core/blob/b5269d2c071003f8e06a053a747bf9edece5de2b/aTrain_core/transcribe.py#L28 as config_yml = os.path.join(os.path.expanduser("~"), ".local/share/aTrain", "models", "diarize", "config.yaml")