Various temp file errors after SylNet completed

[smeylan]

Running the demo (on Ubuntu 14.04),

Found wav files.
Creating config for pyannote.
Done creating config for pyannote.
Extracting KCHI
Feature extractors based on "yaafe" are not available because "yaafelib" could not be found.
/home/stephan/miniconda3/envs/ALICE/lib/python3.6/site-packages/pyannote/audio/applications/base.py:87: YAMLLoadWarning: calling yaml.load() without Loader=... is depreca
ted, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  params_yml = yaml.load(fp)
Test set: 1it [00:05,  5.90s/it]
Test set: 1it [00:00, 11.38it/s]
Extracting CHI
Feature extractors based on "yaafe" are not available because "yaafelib" could not be found.
/home/stephan/miniconda3/envs/ALICE/lib/python3.6/site-packages/pyannote/audio/applications/base.py:87: YAMLLoadWarning: calling yaml.load() without Loader=... is depreca
ted, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  params_yml = yaml.load(fp)
Test set: 1it [00:05,  5.99s/it]
Test set: 1it [00:00, 15.03it/s]
Extracting MAL
Feature extractors based on "yaafe" are not available because "yaafelib" could not be found.
/home/stephan/miniconda3/envs/ALICE/lib/python3.6/site-packages/pyannote/audio/applications/base.py:87: YAMLLoadWarning: calling yaml.load() without Loader=... is depreca
ted, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  params_yml = yaml.load(fp)
Test set: 1it [00:05,  5.94s/it]
Test set: 1it [00:00, 16.37it/s]
Extracting FEM
Feature extractors based on "yaafe" are not available because "yaafelib" could not be found.
/home/stephan/miniconda3/envs/ALICE/lib/python3.6/site-packages/pyannote/audio/applications/base.py:87: YAMLLoadWarning: calling yaml.load() without Loader=... is depreca
ted, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  params_yml = yaml.load(fp)
Test set: 1it [00:05,  5.99s/it]
Test set: 1it [00:00, 18.03it/s]
Extracting SPEECH
Feature extractors based on "yaafe" are not available because "yaafelib" could not be found.
/home/stephan/miniconda3/envs/ALICE/lib/python3.6/site-packages/pyannote/audio/applications/base.py:87: YAMLLoadWarning: calling yaml.load() without Loader=... is depreca
ted, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  params_yml = yaml.load(fp)
Test set: 1it [00:05,  5.90s/it]
Test set: 1it [00:00, 17.73it/s]
SylNet completed
Traceback (most recent call last):
  File "extract_basic_features.py", line 10, in <module>
    with open(curdir + '/tmp_data/features/SylNet_out_files.txt') as csv_file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/stephan/notebooks/ALICE/tmp_data/features/SylNet_out_files.txt'
paste: /home/stephan/notebooks/ALICE/tmp_data/features/SylNet_out.txt: No such file or directory
regress_ALUCs.py:10: UserWarning: genfromtxt: Empty input file: "/home/stephan/notebooks/ALICE/tmp_data/features/final_feats.txt"
  F = genfromtxt(curdir + "/tmp_data/features/final_feats.txt", delimiter='\t')
Traceback (most recent call last):
  File "regress_ALUCs.py", line 25, in <module>
    words_est = np.dot(F,X_words)
  File "<__array_function__ internals>", line 6, in dot
ValueError: shapes (0,2) and (5,) not aligned: 2 (dim 1) != 5 (dim 0)
paste: /home/stephan/notebooks/ALICE/tmp_data/features/SylNet_out_files.txt: No such file or directory
rm: cannot remove '/home/stephan/notebooks/ALICE/tmp_data/features/ALUCs_out_individual_tmp.txt': No such file or directory
getFinalEstimates.py:10: UserWarning: genfromtxt: Empty input file: "/home/stephan/notebooks/ALICE/tmp_data/features/ALUCs_out_individual.txt"
  F = genfromtxt(curdir + "/tmp_data/features/ALUCs_out_individual.txt", delimiter='\t')
ALICE completed. Results written to /home/stephan/notebooks/ALICE/ALICE_output.txt and /home/stephan/notebooks/ALICE/diarization_output.rttm.```

[macramole]

Hi, Leandro here from ROS lab.

Having the exact same output.

This is tmp_data tree before it gets deleted

tmp_data/ ├── features │   ├── ALUCs_out_individual.txt │   └── final_feats.txt ├── helenab-a1-nsm-chops013.wav ├── helenab-a1-nsm-chops014.wav └── short ├── helenab-a1-nsm-chops013_00344150_00358610.wav ├── helenab-a1-nsm-chops013_00374930_00418520.wav ├── helenab-a1-nsm-chops013_00439980_00450110.wav ├── helenab-a1-nsm-chops014_00129920_00145090.wav ├── helenab-a1-nsm-chops014_00224930_00229920.wav ├── helenab-a1-nsm-chops014_00240010_00243250.wav ├── helenab-a1-nsm-chops014_00244300_00245550.wav ├── helenab-a1-nsm-chops014_00246390_00248480.wav ├── helenab-a1-nsm-chops014_00250100_00254710.wav ├── helenab-a1-nsm-chops014_00661250_00662460.wav ├── helenab-a1-nsm-chops014_00662530_00664960.wav ├── helenab-a1-nsm-chops014_00665110_00668370.wav ├── helenab-a1-nsm-chops014_00669670_00682140.wav ├── helenab-a1-nsm-chops014_00690590_00719260.wav ├── helenab-a1-nsm-chops014_00719310_00729940.wav ├── helenab-a1-nsm-chops014_00801410_00810200.wav ├── helenab-a1-nsm-chops014_00823250_00854840.wav ├── helenab-a1-nsm-chops014_00983510_00984590.wav ├── helenab-a1-nsm-chops014_00984930_00990110.wav ├── helenab-a1-nsm-chops014_01190030_01191630.wav └── helenab-a1-nsm-chops014_01194900_01199910.wav

[orasanen]

Hi! Thanks for testing this out! Could you paste contents of sylnet.log here after the crash? Thanks!

[macramole]

I found the bug.

Line 62, when running SylNet you are using &> to save stdout and stderr to file.

This is fine using bash but not using sh (it is not the same thing :D!). In sh this will run SylNet in background and continue processing main script which end up in error since SylNet_out_files.txt has not been created yet.

In your README below Usage you run the commands using sh when you should be running them using bash (as the shebang you defined).

I would make run_alice.sh executable and run it like ./run_alice.sh (since the shebang is well defined) . Not sure if this works properly in Mac though.

[orasanen]

Thanks for the update! Let me look into this. I need to figure out how the sh/bash differ in practice, so this might take a while. (me being a Mac user and stuff)

[macramole]

No problem, I will test ALICE with some short unseen anotated data and update you with the results. I made a quick test of the diarization tool and it had promising results as well.

[smeylan]

Indeed chmod +x run_ALICE.sh and ./run_ALICE.sh demo/ROS_5271_20_01_03600_snippet_mono.wav fixes my problem -- thanks @macramole!

[orasanen]

I see your point now! Thank you!

As a fix, I updated the readme and changed the permissions of the run_ALICE.sh for execution. I will have to study the shebang definitions if it is the correct one now or not, but at least it should work.

[macramole]

Cool. I'm pretty sure it is correct.