montreal_forced_aligner.exceptions.DatabaseError: DatabaseError: Database server has not been initialized (could not find /usr/local/tomcat/temp/mfa-UC207YW-20240524-083510/pg_mfa_global). Please run `mfa server init`.

[fishfree]

I step-by-step followed the course tutorial, when clicked on the Mfa button, it showed errors as below:

nzilbb.ag.TransformationException: Error running mfa align --clean --output_format long_textgrid /usr/local/tomcat/temp/mfa-UC207YW-20240524-083510/corpus english_mfa english_mfa /usr/local/tomcat/temp/mfa-UC207YW-20240524-083510/aligned --beam 10 --retry-beam 40 --uses_speaker_adaptation True --use_postgres : Traceback (most recent call last): File "/opt/conda/envs/aligner/bin/mfa", line 10, in <module> sys.exit(mfa_cli()) ^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__ return super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1685, in invoke super().invoke(ctx) File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/mfa.py", line 146, in mfa_cli check_server() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/utils.py", line 367, in check_server raise DatabaseError( montreal_forced_aligner.exceptions.DatabaseError: DatabaseError: Database server has not been initialized (could not find /usr/local/tomcat/temp/mfa-UC207YW-20240524-083510/pg_mfa_global). Please run `mfa server init`.

Then I logged into the container, run/opt/conda/bin/conda init bash && exit, then re-login into the container, run conda activate aligner && mfa server init, it showed errors as below:

(base) root@ceafc721b6e3:~#conda activate aligner
(aligner) root@ceafc721b6e3:~#mfa server init
 INFO     Initializing the global MFA database server...                                                                                     
 ERROR    pg_ctl stdout:                                                                                                                     
 ERROR    pg_ctl stderr: initdb: error: cannot be run as root                                                                                
          initdb: hint: Please log in (using, e.g., "su") as the (unprivileged) user that will own the server process.                       

Exception ignored in atexit callback: <bound method ExitHooks.history_save_handler of <montreal_forced_aligner.command_line.mfa.ExitHooks object at 0x7fa0bd2f35d0>>
Traceback (most recent call last):
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/mfa.py", line 98, in history_save_handler
    raise self.exception
  File "/opt/conda/envs/aligner/bin/mfa", line 10, in <module>
    sys.exit(mfa_cli())
             ^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/server.py", line 39, in init_cli
    initialize_server()
  File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/utils.py", line 327, in initialize_server
    raise DatabaseError(
montreal_forced_aligner.exceptions.DatabaseError: DatabaseError:

There was an error encountered starting the global MFA database server, please see /root/Documents/MFA/pg_init_log_global.txt for more details and/or look at the logged errors above.

I checked the file /root/Documents/MFA/pg_init_log_global.txt, emtpy.

FYI: https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner/issues/598 But it didn't help me.

[fishfree]

@robertfromont Robert, would you pls have a look at this issue? I think the erorr "ERROR pg_ctl stderr: initdb: error: cannot be run as root " is the root cause. In the docker-deployment, how the MFA & conda are automatically installed? I cannot find in the Dockerfile. FYI: we may need create a non-root user to run MFA.

[robertfromont]

Yes. MFA is under very active development, and each new version brings breaking changes for LaBB-CAT's MFA integration module.

MFA is installed by that module itself (not the Docker file), and it's likely the version of the MFA integration module you're running just installs the 'latest' version (now 3.n.n), but actually the module itself was tested against the stable version of MFA that was released at the time: 2.2.17. This is most likely the source of the error you're seeing.

You can confirm this by visiting the mfa phrase layer configuration page. At the bottom of that page, it displays both the integration module version, and also the version of MFA installed (labelled montreal-forced-aligner) - if this is version 3.something then the module is integrating with a version of MFA that behaves differently to what it expects.

The next released version of LaBB-CAT will include an MFA integration module that installs only the version it has been tested against, to avoid such future-version incompatibilities.

The plan is also to get the module running with MFA version 3, although if that turns out to be a lot of work, it may not make it into the next release of LaBB-CAT, which will hopefully be next month.

[fishfree]

Mine is as below:

$ docker exec -it labbcat /bin/bash
(base) root@ceafc721b6e3:/labbcat# conda activate aligner
(aligner) root@ceafc721b6e3:/labbcat# mfa version
2.2.17
(aligner) root@ceafc721b6e3:/labbcat# which python
/opt/conda/envs/aligner/bin/python
(aligner) root@ceafc721b6e3:/labbcat# python -V
Python 3.11.9

Is it because of the imcompatibility between the versions of MFA and Conda Python?

MFA is installed by that module itself (not the Docker file)

I deployed Labbcat server the docker container way, so when I install the MFA layer manager, it triggered the automatic installation. How can I manually interfere this action? I mean how to specify the conda version, mfa-running user and something else necessary?

Many thanks!

[robertfromont]

Please try this version of the MFA Layer Manager.

To install it:

1. Download and unzip the file MFA_0.2.13.zip
2. In LaBB-CAT, select the layer managers option on the menu at the top
3. Press Choose File at the bottom of the page, and select the MFA_0.2.13.jar file you just downloaded
4. Press Upload
5. Press Install
6. Press Configure

Now try forced alignment again, and let me know how it goes.

[fishfree]

@robertfromont Thank you! I did as your instruction, now it shows errors as below: nzilbb.ag.StoreException: No such task: mfa

Need I delete MFA Manager firstly?

[robertfromont]

Could you please return to the mfa layer configuration page (accessed from the phrase layers page). Then ensure that the Use PostGres option is un-ticked, and press Set Parameters (whether it was previously ticked or not). Then try running forced alignment again. If the error repeats, go to the activity page in LaBB-CAT, press the Log button of the task you see there, and copy/paste me the contents of the resulting log.

[fishfree]

@robertfromont Thank you! After pressing Set Parameters and regenerate, it started to work. However, new errors occured as below:

IndexError: list index out of rangenzilbb.ag.TransformationException: Error running mfa train --clean --output_format long_textgrid /usr/local/tomcat/temp/mfa-AP511_MikeThorpe-20240531-031529/corpus /usr/local/tomcat/temp/mfa-AP511_MikeThorpe-20240531-031529/AP511_MikeThorpe.dict /usr/local/tomcat/temp/mfa-AP511_MikeThorpe-20240531-031529/models --output_directory /usr/local/tomcat/temp/mfa-AP511_MikeThorpe-20240531-031529/aligned --beam 10 --retry-beam 40 --no_use_postgres : INFO Setting up corpus information... INFO Loading corpus from source files... INFO Found 1 speaker across 49 files, average number of utterances per speaker: 49.0 INFO Initializing multiprocessing jobs... WARNING Number of jobs was specified as 3, but due to only having 1 speakers, MFA will only use 1 jobs. Use the --single_speaker flag if you would like to split utterances across jobs regardless of their speaker. INFO Normalizing text... INFO Creating corpus split for feature generation... INFO Generating MFCCs... INFO Calculating CMVN... INFO Generating final features... INFO Creating corpus split with features... INFO Filtering utterances for training... INFO Initializing training for monophone... ERROR There was an error in the run, please see the log. Exception ignored in atexit callback: <bound method ExitHooks.history_save_handler of <montreal_forced_aligner.command_line.mfa.ExitHooks object at 0x7fc17f8a42d0>> Traceback (most recent call last): File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/mfa.py", line 98, in history_save_handler raise self.exception File "/opt/conda/envs/aligner/bin/mfa", line 10, in <module> sys.exit(mfa_cli()) ^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__ return super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/train_acoustic_model.py", line 111, in train_acoustic_model_cli trainer.train() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/acoustic_modeling/trainer.py", line 561, in train trainer.train() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/acoustic_modeling/base.py", line 488, in train self.initialize_training() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/acoustic_modeling/base.py", line 239, in initialize_training self._trainer_initialization() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/acoustic_modeling/monophone.py", line 329, in _trainer_initialization feat_dim = self.worker.get_feat_dim() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/corpus/acoustic_corpus.py", line 917, in get_feat_dim job = self.jobs[0] ~~~~~~~~~^^^ IndexError: list index out of range

[robertfromont]

This is an error generated by MFA, and I'm not sure what it means. However, I note that

1. you've configured MFA for 'train and then align' (i.e. with no pretrained models selected) - this requires something like 20+ hours of training data to work - and
2. you're aligning LaBB-CAT Demo data - specifically AP511_MikeThorpe - which definitely doesn't have enough data for train/align with MFA

This may be the cause of the error, and you may better to do this kind of alignment with a built-in MFA dictionary (e.g. english_uk_mfa) and pre-trained models (e.g. english_mfa).

[fishfree]

@robertfromont Thank you for your tip! Now I select the dictionary and the acoustic model as below: And checked in the container as below:

Now, the result of the regeneration still ignored the installed models and occured the error of "Error running mfa model download acoustic english_mfa" as below:

nzilbb.ag.TransformationException: Error running mfa model download acoustic english_mfa : Exception ignored in atexit callback: <bound method ExitHooks.history_save_handler of <montreal_forced_aligner.command_line.mfa.ExitHooks object at 0x7f6fbb93dd90>> Traceback (most recent call last): File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/mfa.py", line 98, in history_save_handler raise self.exception File "/opt/conda/envs/aligner/bin/mfa", line 10, in <module> sys.exit(mfa_cli()) ^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 367, in __call__ return super().__call__(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/rich_click/rich_command.py", line 152, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/command_line/model.py", line 77, in download_model_cli manager.download_model(model_type, name, version=version) File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/models.py", line 1656, in download_model self.refresh_remote() File "/opt/conda/envs/aligner/lib/python3.11/site-packages/montreal_forced_aligner/models.py", line 1532, in refresh_remote raise ModelsConnectionError(r.status_code, r.json(), r.headers) montreal_forced_aligner.exceptions.ModelsConnectionError: ModelsConnectionError: Current hourly rate limit (60 per hour) has been exceeded for the GitHub API. You can increase it by providing a personal authentication token to via --github_token. The rate limit will reset at 2024-06-01 15:10:17

[robertfromont]

In the current version of the MFA integration module, the dictionaries and models are downloaded from github every time forced alignment is run. If you run forced alignment frequently, unfortunately this means you may run into github's rate limiting. If you wait a while (the error says 2024-06-01 15:10:17 which is now in the past), then you will hopefully find it works.

The next version of the integration module includes re-use of dictionaries and models that are already downloaded, which addresses this annoyance.

[fishfree]

@robertfromont Thank you for your explanation. Pls inform me when the next version is ready, esp. the docker image upgraded :-)

[robertfromont]

Version 20240701.1144 was released today

[fishfree]

@robertfromont Thank you for informing me. I downloaded the latest war from sourceforge and build a new docker, the same nginx https reverse proxy as the last version. However, the error is as below:

Some resource files hardcoded with http:// ?