search

MontrealCorpusTools / Montreal-Forced-Aligner

Command line utility for forced alignment using Kaldi
https://montrealcorpustools.github.io/Montreal-Forced-Aligner/
MIT License
1.34k stars 246 forks source link

[BUG] postgres errors #632

Closed lifeiteng closed 1 year ago

lifeiteng commented 1 year ago

Debugging checklist

[ ] Have you updated to latest MFA version? yes [ ] Have you tried rerunning the command with the --clean flag? yes

Describe the issue

 >mfa server init
There was an error encountered starting the global MFA database server, please see /home/feiteng/Documents/MFA/pg_log_global.txt for more details and/or look at the logged errors above.

>cat /home/feiteng/Documents/MFA/pg_log_global.txt
2023-05-12 13:41:59.184 CST [1007567] FATAL:  could not open lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied
2023-05-12 13:41:59.184 CST [1007567] LOG:  database system is shut down

>sudo chmod 777 /var/run/postgresql
 >mfa server init
2023-05-12 13:43:43.022 CST [1007702] FATAL:  could not open lock file "/var/run/postgresql/.s.PGSQL.5432.lock": Permission denied
2023-05-12 13:43:43.022 CST [1007702] LOG:  database system is shut down

>rm -r /home/feiteng/Documents/MFA/pg_mfa_global/
>sudo rm /var/run/postgresql/.s.PGSQL.5432.lock
>mfa server init
 INFO     Initializing the global MFA database server...
 INFO     Starting the global MFA database server...
waiting for server to start.... done
server started
 INFO     global MFA database server started!

>mfa validate ~/speech/MontrealCorpusTools/Librispeech english_us_arpa english_us_arpa

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"?

>touch /home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432
>mfa validate ~/speech/MontrealCorpusTools/Librispeech english_us_arpa english_us_arpa

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
    Is the server running locally and accepting
    connections on Unix domain socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"?

full error info

 >mfa validate --clean ~/speech/MontrealCorpusTools/Librispeech english_us_arpa english_us_arpa
waiting for server to start.... done
server started
 ERROR    There was an error in the run, please see the log.
 INFO     Stopping the global MFA database server...
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
    return self.pool.connect()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1268, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 283, in _do_get
    return self._create_connection()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 903, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
  File "/home/feiteng/.local/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running locally and accepting
    connections on Unix domain socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/command_line/utils.py", line 282, in check_databases
    with engine.connect():
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3264, in connect
    return self._connection_cls(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2426, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3288, in raw_connection
    return self.pool.connect()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 1268, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/impl.py", line 283, in _do_get
    return self._create_connection()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 903, in __connect
    pool.logger.debug("Error on connect(): %s", e)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/util/langhelpers.py", line 147, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
  File "/home/feiteng/.local/lib/python3.8/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
    Is the server running locally and accepting
    connections on Unix domain socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"?

(Background on this error at: https://sqlalche.me/e/20/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/abc.py", line 243, in initialize_database
    check_databases(self.identifier)
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/command_line/utils.py", line 286, in check_databases
    raise DatabaseError()
montreal_forced_aligner.exceptions.DatabaseError: DatabaseError:

There was an error connecting to the global MFA database server. Please ensure the server is initialized (mfa server init) or running (mfa server start)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/abc.py", line 246, in initialize_database
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['createdb', '--host=/home/feiteng/Documents/MFA/pg_mfa_global_socket', 'Librispeech']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/feiteng/.local/bin/mfa", line 8, in <module>
    sys.exit(mfa_cli())
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/feiteng/.local/lib/python3.8/site-packages/rich_click/rich_group.py", line 21, in main
    rv = super().main(*args, standalone_mode=False, **kwargs)
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/feiteng/.local/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/command_line/validate.py", line 113, in validate_corpus_cli
    validator.validate()
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/validation/corpus_validator.py", line 595, in validate
    self.setup()
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/validation/corpus_validator.py", line 539, in setup
    self.initialize_database()
  File "/home/feiteng/speech/MontrealCorpusTools/Montreal-Forced-Aligner/montreal_forced_aligner/abc.py", line 256, in initialize_database
    raise DatabaseError(
montreal_forced_aligner.exceptions.DatabaseError: DatabaseError:

There was an error connecting to the global MFA database server. Please ensure the server is initialized (mfa server init) or running (mfa server start)
lifeiteng commented 1 year ago

started! ?

>mfa moserver start
 WARNING  The global MFA database server does not exist, initializing it first.
 INFO     Initializing the global MFA database server...
 INFO     Starting the global MFA database server...
waiting for server to start.... done
server started
 INFO     global MFA database server started!

but there is error in pg_init_log_global.txt


createuser: error: could not connect to database template1: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"?
``
frouhi commented 1 year ago

I'm having the same issue! is there some kind of a temporary workaround around it?

lifeiteng commented 1 year ago

NOTE: I install using pip

Fix postgresql listening port

2023-05-13 09:34:53.147 CST [1080548] LOG:  starting PostgreSQL 12.14 (Ubuntu 12.14-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2023-05-13 09:34:53.147 CST [1080548] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-05-13 09:34:53.331 CST [1080549] LOG:  database system was shut down at 2023-05-13 09:34:52 CST

Edit /home/YOURNAME/Documents/MFA/pg_mfa_global/postgresql.conf unix_socket_directories = '/home/YOURNAME/Documents/MFA/pg_mfa_global_socket'

2023-05-13 09:44:55.369 CST [1081298] LOG:  starting PostgreSQL 12.14 (Ubuntu 12.14-0ubuntu0.20.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0, 64-bit
2023-05-13 09:44:55.411 CST [1081298] LOG:  listening on Unix socket "/home/feiteng/Documents/MFA/pg_mfa_global_socket/.s.PGSQL.5432"
2023-05-13 09:44:55.645 CST [1081299] LOG:  database system was shut down at 2023-05-13 09:40:39 CST

"/usr/share/postgresql/12/extension/vector.control": No such file or directory

2023-05-13 09:45:25.452 CST [1081369] STATEMENT:  CREATE EXTENSION IF NOT EXISTS vector
2023-05-13 09:50:09.677 CST [1081674] ERROR:  type "vector" does not exist at character 146

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1963, in _exec_single_context
    self.dialect.do_execute(
  File "/home/feiteng/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 920, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.UndefinedFile: could not open extension control file "/usr/share/postgresql/12/extension/vector.control": No such file or directory

install https://github.com/pgvector/pgvector

sudo apt install postgresql-server-dev-12   # edit 12 to YOUR VERSION

git clone https://github.com/pgvector/pgvector.git
cd pgvector
make
sudo make install
mmcauliffe commented 1 year ago

Is there a reason you can't install it via conda (https://montreal-forced-aligner.readthedocs.io/en/latest/getting_started.html)? That should take care of setting up all of the non-python dependencies and not require you to do anything with the system postgresql.

Pip and source installation are not well supported because basically everything about MFA depends on non-python code (i.e., Kaldi, OpenFst, PostgreSQL, etc), so you'll likely hit more of these issues outside of a conda installation.

lifeiteng commented 1 year ago

@mmcauliffe yes, I'm familiar with kaldi, MFA works now.

frouhi commented 1 year ago

@mmcauliffe thanks for the suggestion. I am using conda tho! in fact, I'm using the colab linked in the documentation here: https://montreal-forced-aligner.readthedocs.io/en/latest/first_steps/example.html#example-1-aligning-librispeech-english (Google Colab notebook

This one also has the same issue: https://colab.research.google.com/drive/1FSu12RGCa3xXiwYwlNXNUd-GKuggwmGW?usp=sharing

they both use conda. I'd appreciate any suggestions or pointers, specifically for the colab environment (maybe a working notebook?).

Also, please let me know if anyone can actually run these notebooks in colab without hitting the error. It might be somehow linked to my own colab env!

thanks