search

passbolt / charts-passbolt

Helm charts to run Passbolt on Kubernetes. No strings attached charts to run the open source password manager for teams!
https://passbolt.com
GNU Affero General Public License v3.0
41 stars 27 forks source link

Connection Refused Error When Deploying Passbolt Helm Chart on EKS #36

Closed localminimum closed 11 months ago

localminimum commented 1 year ago

I am deploying the Passbolt Helm Chart on an AWS EKS cluster followed the blog. However, I'm running into an issue where the Passbolt pods are not able to establish a connection to my MySQL database. The error message in the pod logs is Connection to Mysql could not be established: SQLSTATE[HY000] [2002] Connection refused.

2023-07-18 21:52:48 error: [Cake\Database\Exception\DatabaseException] SQLSTATE[42S02]: Base table or view not found: 1146 Table 'passbolt.email_queue' doesn't exist in /usr/share/php/passbolt/vendor/cakephp/cakephp/src/Database/Schema/Collection.php on line 160
Caused by: [PDOException] SQLSTATE[42S02]: Base table or view not found: 1146 Table 'passbolt.email_queue' doesn't exist in /usr/share/php/passbolt/vendor/cakephp/cakephp/src/Database/Statement/MysqlStatement.php on line 39

Steps to Reproduce: Deploy the Passbolt Helm Chart on an AWS EKS cluster (1.26) following the blog

Expected Behavior: The Passbolt application should be able to connect to the MySQL database and function properly without any connection errors.

Actual Behavior: I'm getting a 1146 Table 'passbolt.email_queue' doesn't exist error from the Passbolt pods when they try to connect to the MySQL database.

I would appreciate any guidance on how to resolve this issue. Please let me know if additional information is needed.

localminimum commented 1 year ago

This issue may be related? I'm not entirely sure

dlen commented 1 year ago

The error you are pasting does not look like a database connection error. It looks like the migrations have not been executed on the database.

You can provide the full passbolt container logs to better identify the issue. The issue you are linking doesn't look like it is related with yours

localminimum commented 1 year ago

Thanks for a quick response. Here's the pod status

kubectl get pods -n passbolt 
NAME                                              READY   STATUS              RESTARTS      AGE
test-my-passbolt-cron-proc-email-28162636-rd7bz   1/2     Terminating         2 (15s ago)   60s
test-my-passbolt-cron-proc-email-28162637-slkvk   0/2     ContainerCreating   0             0s
test-my-passbolt-depl-srv-7ccff6d558-nhq8d        1/2     CrashLoopBackOff    2 (28s ago)   104s
test-my-passbolt-job-create-gpg-keys-cwk67        0/1     Completed           0             113s
test-my-passbolt-mariadb-primary-0                1/1     Running             0             104s
test-my-passbolt-mariadb-secondary-0              1/1     Running             0             104s
test-my-passbolt-redis-node-0                     2/2     Running             0             104s
test-my-passbolt-redis-node-1                     2/2     Running             0             74s
test-my-passbolt-redis-node-2                     2/2     Running             0             42s

As you can see, test-my-passbolt-cron-proc-email repeats crashing->restarting with similar logs as below.

kubectl logs test-my-passbolt-cron-proc-email-28162636-rd7bz -n passbolt --follow
Defaulted container "test-my-passbolt-cron-proc-email" out of: test-my-passbolt-cron-proc-email, test-my-passbolt-cron-proc-email-redisproxy
Sending emails
2023-07-19 09:16:22 error: [Cake\Database\Exception\DatabaseException] SQLSTATE[42S02]: Base table or view not found: 1146 Table 'passbolt.email_queue' doesn't exist in /usr/share/php/passbolt/vendor/cakephp/cakephp/src/Database/Schema/Collection.php on line 160
Caused by: [PDOException] SQLSTATE[42S02]: Base table or view not found: 1146 Table 'passbolt.email_queue' doesn't exist in /usr/share/php/passbolt/vendor/cakephp/cakephp/src/Database/Statement/MysqlStatement.php on line 39
<!DOCTYPE html>
<html class="passbolt no-js no-passboltplugin version" lang="en">
<head>
    <meta charset="utf-8"/>
    <title>Passbolt | An Internal Error Has Occurred.</title>
    <!--

        ____                  __          ____
       / __ \____  _____ ____/ /_  ____  / / /_
      / /_/ / __ `/ ___/ ___/ __ \/ __ \/ / __/
     / ____/ /_/ (__  )__  ) /_/ / /_/ / / /_
    /_/    \__,_/____/____/_.___/\____/_/\__/

    Open source password manager for teams
    Copyright (c) Passbolt SA (https://www.passbolt.com)

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU Affero General Public License as
    published by the Free Software Foundation, either version 3 of the
    License, or (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU Affero General Public License for more details.

    You should have received a copy of the GNU Affero General Public License
    along with this program. If not, see http://www.gnu.org/licenses/.

    -->
    <meta name="description" content="Open source password manager for teams">
    <meta name="keywords" content="Passbolt, password manager, online password manager, open source password manager">
    <meta name="robots" content="noindex, nofollow">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="shortcut icon" type="image/x-icon" href="https://passbolt.local/favicon.ico" />
    <link rel="icon" href="https://passbolt.local/favicon_32.png" sizes="32x32" />
    <link rel="icon" href="https://passbolt.local/favicon_57.png" sizes="57x57" />
    <link rel="icon" href="https://passbolt.local/favicon_76.png" sizes="76x76" />
    <link rel="icon" href="https://passbolt.local/favicon_96.png" sizes="96x96" />
    <link rel="icon" href="https://passbolt.local/favicon_128.png" sizes="128x128" />
    <link rel="icon" href="https://passbolt.local/favicon_192.png" sizes="192x192" />
    <link rel="icon" href="https://passbolt.local/favicon_228.png" sizes="228x228" />
    <base href="https://passbolt.local/">
    <link rel="stylesheet" href="https://passbolt.local/css/themes/default/api_main.min.css?v=4.0.2"/></head>
<body spellcheck="false">
<div id="container" class="error page error-500">
<!-- header navigation -->
<header>
    <div class="header first ">
        <nav>
            <div class="top navigation primary">
                <ul>
                    <li class="left"><a href="https://passbolt.local/"><span>home</span></a></li>
                    <li class="right"><a href="https://passbolt.local/auth/login"><span>login</span></a></li>
                </ul>
            </div>
        </nav>
    </div>
</header>
<div id="content">
<div class="grid">
    <div class="row">
        <h2>An Internal Error Has Occurred</h2>
        <p class="error">
            An Internal Error Has Occurred.        </p>
    </div>
    </div>
</div>
</div>
<footer>
    <div class="footer">
        <ul class="footer-links">
            <li><a href="https://www.passbolt.com/terms">Terms</a></li>
            <li><a href="https://www.passbolt.com/privacy">Privacy</a></li>
            <li><a href="https://www.passbolt.com/credits">Credits</a></li>
            <li id="version">
                <a href="https://www.passbolt.com/credits" class="tooltip-left" data-tooltip="4.0.2">
                    <i class="fa fa-heart-o"></i>
                    <span class="visuallyhidden">Versions</span>
                </a>
            </li>
        </ul>
    </div>
</footer>
</div>
</body>
</html>

Here's the logs for MariaDB pod

kubectl logs test-my-passbolt-mariadb-primary-0 -n passbolt
mariadb 09:15:26.22 
mariadb 09:15:26.22 Welcome to the Bitnami mariadb container
mariadb 09:15:26.23 Subscribe to project updates by watching https://github.com/bitnami/containers
mariadb 09:15:26.23 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mariadb 09:15:26.24 
mariadb 09:15:26.24 INFO  ==> ** Starting MariaDB setup **
mariadb 09:15:26.28 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 09:15:26.29 INFO  ==> Initializing mariadb database
mariadb 09:15:26.32 WARN  ==> The mariadb configuration file '/opt/bitnami/mariadb/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
mariadb 09:15:26.32 INFO  ==> Installing database
2023-07-19  9:15:26 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
mariadb 09:15:28.40 INFO  ==> Starting mariadb in background
2023-07-19  9:15:28 0 [Note] Starting MariaDB 10.6.12-MariaDB-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 101
2023-07-19  9:15:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-07-19  9:15:28 0 [Note] InnoDB: Number of pools: 1
2023-07-19  9:15:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-07-19  9:15:28 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2023-07-19  9:15:28 0 [Note] InnoDB: Using Linux native AIO
2023-07-19  9:15:28 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-07-19  9:15:28 0 [Note] InnoDB: Completed initialization of buffer pool
2023-07-19  9:15:28 0 [Note] InnoDB: 128 rollback segments are active.
2023-07-19  9:15:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-07-19  9:15:28 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-07-19  9:15:28 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-07-19  9:15:28 0 [Note] InnoDB: 10.6.12 started; log sequence number 42108; transaction id 14
2023-07-19  9:15:28 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2023-07-19  9:15:28 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-07-19  9:15:28 0 [Note] InnoDB: Buffer pool(s) load completed at 230719  9:15:28
2023-07-19  9:15:28 0 [Note] Server socket created on IP: '127.0.0.1'.
2023-07-19  9:15:28 0 [Warning] 'user' entry 'root@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
2023-07-19  9:15:28 0 [Warning] 'user' entry '@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
2023-07-19  9:15:28 0 [Warning] 'proxies_priv' entry '@% root@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
2023-07-19  9:15:28 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.12-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
mariadb 09:15:30.43 INFO  ==> Configuring authentication
2023-07-19  9:15:30 7 [Warning] 'proxies_priv' entry '@% root@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
mariadb 09:15:30.62 INFO  ==> Configuring replication in master node
2023-07-19  9:15:30 14 [Warning] 'proxies_priv' entry '@% root@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
mariadb 09:15:30.67 INFO  ==> Running mysql_upgrade
find: '/docker-entrypoint-startdb.d/': No such file or directory
mariadb 09:15:30.72 INFO  ==> Stopping mariadb
2023-07-19  9:15:30 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (initiated by: unknown): Normal shutdown
2023-07-19  9:15:30 0 [Note] InnoDB: FTS optimize thread exiting.
2023-07-19  9:15:30 0 [Note] InnoDB: Starting shutdown...
2023-07-19  9:15:30 0 [Note] InnoDB: Dumping buffer pool(s) to /bitnami/mariadb/data/ib_buffer_pool
2023-07-19  9:15:30 0 [Note] InnoDB: Buffer pool(s) dump completed at 230719  9:15:30
2023-07-19  9:15:30 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2023-07-19  9:15:30 0 [Note] InnoDB: Shutdown completed; log sequence number 42120; transaction id 15
2023-07-19  9:15:30 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: Shutdown complete

mariadb 09:15:31.74 INFO  ==> ** MariaDB setup finished! **

mariadb 09:15:31.76 INFO  ==> ** Starting MariaDB **
2023-07-19  9:15:31 0 [Note] Starting MariaDB 10.6.12-MariaDB-log source revision 4c79e15cc3716f69c044d4287ad2160da8101cdc as process 1
2023-07-19  9:15:31 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-07-19  9:15:31 0 [Note] InnoDB: Number of pools: 1
2023-07-19  9:15:31 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-07-19  9:15:31 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2023-07-19  9:15:31 0 [Note] InnoDB: Using Linux native AIO
2023-07-19  9:15:31 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-07-19  9:15:31 0 [Note] InnoDB: Completed initialization of buffer pool
2023-07-19  9:15:31 0 [Note] InnoDB: 128 rollback segments are active.
2023-07-19  9:15:31 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2023-07-19  9:15:31 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2023-07-19  9:15:31 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2023-07-19  9:15:31 0 [Note] InnoDB: 10.6.12 started; log sequence number 42120; transaction id 14
2023-07-19  9:15:31 0 [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
2023-07-19  9:15:31 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-07-19  9:15:31 0 [Note] InnoDB: Buffer pool(s) load completed at 230719  9:15:31
2023-07-19  9:15:31 0 [Note] Server socket created on IP: '0.0.0.0'.
2023-07-19  9:15:31 0 [Note] Server socket created on IP: '::'.
2023-07-19  9:15:31 0 [Warning] 'proxies_priv' entry '@% root@test-my-passbolt-mariadb-primary-0' ignored in --skip-name-resolve mode.
2023-07-19  9:15:31 0 [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
Version: '10.6.12-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
2023-07-19  9:16:07 7 [Note] Start binlog_dump to slave_server(670), pos(, 4), using_gtid(0), gtid('')

Redis logs

kubectl logs test-my-passbolt-redis-node-0 -n passbolt                                                                             (base) 
Defaulted container "redis" out of: redis, sentinel
 09:15:26.59 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h test-my-passbolt-redis.passbolt.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
Could not connect to Redis at test-my-passbolt-redis.passbolt.svc.cluster.local:26379: Connection refused
 09:15:26.62 INFO  ==> Configuring the node as master
1:C 19 Jul 2023 09:15:26.641 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19 Jul 2023 09:15:26.641 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 19 Jul 2023 09:15:26.641 # Configuration loaded
1:M 19 Jul 2023 09:15:26.652 * monotonic clock: POSIX clock_gettime
1:M 19 Jul 2023 09:15:26.653 * Running mode=standalone, port=6379.
1:M 19 Jul 2023 09:15:26.653 # Server initialized
1:M 19 Jul 2023 09:15:26.667 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 19 Jul 2023 09:15:26.671 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 19 Jul 2023 09:15:26.671 * Ready to accept connections
1:M 19 Jul 2023 09:15:58.315 * Replica test-my-passbolt-redis-node-1.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 asks for synchronization
1:M 19 Jul 2023 09:15:58.315 * Full resync requested by replica test-my-passbolt-redis-node-1.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379
1:M 19 Jul 2023 09:15:58.315 * Replication backlog created, my new replication IDs are '4088d44265b000fda6c384bc27066b9dee44a8c6' and '0000000000000000000000000000000000000000'
1:M 19 Jul 2023 09:15:58.315 * Delay next BGSAVE for diskless SYNC
1:M 19 Jul 2023 09:16:03.829 * Starting BGSAVE for SYNC with target: replicas sockets
1:M 19 Jul 2023 09:16:03.829 * Background RDB transfer started by pid 120
120:C 19 Jul 2023 09:16:03.830 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 19 Jul 2023 09:16:03.830 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
1:M 19 Jul 2023 09:16:03.835 * Background RDB transfer terminated with success
1:M 19 Jul 2023 09:16:03.835 * Streamed RDB transfer with replica test-my-passbolt-redis-node-1.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
1:M 19 Jul 2023 09:16:03.835 * Synchronization with replica test-my-passbolt-redis-node-1.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 succeeded
1:M 19 Jul 2023 09:16:27.332 * Replica test-my-passbolt-redis-node-2.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 asks for synchronization
1:M 19 Jul 2023 09:16:27.332 * Full resync requested by replica test-my-passbolt-redis-node-2.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379
1:M 19 Jul 2023 09:16:27.332 * Delay next BGSAVE for diskless SYNC
1:M 19 Jul 2023 09:16:32.091 * Starting BGSAVE for SYNC with target: replicas sockets
1:M 19 Jul 2023 09:16:32.092 * Background RDB transfer started by pid 250
250:C 19 Jul 2023 09:16:32.093 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 19 Jul 2023 09:16:32.095 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
1:M 19 Jul 2023 09:16:32.100 * Background RDB transfer terminated with success
1:M 19 Jul 2023 09:16:32.100 * Streamed RDB transfer with replica test-my-passbolt-redis-node-2.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
1:M 19 Jul 2023 09:16:32.100 * Synchronization with replica test-my-passbolt-redis-node-2.test-my-passbolt-redis-headless.passbolt.svc.cluster.local:6379 succeeded

And finally, passbolt-depl-srv logs. Let me know if you need the full logs for below

Defaulted container "test-my-passbolt-depl-srv" out of: test-my-passbolt-depl-srv, test-my-passbolt-depl-srv-redisproxy, test-my-passbolt-depl-srv-init (init)
gpg: keybox '/var/lib/passbolt/.gnupg/pubring.kbx' created
gpg: /var/lib/passbolt/.gnupg/trustdb.gpg: trustdb created
gpg: key 943E0756F397540C: public key "Passbolt default user <min@getradiant.ai>" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: key 943E0756F397540C: "Passbolt default user <min@getradiant.ai>" not changed
gpg: key 943E0756F397540C: secret key imported
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
Installing passbolt
<!DOCTYPE html> ... truncated
Running migrations
<!DOCTYPE html> ... truncated
Clearing cake caches
<!DOCTYPE html> ... truncated
dlen commented 1 year ago

Hello!

Sorry for the delay. It looks like the container can't connect to the database to run the migrations. Could you confirm you can connect to your database from passbolt container?

dlen commented 11 months ago

I'm closing this as stale, feel free to reopen with more information