Closed liang2025 closed 1 year ago
It appears that the issue may not be directly related to the Bitnami MySQL container image or Helm chart, but rather to how the application is being utilized or configured in your specific environment.
This seems to be a unique use case that might be challenging to replicate on our end, as it appears closely tied to your particular scenario.
For inquiries about the application itself, customizing its content, or questions about technology and infrastructure usage, we highly recommend referring to the forums and user guides provided by the project responsible for the application or technology.
With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.
I'm sorry I didn't understand what you said. This error I reported was a problem with the startup of bitnami "mysql", not "mongodb".
Sorry, I mean MySQL; I just edited the comment. We're not able to reproduce the issue on our end by deploying bitnami/mysql:8.0.34 and restarting the containers, after this process everything works as expected
Well, maybe you should install version 8.0.32 or 8.0.33, it should be reproducible, if it is the latest version, it may not be very good to reproduce this problem.
I know what the problem is. The problem is: after initializing the database when starting, add two parameters
--skip-slave-start
, because the database version is updated, there will be a warning during initialization, it is recommended to use skip-replica-startupgrade=FORCE
is also the reason why the database is always in the upgrade state and the update fails, causing the POD to be NOT RUNNING all the time.The complete startup parameters are: /opt/bitnami/mysql/bin/mysqld --defaults-file=/opt/bitnami/mysql/conf/my.cnf --basedir=/opt/bitnami/mysql --datadir=/bitnami /mysql/data --socket=/opt/bitnami/mysql/tmp/mysql.sock --bind-address=127.0.0.1 --skip-slave-start --upgrade=FORCE
How should I start without adding these parameters upgrade=FORCE
?
Hi @liang2025,
Could you share the values you are using so I can try to reproduce the change?
Hi @Mauraza , Of course, I would certainly like to share: The default startup command when no changes are made:
/opt/bitnami/mysql/bin/mysqld --defaults-file=/opt/bitnami/mysql/conf/my.cnf --basedir=/opt/bitnami/mysql --datadir=/ bitnami/mysql/data --socket=/opt/bitnami/mysql/tmp/mysql.sock --bind-address=127.0.0.1 --skip-slave-start --upgrade=FORCE
I change the startup command myself:
/opt/bitnami/mysql/bin/mysqld --defaults-file=/opt/bitnami/mysql/conf/my.cnf --basedir=/opt/bitnami/mysql --datadir=/bitnami /mysql/data --socket=/opt/bitnami/mysql/tmp/mysql.sock --pid-file=/opt/bitnami/mysql/tmp/mysqld.pid
Hi @Mauraza , There is another problem that needs to be noted. In the values.yaml file, some parameters of my.cnf have been updated in the new version and will be removed in future versions. I hope to update it in the charts later.
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Hi, I am trying to reproduce the issue but without any luck. I am using this docker-compose file where I am changing just the version of the image.
# Copyright VMware, Inc.
# SPDX-License-Identifier: APACHE-2.0
version: '2.1'
services:
mysql:
image: docker.io/bitnami/mysql:8.0.32-debian-
#image: docker.io/bitnami/mysql:8.0.34-debian-11-r0
ports:
- '3306:3306'
volumes:
- 'mysql_data:/bitnami/mysql/data'
environment:
# ALLOW_EMPTY_PASSWORD is recommended only for development.
- ALLOW_EMPTY_PASSWORD=yes
healthcheck:
test: ['CMD', '/opt/bitnami/scripts/mysql/healthcheck.sh']
interval: 15s
timeout: 5s
retries: 6
volumes:
mysql_data:
driver: local
I am starting the docker-compose and them stoping it and starting it again, and I got not errors when using 8.0.32, 8.0.33 or 8.0.34
09:38 $ docker-compose up
Creating network "debian-11_default" with the default driver
Creating volume "debian-11_mysql_data" with local driver
Pulling mysql (docker.io/bitnami/mysql:8.0.32-debian-11-r0)...
8.0.32-debian-11-r0: Pulling from bitnami/mysql
dfe669f82390: Already exists
d7e57a5543f1: Pull complete
Digest: sha256:87fa5b3abc3ac8143eb307df5c3f2eeb72bef98fa002414e5ee5e705c2236fb8
Status: Downloaded newer image for bitnami/mysql:8.0.32-debian-11-r0
Creating debian-11_mysql_1 ... done
Attaching to debian-11_mysql_1
mysql_1 | mysql 07:39:15.46
mysql_1 | mysql 07:39:15.46 Welcome to the Bitnami mysql container
mysql_1 | mysql 07:39:15.47 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql_1 | mysql 07:39:15.47 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql_1 | mysql 07:39:15.47
mysql_1 | mysql 07:39:15.47 INFO ==> ** Starting MySQL setup **
mysql_1 | mysql 07:39:15.50 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql_1 | mysql 07:39:15.50 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mysql_1 | mysql 07:39:15.51 INFO ==> Initializing mysql database
mysql_1 | mysql 07:39:15.54 INFO ==> Updating 'my.cnf' with custom configuration
mysql_1 | mysql 07:39:15.55 INFO ==> Setting slow_query_log option
mysql_1 | mysql 07:39:15.57 INFO ==> Setting long_query_time option
mysql_1 | mysql 07:39:15.58 INFO ==> Installing database
mysql_1 | mysql 07:39:29.57 INFO ==> Starting mysql in background
mysql_1 | mysql 07:39:31.60 INFO ==> Configuring authentication
mysql_1 | mysql 07:39:31.75 INFO ==> Running mysql_upgrade
mysql_1 | mysql 07:39:31.76 INFO ==> Stopping mysql
mysql_1 | mysql 07:39:33.79 INFO ==> Starting mysql in background
mysql_1 | find: '/docker-entrypoint-startdb.d/': No such file or directory
mysql_1 | mysql 07:39:53.86 INFO ==> Stopping mysql
mysql_1 | mysql 07:39:59.90 INFO ==> ** MySQL setup finished! **
mysql_1 |
mysql_1 | mysql 07:39:59.96 INFO ==> ** Starting MySQL **
mysql_1 | 2023-09-08T07:40:00.326761Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.32) starting as process 1
mysql_1 | 2023-09-08T07:40:00.330931Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
mysql_1 | 2023-09-08T07:40:00.330945Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
mysql_1 | 2023-09-08T07:40:00.348772Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql_1 | 2023-09-08T07:40:00.645268Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql_1 | 2023-09-08T07:40:01.126758Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql_1 | 2023-09-08T07:40:01.126813Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql_1 | 2023-09-08T07:40:01.162595Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock
mysql_1 | 2023-09-08T07:40:01.162886Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.32' socket: '/opt/bitnami/mysql/tmp/mysql.sock' port: 3306 Source distribution.
^CGracefully stopping... (press Ctrl+C again to force)
Stopping debian-11_mysql_1 ... done
09:40 $ docker-compose down
Removing debian-11_mysql_1 ... done
Removing network debian-11_default
09:40 $ docker-compose up
Creating network "debian-11_default" with the default driver
Creating debian-11_mysql_1 ... done
Attaching to debian-11_mysql_1
mysql_1 | mysql 07:40:34.75
mysql_1 | mysql 07:40:34.75 Welcome to the Bitnami mysql container
mysql_1 | mysql 07:40:34.75 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql_1 | mysql 07:40:34.75 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql_1 | mysql 07:40:34.76
mysql_1 | mysql 07:40:34.76 INFO ==> ** Starting MySQL setup **
mysql_1 | mysql 07:40:34.78 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql_1 | mysql 07:40:34.79 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
mysql_1 | mysql 07:40:34.79 INFO ==> Initializing mysql database
mysql_1 | mysql 07:40:34.82 INFO ==> Updating 'my.cnf' with custom configuration
mysql_1 | mysql 07:40:34.83 INFO ==> Setting slow_query_log option
mysql_1 | mysql 07:40:34.85 INFO ==> Setting long_query_time option
mysql_1 | mysql 07:40:34.86 INFO ==> Using persisted data
mysql_1 | mysql 07:40:34.90 INFO ==> Running mysql_upgrade
mysql_1 | mysql 07:40:34.91 INFO ==> Starting mysql in background
mysql_1 | find: '/docker-entrypoint-startdb.d/': No such file or directory
mysql_1 | mysql 07:40:54.97 INFO ==> Stopping mysql
mysql_1 |
mysql_1 | mysql 07:41:01.00 INFO ==> ** MySQL setup finished! **
mysql_1 | mysql 07:41:01.04 INFO ==> ** Starting MySQL **
mysql_1 | 2023-09-08T07:41:01.436424Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.32) starting as process 1
mysql_1 | 2023-09-08T07:41:01.440832Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
mysql_1 | 2023-09-08T07:41:01.440849Z 0 [Warning] [MY-013244] [Server] --collation-server: 'utf8mb3_general_ci' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.
mysql_1 | 2023-09-08T07:41:01.451304Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
mysql_1 | 2023-09-08T07:41:01.739801Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
mysql_1 | 2023-09-08T07:41:02.188202Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
mysql_1 | 2023-09-08T07:41:02.188284Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
mysql_1 | 2023-09-08T07:41:02.224100Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /tmp/mysqlx.sock
mysql_1 | 2023-09-08T07:41:02.224215Z 0 [System] [MY-010931] [Server] /opt/bitnami/mysql/bin/mysqld: ready for connections. Version: '8.0.32' socket: '/opt/bitnami/mysql/tmp/mysql.sock' port: 3306 Source distribution.
^CGracefully stopping... (press Ctrl+C again to force)
Stopping debian-11_mysql_1 ... done
Could you indicate how to reproduce it ?
This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.
Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.
still having the same issue on 8.0.34, tried using extraFlags: "--upgrade=MINIMAL"
but no luck
Solved.
For anyone facing the same error and end up at this issue, based on the answer by @liang2025, append this config at primary.command
in values.yaml
to override and remove the --upgrade
flags
command:
- "/opt/bitnami/mysql/bin/mysqld"
- "--defaults-file=/opt/bitnami/mysql/conf/my.cnf"
- "--basedir=/opt/bitnami/mysql"
- "--datadir=/bitnami/mysql/data"
- "--socket=/opt/bitnami/mysql/tmp/mysql.sock"
- "--pid-file=/opt/bitnami/mysql/tmp/mysqld.pid"
https://github.com/bitnami/containers/issues/44854#issuecomment-1683673108 https://github.com/bitnami/containers/issues/44854#issuecomment-1687261820
If you modify the chart before initialising any MySQL Server, and use the modified chart to create a brand new MySQL Server, you probably will encounter an error.
It appears that the directory /opt/bitnami/mysql
required certain initial steps to be created. If you lock the datadir
before initialising, MySQL will get a “no such file or directory” error.
If you encounter this issue, please undo these configs (by comment them and add a []
on primary.command
).
One of our test database was working fine for a couple of month, until the same issue started somehow (???).
2023-11-14T10:11:42.852620Z 4 [System] [MY-013381] [Server] Server upgrade from '80034' to '80034' started.
But being unable to do so, was loop crashing because being killed due to the livenessProbe. I guess the DB ended up being in a corrupted state, as even after disabling the probes, it didn't fare better
2023-11-14T10:12:38.994473Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement 'EXECUTE stmt; ' failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.
Tried bumping my chart to latest, now it's trying to upgrade from 80034 to 80035, same error.
Ended up wiping the database & starting from scratch, which is relatively easy for us
@rgarrigue just ended up having almost exactly the same thing after a wsl2 crash (running k3d/k3s) locally. I guess the problem is more to do with mysql/mariadb just not being very good (robust to disk failures anyway) rather than anything bitnami-specific.
I also encountered this error in MySQL version 8.0.35-debian-11-r0, but it does not happen every time.
2024-07-15T09:53:50.114237335+08:00 mysql 01:53:50.10
2024-07-15T09:53:50.115894476+08:00 mysql 01:53:50.11 Welcome to the Bitnami mysql container
2024-07-15T09:53:50.117349518+08:00 mysql 01:53:50.11 Subscribe to project updates by watching https://github.com/bitnami/containers
2024-07-15T09:53:50.118831633+08:00 mysql 01:53:50.11 Submit issues and feature requests at https://github.com/bitnami/containers/issues
2024-07-15T09:53:50.120299510+08:00 mysql 01:53:50.11
2024-07-15T09:53:50.121996509+08:00 mysql 01:53:50.12 INFO ==> ** Starting MySQL setup **
2024-07-15T09:53:50.149291453+08:00 mysql 01:53:50.14 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
2024-07-15T09:53:50.210285479+08:00 mysql 01:53:50.20 INFO ==> Initializing mysql database
2024-07-15T09:53:50.232344499+08:00 mysql 01:53:50.23 WARN ==> The mysql configuration file '/opt/bitnami/mysql/conf/my.cnf' is not writable. Configurations based on environment variables will not be applied for this file.
2024-07-15T09:53:50.234739587+08:00 mysql 01:53:50.23 INFO ==> Using persisted data
2024-07-15T09:53:50.425646590+08:00 mysql 01:53:50.42 INFO ==> Running mysql_upgrade
2024-07-15T09:53:50.437022601+08:00 mysql 01:53:50.43 INFO ==> Starting mysql in background
2024-07-15T09:53:51.690759143+08:00 2024-07-15T01:53:51.541460Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead.
2024-07-15T09:53:51.690786042+08:00 2024-07-15T01:53:51.560053Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2024-07-15T09:53:51.690790110+08:00 2024-07-15T01:53:51.560089Z 0 [System] [MY-010116] [Server] /opt/bitnami/mysql/bin/mysqld (mysqld 8.0.35) starting as process 47
2024-07-15T09:53:51.690797029+08:00 2024-07-15T01:53:51.656221Z 0 [Warning] [MY-013242] [Server] --character-set-server: 'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.
2024-07-15T09:53:51.753359521+08:00 2024-07-15T01:53:51.753222Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-07-15T09:53:58.488514068+08:00 2024-07-15T01:53:58.488387Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-07-15T09:54:05.469184930+08:00 2024-07-15T01:54:05.469026Z 4 [System] [MY-013381] [Server] Server upgrade from '80035' to '80035' started.
2024-07-15T09:55:29.356993432+08:00 2024-07-15T01:55:29.356854Z 4 [ERROR] [MY-013178] [Server] Execution of server-side SQL statement '-- Deprecate spatial reference systems -- Deprecation must happen before new SRSs are added/updated since there -- typically are new SRSs with the same names. REPLACE INTO mysql.st_spatial_reference_systems(id, catalog_id, name, organization, organization_coordsys_id, definition, description) VALUES (6956, 1, 'VN-2000 / TM-3 zone 481 (6956 deprecated)', 'EPSG', 6956, 'PROJCS["VN-2000 / TM-3 zone 481",GEOGCS["VN-2000",DATUM["Vietnam 2000",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],TOWGS84[-191.90441429,-39.30318279,-111.45032835,0.00928836,-0.01975479,0.00427372,0.252906278],AUTHORITY["EPSG","6756"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.017453292519943278,AUTHORITY["EPSG","9122"]],AXIS["Lat",NORTH],AXIS["Lon",EAST],AUTHORITY["EPSG","4756"]],PROJECTION["Transverse Mercator",AUTHORITY["EPSG","9807"]],PARAMETER["Latitude of natural origin",0,AUTHORITY["EPSG","8801"]],PARAMETER["Longitude of natural origin",102,AUTHORITY["EPSG","8802"]],PARAMETER["Scale factor at natural origin",0.9999,AUTHORITY["EPSG","8805"]],PARAMETER["False easting",0,AUTHORITY["EPSG","8806"]],PARAMETER["False northing",500000,AUTHORITY["EPSG","8807"]],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["E",EAST],AXIS["N",NORTH],AUTHORITY["EPSG","6956"]]', 'Deprecated since EPSG Dataset 9.3. Superseded by SRID 5896.'); ' failed with error code = 1205, error message = 'Lock wait timeout exceeded; try restarting transaction'.
2024-07-15T09:55:29.367915726+08:00 2024-07-15T01:55:29.367770Z 0 [ERROR] [MY-013380] [Server] Failed to upgrade server.
2024-07-15T09:55:29.367953588+08:00 2024-07-15T01:55:29.367803Z 0 [ERROR] [MY-010119] [Server] Aborting
2024-07-15T09:55:30.848575883+08:00 2024-07-15T01:55:30.848399Z 0 [System] [MY-010910] [Server] /opt/bitnami/mysql/bin/mysqld: Shutdown complete (mysqld 8.0.35) Source distribution.
containers libmysql.sh
mysql_upgrade() {
local -a args=("--defaults-file=${DB_CONF_FILE}" "-u" "$DB_ROOT_USER")
local major_version minor_version patch_version
major_version="$(get_sematic_version "$(mysql_get_version)" 1)"
minor_version="$(get_sematic_version "$(mysql_get_version)" 2)"
patch_version="$(get_sematic_version "$(mysql_get_version)" 3)"
info "Running mysql_upgrade"
if [[ "$DB_FLAVOR" = *"mysql"* ]] && [[
"$major_version" -gt "8"
|| ( "$major_version" -eq "8" && "$minor_version" -gt "0" )
|| ( "$major_version" -eq "8" && "$minor_version" -eq "0" && "$patch_version" -ge "16" )
]]; then
mysql_stop
mysql_start_bg "--upgrade=FORCE"
else
mysql_start_bg
is_boolean_yes "${ROOT_AUTH_ENABLED:-false}" && args+=("-p$(get_master_env_var_value ROOT_PASSWORD)")
debug_execute "${DB_BIN_DIR}/mysql_upgrade" "${args[@]}" || echo "This installation is already upgraded"
fi
}
Name and Version
bitnami/mysql:8.0.34-debian-11-r0
What architecture are you using?
None
What steps will reproduce the bug?
What is the expected behavior?
successfully run or successfully upgraded
What do you see instead?