Closed Champ-Goblem closed 1 year ago
Hi @Champ-Goblem , thanks for the detailed report, could you try nydusify check --source <your_original_mysql_image> --target <your_nydus_mysql_image>
with nydus-image & nydusd 2.1.4
?
Hi @imeoer, the image we are using is standard OCI rather than a RAFS formatted image, the image being used by the helm chart is docker.io/bitnami/mysql:8.0.32-debian-11-r14
, does this change things?
it seems this issue is mainly related to FUSE? IOWs, fuse-backend-rs?
Seems to be solved in 2.1.6
We are experiencing some problems with Nydus since at least version 2.1.4 when running some stateful workloads. The main affected workload is MySQL which fails to start correctly when run on Nydus 2.1.4+ along with Kata 3.0.2. The previous version we ran of Nydus was 2.1.0-rc.3 which when running with Kata 3.0.2 works as expected.
The error that MySQL throws during the startup phase:
This is unexpected because the provisioning scripts start MySQL in the background and enabling debug mode shows no errors during the startup of the background MySQL.
You can recreate it by following these steps:
spec.template.spec.env
containsbinPath
andcontainerdPath
which is dependent upon the installationspec.template.spec.volumes
containsbin
which should point to an active path in$PATH
andcontained
which should point to the location of theconfig.toml
filehelm install mysql bitnami/mysql -f values.yaml
with the below values forvalues.yaml
To note, if you want to
helm uninstall mysql
thenhelm install
again, please delete the persistent volume claim between the uninstall and install steps.When you run the above MySQL in runc, the command in
test.sh
under theinitdbScripts
will print the server status, this works because the Bitnami scripts start MySQL during the init phase in the background, allowing the scripts to make changes to the configs. Whereas when run with Kata and the latest Nydus we see that the same command fails to connect to MySQL over the socket present at/opt/bitnami/mysql/tmp/mysql.sock
even though MySQL is running correctly. You may view the logs for MySQL running in the background by settingimage.debug=true
in the helm values.To get an idea of how stable the newest version of Nydus is, we tried running the xfstest utility against it https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git/tree/. For nydus 2.1.0-rc3 this yielded:
and for version 2.2.0 the results were:
We have made a ticket with the fuse-backend-rs crate with a suggestion about integrating xfstests into their testing regime to try and catch any problems early https://github.com/cloud-hypervisor/fuse-backend-rs/issues/111.