mysql : InnoDB: Unable to lock ./ibdata1 error: 11

[Nick-0314]

What happened:

Mysql:

I forced the operating system to restart (after power restart) mysql startup error InnoDB: Unable to lock ./ibdata1 error: 11 The 2022-03-16 T02:10:29. 964608 z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files. It'll take about three minutes to recover on its own

Some people say it is caused by NFS lock. How to solve it under JuicEFS

What you expected to happen:

Mysql can start normally

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: 1.0.0-beta1 (2022-01-13 28a42cf6)
• Cloud provider or hardware configuration running JuiceFS: Kubernetes csi
• OS (e.g cat /etc/os-release): CentOS7
• Kernel (e.g. uname -a): 4.19
• Object storage (cloud provider and region, or self maintained): self maintained Minio
• Metadata engine info (version, cloud provider managed or self maintained): self maintained Redis
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
• Others:

[SandyXSD]

Do you put data files of MySQL in JuiceFS? Is there any other client running MySQL with the same data files?

[Nick-0314]

Yes, I put the mysql data files into JuicEFS, only this mysql client, mysql reported an error and started about three minutes later (it should have locked the files successfully), but three minutes is too long for our failover, and there is no guarantee that three minutes can be started properly every time

[SandyXSD]

It's probably caused by the crash of JuiceFS process (since you force restarted the host), which leaves some files locked by the previous session. The new session is unable to lock them until the old one is cleaned up (about ~5 mins later). You can check this log message to confirm. BTW if the old session is closed gracefully (say, by umount), the locks will be released immediately.

[davies]

@mytting We change to default session timeout from 5 minutes to 1 minutes by #1591 , so it will be OK when the stale session cleaned.