Open Jonaswinz opened 2 years ago
Was the disk or network busy while the volume became read-only? From the longhorn-manager-e log, the amount of the data transfer was small. Looks like the network connections between the engine and replicas were not stable and cutoff sometimes.
I am trying to get data about the drive and network workload during the next failure. As I understand the logs, longhorn recoveres itselve after network abortion. And only sometimes I get the readonly problem. So what are the exect circumstances for the readonly problem? Does it happens, when the workload pod is trying to use the volume during recovering ?
Moreover, is it really a longorn problem? Or a problem with the workload pod, because I guess the remount whent successfully, but the workload pod unfortunatly uses the volume at the wrong time, and fails into readonly. But on the other hand the automatic restarts does not fix the problem. Only a complete recreation of the pod.
I dont know how to fix the issue. Is there some way to prevent it, or to recover it automatically.
In most cases, Longhorn can rebuild and recover after replica disconnection. However, once the filesystem goes into read-only mode, Longhorn cannot not automatically convert the volume into read-write mode. User must remount it withrw
by itself currently.
But on the other hand the automatic restarts does not fix the problem. Only a complete recreation of the pod.
Do you mean remounting the volume inside the pod still can't fix it?
Longhorn would help reattach the volume automatically after the crash. But it cannot directly do remount for the volume. Therefore, Longhorn introduces a setting for this case. You can check this doc for details: https://longhorn.io/docs/1.2.3/high-availability/recover-volume/
This feature is already enabled and the workload pods (mariadb) are managed by a statefull set, so it should work, but it does not.
This feature is already enabled and the workload pods (mariadb) are managed by a statefull set, so it should work, but it does not.
Similar issue https://github.com/longhorn/longhorn/issues/3325#issuecomment-989562860
Im getting this same issue with postgres-operator clusters and sqlite databases. Longhorn isn't reporting any issues but files migrate to read only.
@B1ue-W01f Does the volume get re-attached (automatically) before encountering the issue? If YES, you can wait for #3325 investigation.
Hi, my wordpress instances are facing a "Could not connect to the database" error once a day. They are all connected to one mariadb instance. When checking the mariadb logs I always saw a "readonly error". After restarting the pods, everything is fine again, for around one day.
I searched alot about longhorn volumes getting readonly, but I didnt found anything helpfull. After scanning though the logs, I noticed a lot of timeout and connection lost logs. But this happens more often, than the "readonly error". As discussed in other issues, I think my network is sufficient enought with 500mBits and sub 1ms ping between the nodes.
Strangely I have to scale the stateful set of mariadb down and up to resolve the error. The automatical restarts of the pods doesnt seem to solve the problem automatically.
Please share some thoughts.
Log or Support bundle
Here are the e and r manager logs of the managers running on the same node as the mariadb pod. I got the "readonly error" around 2022-02-08T10:13 - 2022-02-08T10:20
manager e.txt manager r.txt
There is also the event logs of the volume:
events.txt
And also the support bundle:
https://drive.google.com/file/d/1OE-hMN7sa_R6qMIgHKRZ0ziT4H-34QJQ/view?usp=sharing
Environment