MonicaLemay
commented
2 years ago I need to amend the discovery of this defect. It has nothing to do with CNSA mmshutdown on a scale core pod. The trigger for this is to have s3 service with MD5 enabled and cosbench workload with mixed read and write while doing hashCheck=true.
[root@c83f1-dan4 ~]# cat /root/cosbench/workloads/RW_workloads/rand_read_write_delete_GB_1hour/s5001_GB_random_size_num_of_objects.xml
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="GBrandomsizenumobj" description="GBrandom read and write workload">
<storage type="s3" config="accesskey=QzhyXj9wVDH9DvnK97L9;secretkey=xTwEhfGB/wRXZbk7Ymsq2fYwl52AriraQPVzrZKF;path_style_access=true;endpoint=http://metallb2:80"/>
<workflow>
<workstage name="prepare_rGB">
<work name="prepare_rGB" workers="256" interval="10" rampup="0" type="prepare" ratio="100" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(1,4);objects=r(1,20);sizes=r(2,3)GB;hashCheck=true"/>
</workstage>
<workstage name="rGB_write_read_delete">
<work name="rGB_write_read_delete" workers="256" interval="10" rampup="0" runtime="300">
<operation type="write" ratio="50" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(5,6);objects=r(1,20);sizes=u(2,3)GB;hashCheck=true"/>
<operation type="read" ratio="50" config="cprefix=s5001b;oprefix=s5001o;osuffix=_rGB;containers=r(1,2);objects=r(1,20);hashCheck=true"/>
</work>
</workflow>
</workload>I will see that the cosbench workload fails. When I fails, a manual s3 cp times out, while s3 ls still works.
romayalon
guymguym
akmithal
rkomandu
Environment info
Actual behavior
This is not the same as issue 6930.
In this issue that I am opening, it is true that the node remained in the Ready state so I don't expect any IP failover. This defect is not about metallb IP's not failing over. In this defect, I/O was running to metallb IP 172.20.100.31 which is for node master1. On node master0 , in the CNSA scale core pod, (namespace ibm-spectrum-scale), mmshutdown was issued for just that node. The other nodes remained active and with the filesystem mounted. Master0 has metallb IP 172.20.100.30. There was no I/O going to that IP.
What was observed after mmshutdown on master0 was that all I/O going to 172.20.100.31 stopped. Because of issue 6930, there was no failover. That is fine and expected. But what is not expected is for all I/O to stop.
When mmshutdown was issued, the noobaa endpoint pods only error was Stale file handle
This error is a bit odd because it is on the endpoint pod that was for master0. Master0's metallb IP was 172.20.100.30. Cosbench workload was only set up for 172.20.100.31.
An additional observation is that s3 command for list will work but not for write.
All future PUT to 172.20.100.31 and 172.20.100.32, get a timeout (if I don't CTL-C) and the endpoint pods record a "Error: Semaphore Timeout" From 31 and 32, we can do GET and we can read from the Noobaa database. If we rsh into the endpoint pods for the IP's of 172.20100.31 and 172.20.100.32, we see that Spectrum Scale is still mounted in the correct place and we can write to it manually with touch file. So, this tells us that the IP's 31 and 32 are still alive and that the noobaa db is still online. It also tells us that the Spectrum Scale filesystem is still mounted and writable. The timeout on the subsequent PUT tell us that it makes a connection request but never gets a response.
The endpoint pods never restarted and they sill have their labels.
Also, In the scale core pod we run mmhealth node show -N all and we see that everything is HEALTHY, except of course on the one node that we did mmshutdown.
Something is obviously hung in the PUT connection but logs and noobaa health don't point to anything. When we issue mmstartup the PUT's still fail. The only way to recover is to delete the Noobaa endpoint pods and have new ones generated again.
I have been able to recreate this very easily so if it is required I can set this up on my test stand
Expected behavior
1.When doing mmshutdown on one node, it should not impact cluster wide I/O capability. It should not be an outage. If indeed an outage is expected, then mmstartup should recover I/O capability.
Steps to reproduce
More information - Screenshots / Logs / Other output
Must gather and noobaa diagnose in https://ibm.ent.box.com/folder/145794528783?s=uueh7fp424vxs2bt4ndrnvh7uusgu6tocd
This issue started as HPO https://github.ibm.com/IBMSpectrumScale/hpo-core/issues/659 Screeners determined that it was with Nooobaa. I have also slacked the CNSA team for input but have not heard back.