Distantly client from master - different data centers

[siewa001]

Hi,

I'm running test setup with LizardFS in two data centers (A and B), the typical latency between them is ~10ms. The setup is the same in both DC: master (primary in DC A, shadow in DC B), chunkserver, client. I know that I can configure topology to force clinet B to talk to chunkserver B to improve performance. But what about communication between master and client? Because right now client B is talking to master in DC A and it dramatically slows down communication.

My questions:

• is there any way to improve this setup?
• does client B can talk to shadow server which is in DC B?

After tests, I wanna go with a similar setup to large scale but first I have to solve the geo-communication problem, so any input / concept will be appreciated.

[cratoo]

@siewa001 your issue is very interesting because we've got exactly the same issue. Two DCs, latency for us is even lower (0,3ms vs 1ms) and as soon as the client is not in the same DC as the master the communication is slower with a factor of at least 4x.

There is some other github issue mentioning multiple master/distributed metadata setups coming this year, but I've got no idea if it would actually help.

I also wanted to try out if NFS ganesha would fix it, but was not able to test it yet.

[cratoo]

What I wanted to add: someone I know is having basically the same setup, and also the DCs are very similiar (also in regards to latency) but he's using MooseFS and apparently he's not seeing such issues. I thought LFS was not modified that much especially in this regard, yet, or am I wrong about that?

[cratoo]

Nobody with a similar setup, or similar issues? Are we doing something wrong?

[borkd]

Maybe there is not enough detailed background info in your description on how you use LizardFS and deployment choices and tradeoffs you are comfortable with to start a meaningful conversation?

Consider how helpful would it be for the people who try to help you to know the following:

• Do you believe that you have found a bug or issue with the code itself, or are you looking for support with running the system?
• Have you pored over documentation, open and closed issues on Github, or maybe even source code trying to understand why the issue happens?
• Your general expectations - what problems do you expect LizardFS to help you solve, and on what scale?
• Which branch/version of code is in use?
• What was the issue you ran into? How can someone else reproduce it.
• What did you expect to happen instead?
• Describe your efforts to remedy the problem. What worked and what did not.

[cratoo]

@borkd to be honest, I thought there are enough details in this thread already to start a meaningful conversation (despite the LFS version I'm running of course). But in regards to your questestions:

• I have no idea if this is a bug or an issue with the code, so basically I'm starting with the question, if there might something be wrong with my setup.
• I read the documentation and tried different settings to remedy the problem, but nothing helped (chunk-sync and prefer local chunks to be precise).
• My expectation is/was, that LFS should have no problem in regards to using the master cross-location. Especially if the latency is in the range of 0,3ms-1ms. And as this latency was no problem, when using NFS with DRBD, and the client mounting the storage cross-location.
• Currently I'm running 3.10.4 from jessie backports (for all parts involved); I didn't see any relevant change in the releases after that version. And as said someone I know, running MooseFS in a very similar setup does not experience these issues.
• As said, my expectation was, that it should be irrelevant, if I mount LFS cross-location. At least the performance might be impacted, but not with a factor of 4x or even worse.
• Also as said, I tried:
  • configuring the master to prefer a local chunk (I mainly run 4 nodes, each having a master and a chunk, but also in a setup of separated masters/chunks the problem exists)
  • configuring topology, so that the master would respond with a local chunk, so a client from DC-A would get chunks from DC-A. Afterwards the client only had connections to local chunks, but the performance issue was the same, leading me to the conclusion that the client/master traffic being cross location is the problem.
  • configuring the chunks not to perform fsync, didn't change anything in regards to my issue.

I hope I could add more valuable information, to find out if this is an issue in my setup, something that can be fixed with configuration or an actual bug. If I can do something to help debug it, just tell me what you need me to do.