search

katharostech / docker-plugin_lizardfs

Docker volume plugin for mounting LizardFS
Other
19 stars 2 forks source link

LizardFS with docker plugin over VLAN (IPv6) #6

Closed wglanzer closed 5 years ago

wglanzer commented 5 years ago

My current server structure, that gives me headaches, looks like this:

Data_Srv1 ("fd4a:1ffe:1cc:858a::11"):
- LizardFS Master (with docker-compose)
- LizardFS Chunkserver (with docker-compose)
- Doc Server (with docker swarm)
Mgmt_Srv1 ("fd4a:1ffe:1cc:858a::1"):
- CI Server (with docker swarm)

Data_Src1 and Mgmt_Srv1 are connected over a VLAN (and the docker swarm cluster), only accessable by IPv6 addresses. So far so good - the VLAN is okay, I tripple checked that. The main problem I got is, that LizardFS only accepts IPv4 addresses - please correct me if I am wrong.

So to work around this problem, I created a ip4ip6 tunnel with socat, from my Mgmt to Data server: socat TCP4-LISTEN:9421,fork TCP6:[fd4a:1ffe:1cc:858a::11]:9421

So, now comes the strange: The Doc-Stack (on Data_Src1) is able to create new files, fill them with content and delete them afterwards. But the CI-Stack (on Mgmt_Src1) is not - file creation is ok, but if I want to fill the file with content, the whole ssh session begins to hang.

I think it has something to do with the chunkservers not reachable from the Mgmt_Srv1? Are there any better approaches, especially for docker (which was the main reason, I created this ticket in this project 😁 )? (The LizardFS Containers do not have to be in swarm mode - I only want to have one Data-Server atm)

wglanzer commented 5 years ago

Just saw, that you provide a swarm file - I'll try it out in the afternoon 👍

eleaner commented 5 years ago

it feels like the client can connect to the master but cannot to the chunkservers. As a result, you have files and folders created but no content gets stored. if I recall chunkserver registers itself to master with own IP then the master will inform client where to connect to store or fetch the data. In swarm you'll have to use host network, otherwise, it will connect with swarm internal IP, which is likely not seen by the client.. It all works good on IPv4, never seen it working on IPv6

You can see the registered IP of chunkservers in the cgi (that is assuming that it can show IP6)

wglanzer commented 5 years ago

@eleaner You are right, but maybe I can completely ignore the host-binding, and use the overlay swarm network. I think the original swarm file ( here ) provides a solution for this problem. Do you have any experiences with that?

eleaner commented 5 years ago

the file you are referring to did not work that great on swarm network (for starters cgi could not locate harddrives) I asked questions about that and got suggestions to move to host network I never looked back to hide the traffic I am using VPN between machines but all IP4

zicklag commented 5 years ago

Yeah, the swarm overlay didn't work too well, because the Docker plugin has to run on the host network, unless you do some extra goofy stuff to run the plugin as a normal container on the overlay. Running all of the LizardFS services in host network mode seems to be the best way to do it.

The problem that you are having about not being to write is almost definitely because the client can talk to the master but not to the chunkserver, like @eleaner said. The chunkserver will register with the master on its own IP address and the clients are expected to be able to reach the chunkservers on the same addresses that they join the master with.

I don't have any experience with IPv6 or socat, and I wouldn't know whether or not LizardFS supports IPv6.

wglanzer commented 5 years ago

@eleaner @zicklag Do you have any working example for swarm, where external clients can access the lizardfs master - with the network driver set to "host"?

zicklag commented 5 years ago

I don't have an example, but I know somebody that did successfully deploy it with the host networking mode. If you just need to know how to put a Swarm container on the host network checkout this stack. It is for SeaweedFS, not LizardFS, but it shows you how to run the swarm services in host networking mode instead of on the overlay.

wglanzer commented 5 years ago

Thank you for your patience. I think i fixed my problem with this workaround - not the best one, but a working solution:

Data_Srv1 connects via localhost - that is / was no problem.

Mgmt_Srv1 has iptable routings, that routes all traffic on 9419-9425 ports to localhost. This localhost traffic will be moved with socat from localhost to "fd4a:1ffe:1cc:858a::11". From there it will be redirected to the docker containers.

The docker plugin and lizardfs work like a charm now 😁

zicklag commented 5 years ago

Awesome! So glad you got it working. :tada: :+1:

wglanzer commented 5 years ago

If anyone has the same problem as I had, just try the following iptables rules, on all non LizardFS Hostsystems (like my Mgmt_Srv1):

# Redirect all specific IPs to the correct hosts
/sbin/iptables -t nat -A OUTPUT -d [MFSMASTER_IP] -j DNAT --to-destination [LIZ_HOST]
/sbin/iptables -t nat -A OUTPUT -d [CHUNKSERVER_IP] -j DNAT --to-destination [LIZ_HOST]

# MASQ all outgoing traffic, to ensure that the source ip address is set to a reachable one
/sbin/iptables -t nat -A POSTROUTING -o [VLAN_INTERFACE] -j MASQUERADE

So you do not need socat and it seems to work just fine 🎉