Storage LAN - Githubissues

clincha commented 1 year ago

When I have a clustered storage solution

I want storage communication to happen on a different network to my application communication so that one type of traffic doesn't bottleneck another
I want fast access to storage so that my applications are snappy
I want different tiers of storage to have different networks so that one type of traffic doesn't bottleneck the other
I want to continue using the servers that I currently have so that I don't have to spend lots of money to do this

clincha commented 1 year ago

I can buy these NICs which can give me up to two more Gigabit Ethernet ports to use.

Cost of one: £20.99 Cost of six: £125.94 Cost of eight: £167.92

clincha commented 1 year ago

That's not a SAN, this is a SAN...

Aggregation Switch: £261.27 3x Dual Port SFP+ 10GbE cards: £590.45 (£192.98) SFP+ to RJ45 (for router uplink): £17.99 6x SFP+ DAC cables: £71.04 (£11.99) 6x 2 port SATA expansion cards: £100.62 (£16.77)

£1041.37

The cost of safe, reliable and fast storage. Hot damn

clincha commented 1 year ago

I found a cheaper NIC which should hopefully work: https://www.techbuyer.com/uk/solarflare-10gbe-dual-port-server-adaptor-sf432-1012-lp-136066

Only £18 each which would drop my overall cost by £524.45 🤯

Total would be:

Aggregation Switch: £261.27 3x Solarflare 10GBE Dual Port Server Adaptor: £66 (£22) SFP+ to RJ45 (for router uplink): £17.99 6x SFP+ DAC cables: £71.04 (£11.99) 6x 2 port SATA expansion cards: £100.62 (£16.77)

£516.92

clincha commented 1 year ago

What about these SATA expansion cards too? That could mean an extra 12 SATA ports across the cluster.

However, that comes with a price increase of £115.32. I could get 3 to start and scale up when I need to. That would leave the cost around the same

clincha commented 1 year ago

Purchased the NICs from Techbuyer Purchased the Aggregate switch from Unifi Purchased the DAC cables and SATA cards from Amazon

clincha commented 1 year ago

Everything has been powered down now. There are a few jobs that need doing and I need to make sure the cluster can boot and be healthy after each one so that I can be sure which change caused the issue. There are some other changes I'm making at the same time while the cluster is down.

Increase RAM on all hosts (64GB/80GB)
New 10Gbps NICs
New HBAs
Upgrade to Proxmox 8

I'm hoping this will solve some issues I've been having that could be caused by slow i/o or not enough memory issues. Also it'll make the whole thing even more overkill :)

clincha commented 1 year ago

root@bri-s-03:~# free -h
               total        used        free      shared  buff/cache   available
Mem:            78Gi       1.6Gi        76Gi        61Mi       385Mi        76Gi
Swap:          8.0Gi          0B       8.0Gi

clincha commented 11 months ago

root@bri-s-02:~# iperf3 -c 10.1.2.11 Connecting to host 10.1.2.11, port 5201 [ 5] local 10.1.2.12 port 46310 connected to 10.1.2.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.44 Gbits/sec 0 1.22 MBytes [ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.22 MBytes [ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.22 MBytes [ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.28 MBytes [ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.28 MBytes [ 5] 5.00-6.00 sec 1.10 GBytes 9.41 Gbits/sec 0 1.28 MBytes [ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.28 MBytes [ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.35 MBytes [ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.42 MBytes [ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.50 MBytes

[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.0 GBytes 9.42 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver

iperf Done. root@bri-s-02:~# iperf3 -c 10.1.3.11 Connecting to host 10.1.3.11, port 5201 [ 5] local 10.1.3.12 port 49688 connected to 10.1.3.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.10 GBytes 9.44 Gbits/sec 0 1.20 MBytes [ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.26 MBytes [ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.26 MBytes [ 5] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.26 MBytes [ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.26 MBytes [ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.26 MBytes [ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.33 MBytes [ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.33 MBytes [ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.33 MBytes [ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.33 MBytes

[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 11.0 GBytes 9.42 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec receiver

iperf Done. root@bri-s-02:~# iperf3 -c 192.168.1.11 Connecting to host 192.168.1.11, port 5201 [ 5] local 192.168.1.12 port 55940 connected to 192.168.1.11 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 113 MBytes 951 Mbits/sec 0 250 KBytes [ 5] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 250 KBytes

[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver

iperf Done.

clincha-org / clincha

Storage LAN #82