mathiash98
commented
1 year ago I'll vote for option 2 ZFS with 2x4TB SSDs in each node. Would be easy to manage and fairly easy to setup Then we are not limited to network speed, and can upgrade network lated
Suggestion for setup of 3 Node High availability cluster with proxmox on new servers!
Option 1: Replication
With this setup, we will set up replication jobs on each proxmox machine, this will allow for backup ever X minutes (15 maybe?)
Pros:
Simple No network speed bottleneck from services
Cons:
Multiple copies of each replication Transferring with little change Slow startup of clients
Option 2: ZFS mirror with replication
CORRECTION! This also requires replication, seems my research was bad xD
With this setup, the storage of each is mirrored to the 3 other machines.
Pros:
No network speed bottleneck as it writes to local and then to the remote version of the mirror Not so complicated (Replication is easier) 3 copies of data gives good redundancy Lets us use High availability (automatic startup of VMs if one node goes down)
Cons:
Not so efficient storage Might bog down network if large files are transferred between nodes
Option 3: CephFS
With this setup, we use CephFS to store data on multiple nodes in a clustered file system
Pros:
Little data duplication High performance (if upgrading from 1 Gbit network)
Cons:
Complicated to setup and maintain Might break and can cause data loss Speed is limited by network (We have to get 10Gbit network between nodes)
What should we buy with each solution:
I would recommend SSDs as HDDs are so 2010s
I propose these setups:
256/512 gig ssd for OS (proxmox) This is required on all solutions!
Each server only has 4 drive bays:
Option 1:
256/512GB OS SSD 3x2TB SSDs in Raidz1
This option does not allow for future upgrade but is cheaper now
Optation 2:
256/512GB OS SSD 2x4TB SSDs in Raidz1
This option leaves one drive slot free if we run out of space, i highly recommend this solution if we use ZFS mirror
Here are some drive suggestions:
4TB : https://www.proshop.no/SSD/Samsung-870-QVO-25-SSD-4TB/2864187
2TB: https://www.proshop.no/SSD/Lexar-NS100-2TB/3072224
I think buying cheap SSDs are fine in this case, as we would have a lot of redundancy (except if we go Ceph, then we should use better SSDs)
Old suggestions can be found below
==============================================================
Handleliste
Mål: Enkel å bruke NAS server for å muliggjøre NAS verting for kunder
SSD-er:
2 billige ssd-er på TB
https://www.proshop.no/SSD/Lexar-NS100-SSD-2-TB-internal-25-SATA-6Gbs/3072224
https://www.proshop.no/SSD/Patriot-2-TB-P210S2TB25/2868030
rundt 1500 per SSD. Vi trenger 8 av dem. 12000 til sammen for SSD-er til to maskintjenere.
HDD-er:
Vi tenkte først HDD-er med rundt 12-16TB hver, slik at vi kan drive fillagring for BSTV og samtlige andre studorger. I tillegg kjøpe et nytt NAS, kanskje med rundt 8 brønner.
Kostnad:
rundt 2200 for hver 8TB-HDD. Vi trenger 8 HDD-er, 6 for RAID og 2 for erstatning. Det blir 17600 for HDD-er. Et nytt NAS med 12 brønner vil bli rundt 20000.
Seagate Exos series 8TB https://prisguiden.no/produkt/seagate-exos-7e8-st8000nm000a-492588 2100kr deal.no
Brønner til fujitsu:
Raid controller skal visst støtte 8 disker allerede. det gjorde den ikke, den støtter bare 4. 2 brønner står uten tilgang til raid-kontroller.
Nettverkskort:
https://www.multicom.no/startech-10g-pcie-sfp-card/cat-p/c/p1002453666?ref=prisjakt
Prisene varierer mellom 700 - ca 2000. Vi trenger 3 nettverkskort som støtter SFP, én til hver maskintjener. Vi trenger også nok kabler til alle tilkoblingene mellm ubiquity-switcheren og maskintjenerne, som er 3.
Specs:
https://www.fujitsu.com/tw/Images/ds-py-rx200-S7.pdf
Fiberkabler:
Typen SFP-kabler er OM1-5, men helst OM5 siden de er den nyeste versjonen. Vi har en SFP-kabel liggende på serverrommet, men den er av typen OS1.
Konrad og Fergus har 2 fiberkort: 8 Gigabit
Gjertrud har ingen fiberkort