Switch to a seperate NFS file server for user data

[moorepants]

From Richard:

First steps here are to install CentOS on a virtual machine and set up and learn about ZFS, here: https://zfsonlinux.org

[Miniland1333]

Would a standard NFS or a more distributed version like Gluster or Ceph be more suitable for production? Is the projected user load going to exceed the capabilities of single-node NFS?

[moorepants]

I think we are just going to use what Richard has and what he's familiar with. You are welcome to suggest and explain other options.

[moorepants]

@lux12337 Richard has asked Dean Lavelle to give you a hand setting this up. Get in contact with him. See the zulip convo.

[lux12337]

I met with with Dean and Mike, and we discussed the current setup of the cluster and what it needs in terms of NFS storage. The main variables that affect the decision of how to setup an optimal NFS server are:

• How many users we expect to be using the cluster and how fast it is gonna scale? ZFS can be optimized for scalability.
• How fast is our cluster network? ZFS is fast, but our network speed might be a bottleneck.
• How much storage per user do we need? They said 500MB seems small, but it might depend on the use case.

Some options that Dean thought could work for us are:

• ZFS setup with 4TB x 3 drives for parity, and 4TB x 8 drives for storage, equaling to around 32TB of storage. For optimal performance on a ZFS server, we shouldn't surpass 75% storage, leaving us with around 24TB of storage. Anything beyond 90% would significantly degrade performance. All data stored on the ZFS can be compressed, so we will be getting a lot more than 24TB of actual storage.
• Reuse the "Hyperserver", a retired NFS server, for our needs.

More discussion is needed to finalize the design decision, Dean will probably need to touch base with Richard again.

[lux12337]

@moorepants do we have an idea of how many users to expect after we launch?

[moorepants]

Probably not a lot :). But we should offer similar space specs as other Jupytehub installs. We also have to accommodate binder caching (however that works). There could be a binder cache per page on LibreTexts if users have custom environments. Maybe there will be 10 professors with an average class size of 50 this coming year as a basic number. I have no idea how things will grow or what people will do. We need to be able to accommodate that if it grows. Your numbers of 32TB and 24TB sounds like plenty to start.

[lux12337]

There is a lot of work to be done before the ZFS server can go live:

• [x] Update firmware/BIOS for all the hardware in the Hyperserver:
  • [x] Update motherboard bios(Supermicro X9DRH-7TF/7F/iTF/iF).
  • [x] Update the firmware for the HDDs. All 36 HDDs have been updated.
  • [x] Update the firmware on the LSI controller cards. The firmware for one of the LSI card was already up to date. The other we don't need for now?
  • [x] Update firmware on SSDs. Updated the firmware for 2 intel DC SSDs and over provisioned them to 70% capacity to improve performance and life expectancy. Updated the firmware for 2 Seagate enterprise SSDs.
  • [x] Update IPM firmware
• [x] Install zil cache using SSDs.
• [x] Install Centos 7 and ZFS v0.8.1
  • [x] Centos 7 with updates
  • [x] ZFS v0.8.1 dkms
• [ ] Install l2arc cache? Probably not necessary for now.
• [x] Setup pool of drives, hot spares, and caches in ZFS.
• [x] Setup the networking for ZFS.
• [x] Switch the cluster from the NFS on rooster to ZFS.

[lux12337]

Important Notes

• The SSD used are Data Center(DC) SSDs, they contain a super capacitor that will have enough power to complete a write in the case of sudden power failure.
• We should leave some empty space when using a SSD to improve performance with TRIM, and to have unused memory blocks available to switch to if a memory blocks fails.
• Create separate partitions from root for var and tmp, this prevents the root partition from crashing if var and tmp start taking up all the space. We should do this with all the nodes.

  ZFS Server Specs

• 256 GBs of DDR3 RAM
• 4U server with 36 bays
• 2 LSI controllers?(gotta confirm this)
• 2 power supply units

  Design

  After discussion today, we have come up with a final design for the ZFS server:

• 4 total stripes with raidz2, each stripe is compromised of 4 HDDs for data and 2 HDDs for parity. The total number of HDDs that can fail per stripe without bringing the server down is 2. 24 total drives(data + parity).
• 2 SSDs with RAID1 for the Centos OS.
• 2 SSDs for zil cache. This is a write cache that significantly improves the speed of writes to the ZFS server.
• (Optional)2 SSDs for l2arc cache. This is a read cache that improves the speed of reads from the ZFS server, not sure if necessary given the amount of RAM we have. Not sure if BinderHub will benefit from this?
• 2 HDDs hot spares. These will be setup such that if any storage drive breaks, a hot spare will be used to rebuild that stripe immediately with zero downtime.

[lux12337]

The ZFS server has been racked and fully setup, Kubernetes has been switched over to ZFS. JupyterHub is working with ZFS now.

[moorepants]

Really! Was there any data transfer required?

[lux12337]

Any user that had logged in previously will still have a persistent volume claim on rooster, but any new users that login from now on, kubernetes will use the ZFS for storage. So we should get those old users on the ZFS too, the easiest solution would be to reach out to those users and just let them know that they need to download any data that they want to keep, and then we can delete their old persistent volume claim on rooster so that the next time they login Kubernetes will use the ZFS.

[moorepants]

Is it possible for you to look at their claims and if there is nothing there, then we can just move them?

[lux12337]

Yes, I will do that now

[lux12337]

I have moved all the users that had nothing in their accounts to ZFS. These accounts still have something in them though:

• ktmikowicz@ucdavis.edu
• rebelford@ualr.edu
• sunghwan.martin.kim@gmail.com

[moorepants]

Can you email them and ask if they will download their files so you can switch them over?

[lux12337]

I have sent them an email.

[lux12337]

Migration status:

• [x] ktmikowicz@ucdavis.edu
• [x] rebelford@ualr.edu
• [x] sunghwan.martin.kim@gmail.com

[lux12337]

All users have been moved over to the ZFS server.