Rebuild Entire Lab

[oceanfabreeze]

Failed datastore occurred November 19th at 6:19AM Eastern time.

All virtual machines immediately failed and went to an unrecoverable state.

I attempted to remove the datastore and re-mount but once I removed it disappeared completely. One of the 500gb drives seemed to intermittently display in the ESXi dashboard, and when it does show the filesystem is "Unknown".

Root cause pending, but initial investigation points to a failed Seagate Barracuda 7200 500GB drive that shared datastore1 with another drive of the same make/model. Attempting to format both drives and see if I can continue to use until I'm able to make it out to Micro Center and pick up some new drives or SSD's. I would use a flash drive but unfortunately I don't have one on hand with enough storage to feel comfortable storing VM's.

Lessons learned here is that I need to invest in a backup solution sooner rather than later. Losing all that data is rough. Synology RackStation was going to be my go-to for this task, but unfortunately cost kept me from purchasing and implementing. I was saving for other things in the moment...

Current tasks:

• [x] Try and recover data from hydrogen by importing pool into TrueNAS. (Movies, TV)
• [x] Rebuild Star (Plex)
• [x] Rebuild Transmission (Transmission, Sonarr, Radarr, Jackett, VPN)
• [x] Rebuild Pi-Hole
• [x] Rebuild VaultWarden (Portainer, NGINX PROXY)
• [x] Rebuild Automator
• [x] Re-Tunnel Personal VPN
• [x] Re-Implement login_duo.

[oceanfabreeze]

9QM42LKY formatted successfully in SAS controller. Trying next drive.

[oceanfabreeze]

Second drive formatted correctly. Not sure what the deal is here.

[oceanfabreeze]

Will set-up a way to backup datastore to midnight hopefully. While I save for the synology box.

[oceanfabreeze]

ESXi 7.0 U3 installed on one of the 500gb drives

[oceanfabreeze]

Up and configured. 2 separate datastore's this time in case of drive failure. Then we only lose some of the VM's.

[oceanfabreeze]

Data Pool Recovered. Thankfully we didn't lose any data from our TV or Movie Library.

Next up, rebuilding star. Downloading latest RHEL 8 image. Uploading image to OS1 Datastore.

[oceanfabreeze]

RHEL installed on star. Need to register in subscription manager, install plex, mount fileshares.

[oceanfabreeze]

Plex is back up. It has transcoded and processed all intro's for TV.

Installation of RHEL is complete on pihole and grabber(replacement for transmission).

[oceanfabreeze]

Pi-Hole is up and configured.

Spinning up ansible to replace automator and need to install services on grabber.

Depression is keeping me from making large strides with this work.

[oceanfabreeze]

Transmission installed, sonarr snap installed.

Radarr and Jackett and VPN + config is all that's left for grabber

[oceanfabreeze]

Radarr, Sonarr, Transmission, Jackett, OpenVPN all installed. Config will need to be done at some point but for now I'm satisfied.

[oceanfabreeze]

openvpn file downloaded and added to grabber. transmission configured.

[oceanfabreeze]

Importing movies into radarr. Primary indexer down at the moment, but otherwise jackett is good to go. Next is sonarr.

[oceanfabreeze]

Sonarr configured.

[oceanfabreeze]

ESXi is experiencing issues again. Really need to replace these drives....maybe one of the parents can go to microcenter and do the swap for me...

[oceanfabreeze]

OS1 Datastore failed again. Lost Plex, Hydrogen, Ansible.....

[oceanfabreeze]

Downloading VM's from OS2 in case.

[oceanfabreeze]

2 Inland Professional 512GB SSD's installed to replace the Seagate Barracuda's. Also installed two more 4TB Ironwolf for future media expansion.

Hydrogen is rebuilt and running. Some config needed for the shares but otherwise it's straight.

Ansible, webby, pihole currently installing RHEL. RHEL installed on star.

Need to upload vmdk for grabber and hope for the best as far as virtual machine creation goes.

[oceanfabreeze]

Grabber imported. Taking a mental health break.

[oceanfabreeze]

Running upgrade on all servers.

[oceanfabreeze]

Plex is running on star.

[oceanfabreeze]

Servers were shut down due to power usage concerns. So work had to stop

[oceanfabreeze]

Upgrades will have to be issued on all the VM's due to Power/Downtime

[oceanfabreeze]

Beginning Rebuild Today. Have to make sure everything is ready for the new apartment/house.

[oceanfabreeze]

All systems resubscribed running updates.

[oceanfabreeze]

Upgrades complete. Shutdown all systems to prepare for reimplementation of Automation and Duo

[oceanfabreeze]

Homebridge will need to be rebuilt as well. #16

[oceanfabreeze]

Grabber currently hosting VPN? Maybe move to unifi controller? #26

[oceanfabreeze]

Entire lab is almost rebuilt. Just need portainer and web hosted services.

[oceanfabreeze]

Vaultwarden is up! We are resolved! Duo and other security will need to be implemented but WOOOOOOOOOOOOOOO.