This is a bit odd :) - Githubissues

So I like Bluefin and the idea of the rpm-ostree model but I found it has a lot of Linux-y quirks to be a daily driver. Do you mind explaining your approach as I saw you deleted the first iteration just as I was copying it and adding my own keys/changing my name, etc.

It looks like you're just setting this custom build up to copy your machine keys and get access to a a private repo that actually does the install. Is the idea to use this eventually as a "package manager," for lack of better terms, to adding things to Bluefin? I saw you went through a couple iterations of trying to get nix on Bluefin and from experience trying to get anything outside of flatpack or brew installed is doable but difficult.

Also how are you using Bluefin? With the graphical interface or just through the terminal? I assume you have your actual repositories stored on a NAS and this is "headless"? I work on a lot of different projects (as it looks like you do too), so I have to be pretty flexible in how I do things and Bluefin seems to basically have everything I need to just get anything to "work" ... would be very interested in how you're using it.

I have a few other questions about your ops workflow especially at home that might be better just messaging you. Thanks! :)

Hi!

Yes, I'm a bit surprised someone is looking in here 😅 No worries, happy to share what I have going on, but it might be a wall of text!

First, some history: I previously used NixOS. It became problematic over time because NixOS is incapable of handling binaries dynamically linked outside the Nix store (which basically means all foreign binaries). It affected several of my workflows, most notably using VSCode Remote (which copies its own version of node to the remote machine).

So I decided to switch to Bluefin to maintain a somewhat "immutable" system without the pain that NixOS brings with its opinionated design. The actual machine that Bluefin is installed on is a desktop that sits in my office. I don't use the GUI (although having it as a backup is nice) and instead do all work on it remotely. I have a Macbook Pro as well as a Mac Studio which are my two daily drivers. For most tasks, I use VSCode Remote in order to connect to the desktop. I often also have other terminals open, but again everything goes through the office (Bluefin) machine.

My goal for these types of systems is that I can blow them away for any reason and have them back in working condition in as little time as possible. That's what most of the work you've seen is attempting to do. The current flow is as such:

Generate an ISO file if not available
Boot the ISO on the office machine and run through the installer
The system boots to a static IP address to which I have a public DNS record pointing to. This makes it easy to find the system.
I run the bootstrap.sh script from either one of my macs which already have gopass configured on them. The script gives the office machine its SSH key (which is needed for fetching from git).
The bootstrap.sh script automatically connects to the newly created machine at which point I execute the setup.sh script
This script currently does a number of things, but it's all mostly related to getting the system configured. The biggest thing it does is applies my home-manager configuration which handles the bulk of the configuration process (installs all tools, creates appropriate dotfiles, starts some custom user services, etc.).
After running the setup.sh the system is basically "ready to go"

I do happen to have a NAS that is running Syncthing on it. This is where all of my project code is stored. When home-manager configures the system it automatically creates a service which starts Syncthing. After bootstrapping, I just need to configure Syncthing to point to the NAS and after a bit of time I'll have all of my projects synced to the machine. That pretty much is the last step as it's all I need to get back to working again.

I hid the other repo because I decided I needed to put some sensitive details in there related to my work. However, I'll paste the setup.sh with a few things omitted so you can get an idea of what I'm doing:

#!/usr/bin/env bash

####################################################################################
# This script is intended to be run on a new VM to bootstrap the DevOS environment #
####################################################################################

# Install Nix
echo ">>> Installing Nix..."
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

# Install Devbox
echo ">>> Installing Devbox..."
curl -fsSL https://get.jetify.com/devbox | bash

# Pull GPG key
echo ">>> Configuring GPG..."
gpg -k # Generate .gnupg
rm ~/.gnupg/common.conf
echo "pinentry-program /usr/bin/pinentry" >~/.gnupg/gpg-agent.conf
gpg --keyserver keys.gnupg.net --recv-keys 123456789

# Change default shell to zsh
echo ">>> Changing default shell to zsh..."
chsh -s /bin/zsh

# Switch to home-manager
echo ">>> Switching to home-manager..."
# shellcheck disable=SC1091
source /etc/bash.bashrc
rm ~/.zshrc
nix run home-manager switch --flake /home/josh/.devos#josh@devos

# Configure gopass
echo ">>> Configuring gopass..."
# shellcheck disable=SC1090
source ~/.zshrc
gopass clone git@github.com:jmgilman/gopass.git

# Add SSH keys
echo ">>> Restarting pcscd..."
sudo systemctl restart pcscd

echo ">>> Adding SSH keys..."
rm ~/.ssh/authorized_keys
gopass show ssh/macbook-pub >> ~/.ssh/authorized_keys
gopass show ssh/studio-pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

echo ">>> Upgrading system..."
sudo rpm-ostree upgrade

echo ">>> Restarting in 5 seconds (Ctrl+C to cancel)..."
sleep 5
sudo reboot

Ah that makes so much more sense. Yeah Bluefin would work great as a server but it has a long way to go to be immutable on a laptop. I was going to go your route but due to construction which has taken years I had to go from fiber to DSL. I wanted to use something to quickly add services to Bluefin as Homebrew + Flatpak don't cut it. I had no problems using Home Manager to manage global state and now I'm seeing there's other issues I didn't even think about over SSH.

I really liked how Nix switched environments and the package manager. What are you using now? I mean the amount of packages, pre-commit, post-commit, linting, etc. really helped keep my large monorepo projects working. As much as I could get basically builds running through the suite of tests locally the better as sometimes an outside vendor would be required to setup the actual CI/CD pipeline which .... usually skipped things instead of fixing them.

Not really a big environment but it'd be Typescript, Golang, Python and maybe some ML. So if not Nix what are you doing? There's Earthly/Cue/Dagger and a bunch of other things. Trying not to chase the latest hippest thing but was just curious. I don't want to go back to Makefiles and bash scripts, feels like there needs to be something holding them all together even if it isn't perfect. If I can't control the CI/CD environment for whatever reason I need a way to get this linting, pre-commit, post-commit, etc that Nix provided along with environment switching. The projects aren't big enough to warrant Bazel it is just as I'm sure you're aware if you are forced to work with outside vendors instead of sticking to Python throughout they'll use whatever it is they know how to do which leads to unnecessarily large amounts of languages.

It's a hard problem space, for sure. For total transparency, my company heavily uses Nix for just about everything (up to including IaC and even generating monitoring). Most engineers treat it like the world's largest glue gun and use it to generate JSON, HCL, YAML, you name it. When I got to my current position, I quickly realized the complexity created by all this severely delayed delivery. In fact, I was already hated on day one for having the title of SRE because, apparently, the team was seeing 6-month+ delays due to operational holdups.

I quickly pivoted away from the Nix monstrosity into something more "cloud-native," and that was understood by a broader group of people. That meant moving to Kubernetes (away from a Nomad/Consul frakenstein) and Terraform (using Terragrunt) with vanilla HCL. For deployments, we stuck with Nix for awhile, but developers absolutely hated writing packages for it (and I don't blame them to be honest). For NodeJS/TS and Python, Nix packaging is especially painful.

So, I moved the team to Earthly, and they've been much happier. To be fair, there are a number of tools out there that could have improved things, so I don't want to sell Earthly as something more than it is. You can see the final product of my efforts here. It was also recently highlighted by the Earthly team.

I've actually extracted most of our tech stack into a self-starting system that will likely be open-sourced soon. I'd be happy to come back and share it once that happens.

Ah that took me a bit to understand what you were doing but it conceptually is what I’m trying to do. We are working on a different set of apps but we could have multiple deployments out (depending on traffic really, to get a good sample set). This might bd one service or a cluster of services. Rarely do we get errors it is more like what deployment is matching certain KPIs and metrics they steadily advance that to more and more visitors. Seasonality also impacts this, like Black Friday might have essentially have a different application all determined through statistics.

Also enough stakeholders are involved in the process (designer uploads a PSD or figma to a Slack or equivalent channel, creates a new page in the design system, ephemeral environment corresponding to that, then they also become an approved of the feature request, etc.) sounds silly but if non-developers can interact through essentially a chat bot they’re familiar with they don’t need to know the intricacies of the environment.

I solved the NPM management problem by bringing in any “non-standard” libraries in source. I tried Deno and Bun and they work great but fall into the same traps. Usually if you have junior bootcamp developers they’ll just randomly pull stuff in and use one function. On large teams this is hard to control but it gets them thinking what am I using this package for.

Probably too much info but Earth was My next step. Too bad you have to use relative paths and not @base-earthfile like I think Turborepo treats local packages.

Great work! Awesome to see what others are doing in reality. I really need to open source my stuff.

jmgilman / devos

This is a bit odd :) #1