research: what data science sys admins do

[gvwilson]

Suppose you had a room full of data scientists for a day. What would you teach them about systems administration, package management, dev ops, and everything else they need to know to get from "works for me on my machine" to "works for everyone else a year from now"?

Sean Aubin:

• installing software packages compatible with Nvidia GPUs and making them discoverable by Python.
• Fixing/maintaining CI pipelines.
• Debugging Temporal pipelines and their interaction with Anyscale clusters/environments.

Kristine Willis

• adding new users and updating the system

tyx

• Building R or Python envos for people with less sysadm skills.
• Lots of package/dependency management (w/occasional patches for lib API changes).
• Building of docker images, VMs sometimes.
• Setting up and keeping servers for a bunch of in-house apps running (basic DNS, LAMP+Nginx).
• Lots of performance profiling/pipeline optimization (fs tricks, core affinity etc.).
• Some debugging (strace level mostly).
• Basic DBA for some SQL-based things (not quite sysadmin job though).

nikkid

• I have admin credentials to our AWS account and own our redshift cluster.
• Lately I have had to migrate all our user configurations and subscriptions to comply with data lake infrastructure updates and I own our DMA and PII compliance processes.
• Currently trying to figure out security group configurations a lot of people I work with have gotten flagged for.

Mike Spencer

• Running my domestic network. Security, media sharing, backups, etc.
• The very occasional VM setup and management for work purposes.

Miles McBain

• Provision machines, install / update packages, debug system library version issues.
• Provision users for automations, set permissions, build container configurations, deploy containers.
• Schedule ETL jobs.
• Perform git surgery.
• Make backups.
• Provisioning and storing security keys

Lluís Revilla

• setting up a server
• configuring a nologin user for data transfer between our group and collaborators
• configuring websites and ssl certificates
• secure connections and block unauthorized logins
• set up RStudio server,
• update OS, R and other system wide software
• I had not to configure networks

Noam Ross

• Setting up shared HPC, including installing hardware (HDs and GPUs) into computers
• software environment and user management
• setting up AWS servers for deploying websites and shared resources
• setting up the office network when the pandemic prevented any any IT folks from coming in

Jarek Bryk

• installing software (bioconda), adding users, various checks of system use and space
• learning on myself how the job scheduler works

[defuneste]

Lot of already good stuff! I think I would start even "easier":

• What is a package manager how can I see dependencies and why does it matter.
• Moving stuff manually (Rsync, sftp, etc ..) then automate it with a bit of SSH
• Huge +1 on pwd manager and keyrings
• Editing: not panicking when vi is called (vim basic)

[lo5an]

I wonder if any of these would fit in with where you're going:

• Using tmux or screen to manage long-running commands on remote systems, using time to accurately track the execution time of long running commands, and using commands like htop to watch the memory and processor load of particular tasks.
• Guidance on secrets management in Python to help people keep their API keys and other secrets from ending up in version control. Maybe paired up with using configuration variables to support working with containers in dev/test/prod environments.
• A discussion of PEP-668, the differences between OS-managed and PIP-managed packages, the risks of malicious PyPI packages, and how to us tools like pip-audit to scan for vulnerability might be a good compliment to the suggested material on virtual environments and requirements.txt.

[jlin]

Many of the topics I wanted to highlight are already covered.

Stuff that I think deserves some additional time:

• system packagers vs pip (global and/or local pips) vs conda packaging: which to use, when to use them, and how to stop them from trampling one another

• brief intro to the idea of reproducible computing. Explain why it's not attainable (in real life circumstances) at the present moment, and how to get closer to the ideal state without going overboard.

• using the 80/20 Pareto principle on system administration. Prioritizing on the 20% that is needed to achieve 80% results is good enough.

Side topic

• scheduling tasks and working with long running tasks

[sellorm]

Focusing specifically on "data science sys admins" I'd want to cover the following:

• Choosing a Linux distro - what makes a good distro and why?
• Installing the chosen distro - include things like bare-metal vs cloud VM
• Configuring the chosen distro
  • Networking
  • Disk
  • Authentication (eg local users vs centralised identity management - like Active Directory)
• Accessing the server - ssh and ssh keys
• Working on the command line
  • navigating the file system
  • common command line tools
    • man pages, and "--help" text.
  • Editing files - This is essential for config files can can be incredibly burdensome for new admins (eg. how do I quit vim?!? 'nano' is probably a better choice for a new admin, but it's rarely installed by default and sometimes things open in vim anyway - what to do then?)
  • Shell scripting 101
  • root and sudo - who gets to do what and why?
    • Configuring sudo with "visudo"
• Configuration part two - optional software
  • installing new software from standard OS repositories using OS specific package management tooling
  • This could very a lot depending on the audience but might include things like:
    • A webserver?
      • might be a good opportunity to sneak in some stuff about certificate management which can be a real pain
      • As well as learn more about network "ports"
    • R and Python (though this might not be the recommended approach (why not?)
    • Additional command line tools that aren't installed as standard eg 'htop' and 'tree'
  • installing additional software not from the standard OS repositories
    • What's the difference?
    • when and why?
  • Configuring optional software - editing config files again
  • Managing "services" with systemctl
  • adding network shares and/or sharing a filesystem over the network.
  • Scheduling tasks with cron - user and root level examples.
• Long-term maintenance
  • Log files - where are they and what can we learn from them
  • what to back up, when and how?
    • cloud providers often provide snapshotting tools, but you don't have these in bare metal environments, though you might with an enterprise deployment of VMWare etc.
  • Keeping installed packages up to date
  • OS upgrades
  • Managing servers with a configuration management tool like Ansible - almost certainly too much when starting out, but even mentioning it now, might help in the future
• Where to find more info (perhaps the most important skill of all!)
  • finding error messages so you can search for them
  • Good search practice with error messages - eg. adding the distro name and version, enclosing the error in speech marks etc.
  • Search tips and useful sites for learning more eg. https://explainshell.com

That's all that comes to mind right now, but I'm sure I've missed lots.