search

LLNL / ATS

ATS - Automated Testing System - is an open-source, Python-based tool for automating the running of tests of an application across a broad range of high performance computers.
BSD 3-Clause "New" or "Revised" License
7 stars 5 forks source link

Machines as plugins #108

Closed jwhite242 closed 1 year ago

jwhite242 commented 1 year ago

This PR enables finding machines installed using setuptools' entry points plugin mechanism:

Sample toml file in a project with these machines (note project/package name is a dummy and not real):

[project]
name = "myats"
version = "0.0.1"
description = "MyATS Wrapper"
dependencies = [
    "ats",
]
requires-python = ">=3.8"

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
where = ["."]
include = ["myats"]

[project.scripts]
myats = "myats.myats:main"  # install console script to launch myats wrapper/driver

[project.entry-points."ats.machines"]
my_slurm_proc_sched = "myats.myAtsMachines.myAtsSlurmProcessorScheduled:MyAtsSlurmProcessorScheduled"

This will register the MyAtsSlurmProcessorScheduled custom machine as a plugin names 'my_slurm_proc_sched', which are found within importlib.metadata.entry_points(). These get installed into known groups that the machine import knows to look for, which is currently called 'ats.machines'.

Note, this does not address the separate config issue, using -1 as the default for the constructor of the actual base machine object inherited from. Our wrapper overrides this in our config anyway so this default works fine, but longer term I think this would pair quite well with a separate config as input type approach rather than having such cluster specific config/init settings baked into the machine source files using that custom header comment; as a bonus there'd be much less code churn to adapt to new clusters.

Jeremy

dawson6 commented 1 year ago

Hi @jwhite242 would like to test this and then get it merged. But it is on a fork or something rather than a branch. I know how to 'rebase' or 'merge' to main from branches. But do not know the method for doing so from a fork. Can you get this updated so we can test it. In particular, there are some line conflicts with the current repo 'main' branch that I noticed when I did a simple xxdiff of this file and the main branch. We need to keep the 'main' branch code but get this MR rebased or merged with it.