search

eth-cscs / manta

Another CLI for Alps
https://eth-cscs.github.io/manta/
BSD 3-Clause "New" or "Revised" License
14 stars 3 forks source link

Add pattern when searching for CFS configurations #70

Closed miguelgila closed 3 months ago

miguelgila commented 4 months ago

Description

At the moment, when we pass a CFS configuration to any of the commands in manta, it has to be a name that matches perfectly the desired configuration. I would like it to be so we can use a regex to the name to match multiple configurations; this could be useful for example when deleting configurations that are named similarly (like image-xxxx-r0.0.1 and runtime-xxxx-r0.0.1).

Use case 1

For example, this shows no configurations named "rigi", but there are several for the cluster Rigi:

❯ manta get configuration -n rigi
No CFS configuration found!
❯ manta get configuration -H rigi -l 3
+-------------------------------------------+----------------------+----------------------------------------------------+
| Config Name                               | Last updated         | Layers                                             |
+=======================================================================================================================+
| rigi-mc-compute-config-cscs-24.3.0.r0-0.1 | 2024-04-25T12:29:30Z | Name:     csm-packages-1.5.0                       |
|                                           |                      | Playbook: csm_packages.yml                         |
|                                           |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                           |                      |                                                    |
|                                           |                      | Name:     shs-cassini_install-cscs-24.3.0          |
|                                           |                      | Playbook: shs_cassini_install.yml                  |
|                                           |                      | Commit:   56fbae0c04d4a302fcf52539c5b69bdfab7ce66c |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-cscs-24.3.0                  |
|                                           |                      | Playbook: cos-compute.yml                          |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
|                                           |                      |                                                    |
|                                           |                      | Name:     csm-diags-compute-1.5.26                 |
|                                           |                      | Playbook: csm-diags-compute.yml                    |
|                                           |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                           |                      |                                                    |
|                                           |                      | Name:     sma-ldms-compute-1.9.14                  |
|                                           |                      | Playbook: sma-ldms-compute.yml                     |
|                                           |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                           |                      |                                                    |
|                                           |                      | Name:     cscs                                     |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   bda7b4b3c807879fe4252e9c817bc9f7fbc733d6 |
|                                           |                      |                                                    |
|                                           |                      | Name:     cpe-pe_deploy-cscs-24.3.0                |
|                                           |                      | Playbook: pe_deploy.yml                            |
|                                           |                      | Commit:   f00f0cece67b88e2f58d14179ea51f6262aaa5d1 |
|                                           |                      |                                                    |
|                                           |                      | Name:     slurm-site-cscs-24.3.0                   |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   87d075de5d33cf71d17df0a5c8456293e732530d |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-last-cscs-24.3.0             |
|                                           |                      | Playbook: cos-compute-last.yml                     |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
|-------------------------------------------+----------------------+----------------------------------------------------|
| rigi-mc-compute-config-cscs-24.3.0.r1-0.1 | 2024-04-25T12:29:36Z | Name:     csm-packages-1.5.0                       |
|                                           |                      | Playbook: csm_packages.yml                         |
|                                           |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                           |                      |                                                    |
|                                           |                      | Name:     shs-cassini_install-cscs-24.3.0          |
|                                           |                      | Playbook: shs_cassini_install.yml                  |
|                                           |                      | Commit:   56fbae0c04d4a302fcf52539c5b69bdfab7ce66c |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-cscs-24.3.0                  |
|                                           |                      | Playbook: cos-compute.yml                          |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
|                                           |                      |                                                    |
|                                           |                      | Name:     csm-diags-compute-1.5.26                 |
|                                           |                      | Playbook: csm-diags-compute.yml                    |
|                                           |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                           |                      |                                                    |
|                                           |                      | Name:     sma-ldms-compute-1.9.14                  |
|                                           |                      | Playbook: sma-ldms-compute.yml                     |
|                                           |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                           |                      |                                                    |
|                                           |                      | Name:     cscs                                     |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   9a32b3d86f9931502317aa9e58c81332c14b4d17 |
|                                           |                      |                                                    |
|                                           |                      | Name:     cpe-pe_deploy-cscs-24.3.0                |
|                                           |                      | Playbook: pe_deploy.yml                            |
|                                           |                      | Commit:   f00f0cece67b88e2f58d14179ea51f6262aaa5d1 |
|                                           |                      |                                                    |
|                                           |                      | Name:     slurm-site-cscs-24.3.0                   |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   84d6eed4a999be8db0ddd13ccfa8bf12bf5b976f |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-last-cscs-24.3.0             |
|                                           |                      | Playbook: cos-compute-last.yml                     |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
|-------------------------------------------+----------------------+----------------------------------------------------|
| rigi-mc-compute-config-cscs-24.3.0.r1-0.2 | 2024-04-25T12:29:42Z | Name:     csm-packages-1.5.0                       |
|                                           |                      | Playbook: csm_packages.yml                         |
|                                           |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                           |                      |                                                    |
|                                           |                      | Name:     shs-cassini_install-cscs-24.3.0          |
|                                           |                      | Playbook: shs_cassini_install.yml                  |
|                                           |                      | Commit:   56fbae0c04d4a302fcf52539c5b69bdfab7ce66c |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-cscs-24.3.0                  |
|                                           |                      | Playbook: cos-compute.yml                          |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
|                                           |                      |                                                    |
|                                           |                      | Name:     csm-diags-compute-1.5.26                 |
|                                           |                      | Playbook: csm-diags-compute.yml                    |
|                                           |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                           |                      |                                                    |
|                                           |                      | Name:     sma-ldms-compute-1.9.14                  |
|                                           |                      | Playbook: sma-ldms-compute.yml                     |
|                                           |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                           |                      |                                                    |
|                                           |                      | Name:     cscs                                     |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   8245d5278756a15cbaa7af1762e360770f46f36a |
|                                           |                      |                                                    |
|                                           |                      | Name:     cpe-pe_deploy-cscs-24.3.0                |
|                                           |                      | Playbook: site-cscs.yml                            |
|                                           |                      | Commit:   69567783eb3e4f0b10f4da93c6a465aac7c80c52 |
|                                           |                      |                                                    |
|                                           |                      | Name:     slurm-site-cscs-24.3.0                   |
|                                           |                      | Playbook: site.yml                                 |
|                                           |                      | Commit:   84d6eed4a999be8db0ddd13ccfa8bf12bf5b976f |
|                                           |                      |                                                    |
|                                           |                      | Name:     cos-compute-last-cscs-24.3.0             |
|                                           |                      | Playbook: cos-compute-last.yml                     |
|                                           |                      | Commit:   1d4719ca3cf020a0abb2aa92e52632312d075ab4 |
+-------------------------------------------+----------------------+----------------------------------------------------+

Use case 2

Imagine you want to delete the artifacts relative to the following:

|------------------------------------------------------+----------------------+----------------------------------------------------|
| image-wildhorn-mc-compute-config-cscs-24.3.0.r1-0.0  | 2024-05-08T12:34:40Z | Name:     nomad-orchestrator                       |
|                                                      |                      | Playbook: site-client.yml                          |
|                                                      |                      | Commit:   907a1dc5442824bd5276016786ed85be37b33321 |
|------------------------------------------------------+----------------------+----------------------------------------------------|
| wildhorn-mc-compute-config-cscs-24.3.0.r1-0.0        | 2024-05-08T12:34:44Z | Name:     csm                                      |
|                                                      |                      | Playbook: csm_packages.yml                         |
|                                                      |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     slingshot-host-software                  |
|                                                      |                      | Playbook: shs_cassini_install.yml                  |
|                                                      |                      | Commit:   4dead2f7ebd1080d6a65d4b374618cf727624215 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     uss                                      |
|                                                      |                      | Playbook: cos-compute.yml                          |
|                                                      |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     csm-diags                                |
|                                                      |                      | Playbook: csm-diags-compute.yml                    |
|                                                      |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     sma                                      |
|                                                      |                      | Playbook: sma-ldms-compute.yml                     |
|                                                      |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     uss                                      |
|                                                      |                      | Playbook: cos-compute-last.yml                     |
|                                                      |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
|                                                      |                      |                                                    |
|                                                      |                      | Name:     nomad-orchestrator                       |
|                                                      |                      | Playbook: site-client.yml                          |
|                                                      |                      | Commit:   907a1dc5442824bd5276016786ed85be37b33321 |
+------------------------------------------------------+----------------------+----------------------------------------------------+

Using a wildcard like *wildhorn-*compute-config-cscs-24.3.0.r1-0.0 would make it much easier than going config per config.

Masber commented 3 months ago

use case 1: The fact that manta g c -H rigi did not return any value seems like a bug. Which version did yo use? I can't reproduce it in master branch. Which version did you test this?

use case 2: Yes this is very useful indeed. I have q question: Do you want to use globbing or regex style to define the patterns?. I assume this functionality would need to be consistent and expanded across other entities (CFS sessions, BOS sessiontemplates, IMS images) at least for searching

Masber commented 3 months ago

This is an example:

New pattern argument added to the get configuration command, manta will filter CFS configuration names following glob strategy

$ manta g c --help`
Get information from Shasta CFS configuration

Usage: manta get configuration [OPTIONS]

Options:
  -n, --name <CONFIGURATION_NAME>
          configuration name
  -p, --pattern <CONFIGURATION_NAME_PATTERN>
          Glob pattern for configuration name
  -m, --most-recent
          Only shows the most recent (equivalent to --limit 1)
  -l, --limit <VALUE>
          Filter records to the <VALUE> most common number of CFS configurations created
  -o, --output <FORMAT>
          Output format. If missing, it will print output data in human redeable (tabular) format
          [possible values: json]
  -H, --hsm-group <HSM_GROUP_NAME>
          hsm group name
  -h, --help
          Print help

Example:

$ manta g c -p 'pilatus*'
+---------------------------------------------------+----------------------+----------------------------------------------------+
| Config Name                                       | Last updated         | Layers                                             |
+===============================================================================================================================+
| pilatus-mc-compute-config-cscs-24.3.0.r1-0.1      | 2024-05-10T12:29:18Z | Name:     csm-packages-1.5.0                       |
|                                                   |                      | Playbook: csm_packages.yml                         |
|                                                   |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     shs-cassini_install-cscs-24.3.0          |
|                                                   |                      | Playbook: shs_cassini_install.yml                  |
|                                                   |                      | Commit:   4dead2f7ebd1080d6a65d4b374618cf727624215 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cos-compute-cscs-24.3.0                  |
|                                                   |                      | Playbook: cos-compute.yml                          |
|                                                   |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     csm-diags-compute-1.5.26                 |
|                                                   |                      | Playbook: csm-diags-compute.yml                    |
|                                                   |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     sma-ldms-compute-1.9.14                  |
|                                                   |                      | Playbook: sma-ldms-compute.yml                     |
|                                                   |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cscs                                     |
|                                                   |                      | Playbook: site.yml                                 |
|                                                   |                      | Commit:   82fb51e1393408a8c30724e68af5e97831a586f6 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cpe-pe_deploy-cscs-24.3.0                |
|                                                   |                      | Playbook: site-cscs.yml                            |
|                                                   |                      | Commit:   12f23283fcc0ffa52e86b92a8f70b19e5883efc8 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     slurm-site-cscs-24.3.0                   |
|                                                   |                      | Playbook: site-cscs.yml                            |
|                                                   |                      | Commit:   95bd128a247ed656d68442fb441cc0bda1921542 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cos-compute-last-cscs-24.3.0             |
|                                                   |                      | Playbook: cos-compute-last.yml                     |
|                                                   |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
|---------------------------------------------------+----------------------+----------------------------------------------------|
| pilatus-mc-compute-config-cscs-24.3.0.r1-0.1-temp | 2024-05-16T07:58:35Z | Name:     csm-packages-1.5.0                       |
|                                                   |                      | Playbook: csm_packages.yml                         |
|                                                   |                      | Commit:   6423c550ea38a3b6befc4867ea7319157b48c554 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     shs-cassini_install-cscs-24.3.0          |
|                                                   |                      | Playbook: shs_cassini_install.yml                  |
|                                                   |                      | Commit:   4dead2f7ebd1080d6a65d4b374618cf727624215 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cos-compute-cscs-24.3.0                  |
|                                                   |                      | Playbook: cos-compute.yml                          |
|                                                   |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     csm-diags-compute-1.5.26                 |
|                                                   |                      | Playbook: csm-diags-compute.yml                    |
|                                                   |                      | Commit:   3b59bd64682b8e55c7f49c6317442b48cea6bb53 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     sma-ldms-compute-1.9.14                  |
|                                                   |                      | Playbook: sma-ldms-compute.yml                     |
|                                                   |                      | Commit:   09b922e6a608f273dd28dfbc9eaf089b355a56b8 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cscs                                     |
|                                                   |                      | Playbook: site.yml                                 |
|                                                   |                      | Commit:   b01d095766627418959dfe3382c0e39329d4c8dd |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cpe-pe_deploy-cscs-24.3.0                |
|                                                   |                      | Playbook: site-cscs.yml                            |
|                                                   |                      | Commit:   12f23283fcc0ffa52e86b92a8f70b19e5883efc8 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     slurm-site-cscs-24.3.0                   |
|                                                   |                      | Playbook: site-cscs.yml                            |
|                                                   |                      | Commit:   95bd128a247ed656d68442fb441cc0bda1921542 |
|                                                   |                      |                                                    |
|                                                   |                      | Name:     cos-compute-last-cscs-24.3.0             |
|                                                   |                      | Playbook: cos-compute-last.yml                     |
|                                                   |                      | Commit:   18da1ec8ea46d38b6ad9b4c28208ef8ed2a597f6 |
+---------------------------------------------------+----------------------+----------------------------------------------------+
Masber commented 3 months ago

done in v1.34.0