New bill teams and related improvements - [merged] #89

Closed alexander-haller closed 4 months ago

In GitLab by @mhxion on Apr 2, 2024, 15:52

Merges new-bill-teams -> dev

This PR doesn't introduce all the necessary changes to bill-teams command yet, as it is still WIP. I have made small but related and imported changes that I intend to discuss in this PR instead of opening multiple separate PRs. I will update the description and make a new review request when a new change has been concluded and pushed to the PR.

1. Restrict `host` URL for testing

Since I am using multiple elapi.yml configuration files, it slips my mind sometimes exactly which host URL I am targeting, namely if I am targeting a dev instance or the production instance! show-config alleviates that confusion but I still sometimes find myself running code often times without properly checking show-config first. I believe others (especially people writing automation scripts using elAPI) might find themselves in the same situation.

So, I introduce a simple and straightforward way to restrict running any code to some predefined host URLs. Examples:

• Validate `host` without restriction

HostIdentityValidator validation class was introduced in v1.0. HostIdentityValidator checks if given host and api_token in elapi.yml are without errors. E.g., if there's a typo in host or an expired api_token is being used, HostIdentityValidator will immediately throw a validation error and will stop elAPI from proceeding. The following is the most basic use-case:

from elapi.validators import Validate, HostIdentityValidator

validate = Validate(HostIdentityValidator())
validate()

• Validate `host` restricting to development instance only

We modify HostIdentityValidator to add support for this restriction feature.

from elapi.validators import Validate, HostIdentityValidator

validate = Validate(HostIdentityValidator(restrict_to="https://elabftw-dev-002.uni-heidelberg.de/api/v2"))
validate()

The above validation will throw validation error, if the detected host from elapi.yml file is anything besides "https://elabftw-dev-002.uni-heidelberg.de/api/v2". Now we can confidently run any code without having to worry about if we are accidentally making changes to production.

• Validate `host` restricting to multiple instances

We can also pass multiple URLs and a validation error will be raised, if detected host URL is not either of the given restrict_to URLs.

from elapi.validators import Validate, HostIdentityValidator

validate = Validate(
    HostIdentityValidator(
        restrict_to=[
            "https://elabftw-dev.uni-heidelberg.de/api/v2",
            "https://elabftw-dev-002.uni-heidelberg.de/api/v2",
        ]
    )
)
validate()

Note

This restrict_to parameter is mainly intended for when the developer is in testing phase. When the a script is ready and stable, the developer does not need to put any restriction in most cases. We also don't ship any stable build where restrict_to has been used!

2. `store-info`

store-info works and behaves the same way as we've discussed so far. elapi bill-teams store-info --help explains the basic usage. store-info accepts the following arguments:

--root-dir: The root of the directory where store-info will create the billing directory hierarchy.
--meta-source: Source path for owners information CSV file. Internally referred to as the 'billing metadata'. --meta-source is not required when --teams-info-only is passed, otherwise it is required.
--target-date: ISO 8604 'YYYY-MM' for which 'YYYY/MM' directory to store.
--teams-info-only: If passed, then only teams-info will be stored.Both --teams-info-only and --owners-info-only cannot be passed
--owners-info-only: If passed, then only owners-info will be stored. Both --teams-info-only and --owners-info-only cannot be passed.

• Basic usage

elapi bill-teams store-info --root-dir ~/bill-teams --meta-source <CSV file path to owners information> will store the information in the following structure:

  ~/bill-teams/
  └── YYYY/
      └── MM/
          ├── <YYYY-MM-DD_HHMMSS>_owners_info.json
          └── <YYYY-MM-DD_HHMMSS>_teams_info.json

• Fix mistake of previous month

Let's assume we've the following information stored for 2024-04 to 2024 2024-05.

  ~/bill-teams/
  └── 2024/
      └── 04/
          ├── 2024-04-10_201824_owners_info.json
          └── 2024-04-10_201824_teams_info.json
      └── 05/
          ├── 2024-05-10_210000_owners_info.json
          └── 2024-05-10_210000_teams_info.json

In May 11, 2024, at 10:51 AM, we find out we've made mistake in April 2024 in our owners information (metadata) collection. We would like to inject the corrected owners information into April's directory.

$ elapi bill-teams store-info --root-dir ~/bill-teams --meta-source <path to corrected CSV> --target-date 2024-04 --owners-info-only

The above will add a new owners_info JSON file inside 04 directory.

  ~/bill-teams/
  └── 2024/
      └── 04/
          ├── 2024-04-10_201824_owners_info.json
          ├── 2024-04-10_201824_teams_info.json
          └── 2024-05-11_105122_owners_info.json
      └── 05/
          ├── 2024-05-10_210000_owners_info.json
          └── 2024-05-10_210000_teams_info.json

The file name prefix 2024-05-11_105122 in 2024-05-11_105122_owners_info.json will indicate the upcoming generate-table sub-command that 2024-05-11_105122_owners_info.json is latest owners_info metadata file for April 2024. This way we can avoid having to manually edit the JSON files, and it gives us a Git-like clean version history. Though modifying the JSON files are always an option.

Notes

store-info sub-command makes sure that the given path to --root-dir is indeed a directory and not a file, if it can actually write inside given --root-dir, thanks to elAPI's library-wide path validator. E.g., --root-dir /foo will fail and show error message "--root-dir path '/' could not be validated! Data could not be stored in desired location." elapi will abort if --root-dir is invalid.
It is possible to pass more than just "YYYY-MM" to --target-date, but anything besides the year and month is ignored. E.g., if --target-date 2024-04-11T01:14:21.244220 is passed, the entire 11T01:14:21.244220 part will be silently ignored.
If both --teams-info-only and --owners-info-only are passed, store-info will show an appropriate error message and abort.

In GitLab by @mhxion on Apr 2, 2024, 15:53

requested review from @alexander-haller

In GitLab by @project_994_bot_1f89ebd5af89f5e987c71c650dd64895 on Apr 2, 2024, 16:22

added 1 commit

512d7a7c - Fix post command CLI documentation

uhd-urz / elAPI