The arc-validate-package-registry (avpr) repository contains:
arc-validate
.arc-validate
to install and sync validation packages.Read more at avpr.nfdi4plants.org/about
This repo runs an extensive CI/CD pipeline on every commit and PR on the main
branch. The pipeline includes:
avpr-preview-index
on this repo's preview-index releaseAVPRIndex
and AVPRClient
libraries, as well as a docker container for the PackageRegistryService
web API.flowchart TD
setup("<b>setup</b> <br> (determines subsequent jobs <br>based on changed files)")
batp("<b>Build and test projects</b><br>any of [AVPRIndex, AVPRClient, API]")
tsa("<b>Test staging area</b><br>test all packages in the staging area")
sapc("<b>Staging area pre-publish checks</b><br>hash verification, prevent double publication etc")
nr("<b>Release (nuget)</b><br>any of [AVPRIndex, AVPRClient]")
dr("<b>Release (docker image)</b><br>API")
upi("<b>Update preview index</b><br>update the github release index json file")
ppp("<b>Publish pending packages</b><br>Publish packages to production DB")
setup --when relevant project<br> files change--> batp
setup --changes in the<br> staging area--> tsa
batp --when tests pass and<br> release notes change--> nr
batp --when tests pass--> dr
tsa --when tests pass--> sapc
sapc --when checks pass--> upi
sapc --when checks pass<br> and any new packages<br> are pending--> ppp
The package staging area is intended for development and testing of validation packages.
Files in this folder must follow the naming convention <package-name>@<major>.<minor>.<patch>.*
and contain a yml frontmatter at the start of the file. These files must additionally be inside a subfolder exactly named as the package name. This leads to a folder structure like this:
StagingArea
β
βββ some-package
β βββ some-package@1.0.0.fsx
β βββ some-package@2.0.0.fsx
β βββ some-package@2.1.0.fsx
β
βββ some-other-package
βββ some-other-package@1.0.0.fsx
βββ some-other-package@2.0.0.fsx
βββ some-other-package@3.0.0.fsx
As all reference implementations are written in F#/.NET, the only currently allowed file format for validation packages is .fsx
(F# script files). This can and will be expanded in the future.
Any change to a package in the staging area triggers the tests located at /StagingAreaTests, which are run on every package. (see also CI/CD pipeline). publishing packages to the preview index or the production registry is only possible if all tests pass.
In principle, packages can be published via 2 channels:
The preview index:
The pipeline includes a Update preview index
CI step that extracts metadata from the yml frontmatter of every .fsx
file in the staging area and (if tests and sanity checks pass) adds it to the avpr-preview-index.json
on this repo's preview-index release.
This process is done automatically for ervery package mewrged into main, and needs no manual intervention.
The production registry:
Packages set to publish: true
in their yml frontmatter will be published to the production registry database if they pass all tests and pre-publish checks.
Publishing a package to the registry is a multi-step process:
Suppose you want to develop version 1.0.0 of a package called my-package
.
my-package@1.0.0.fsx
file to the staging area in the folder my-package
.main
branch at any time.publish: true
to the yml frontmatter of the package file. This will trigger the CI to build and push the package to the registry once the PR is reviewed and merged.stage | availability | mutability |
---|---|---|
staging: development in this repo | version of current HEAD commit in this repo via github API-based execution in arc-validate CLI |
any changes are allowed |
published: available in the registry | version of the published package via the registry API | no changes are allowed |
Packages SHOULD be versioned according to the semantic versioning standard. This means that the version number of a package should be incremented according to the following rules:
Package metadata is extracted from yml frontmatter at the start of the .fsx
file, indicated by a multiline comment ((* ... *)
)containing the frontmatter fenced by ---
at its start and end:
(*
---
<yaml frontmatter here>
---
*)
You can additionally bind YAML frontmatter as a string inside your package. This is recommended because you can now re-use the metadata in your package code.
This binding must be placed at the start of the file to the name PACKAGE_METADATA
with a [<Literal>]
attribute exactly like this:
let [<Literal>] PACKAGE_METADATA = """(*
---
<yaml frontmatter here>
---
*)"""
further down in your package code, you can now extract and use this metadata. This for example prevents you from having to repeat the package name in your package code.
#r "nuget: ARCExpect"
#r "nuget: AVPRIndex"
let metadata = ValidationPackageMetadata.extractFromString PACKAGE_METADATA
let validationCases = ...
cases
|> Execute.ValidationPipeline(
metadata = metadata // use metadata to determine output paths and names instead of doing it manually
)
Field | Type | Description |
---|---|---|
Name | string | the name of the package |
MajorVersion | int | the major version of the package |
MinorVersion | int | the minor version of the package |
PatchVersion | int | the patch version of the package |
Summary | string | a single sentence description (<=50 words) of the package |
Description | string | an unconstrained free text description of the package |
Field | Type | Description |
---|---|---|
Publish | bool | a boolean value indicating whether the package should be published to the registry. If set to true , the package will be built and pushed to the registry. If set to false (or not present), the package will be ignored. |
Authors | author[] | the authors of the package. For more information about mandatory and optional fields in this object, see Objects > Author |
Tags | string[] | a list of tags with optional ontology annotations that describe the package. For more information about mandatory and optional fields in this object, see Objects > Tag |
ReleaseNotes | string[] | a list of release notes for the package indicating changes from previous versions |
Author metadata about the people that create and maintain the package. Note that the
Field | Type | Description | Mandatory |
---|---|---|---|
FullName | string | the full name of the author | yes |
string | the email address of the author | no | |
Affiliation | string | the affiliation (e.g. institution) of the author | no |
AffiliationLink | string | a link to the affiliation of the author | no |
Tags can be any string with an optional ontology annotation from a controlled vocabulary:
Field | Type | Description | Mandatory |
---|---|---|---|
Name | string | the name of the tag | yes |
TermSourceREF | string | Reference to a controlled vocabulary source | no |
TermAccessionNumber | string | Accession in the referenced controlled vocabulary source | no |
Prerequisites:
Advanced local dev functionality has only been tested on Windows with Visual Studio. For that, install the ASP.NET core workload including container features, which will enable running the Docker Compose
project in Debug mode.
The AVPRIndex
and AVPRClient
libraries are located in /src
and are intended for use in consuming applications.
To build them, just run dotnet build
in the respective project folders or build the arc-validate-package-registry
solution.
csproj
or fsproj
fileThe PackageRegistryService
project located in /src
is a simple ASP.NET Core (8) web API that serves validation packages and/or associated metadata via a few endpoints.
It is developed specifically for containerization and use in a docker environment.
To run the PackageRegistryService
locally, ideally use VisualStudio and run the Docker Compose
project in Debug mode. This will launch the stack defined at docker-compose.yml
, which includes:
PackageRegistryService
applicationpostgres
database seeded with the latest indexed packagesadminer
instance for database management (will maybe be replaced by pgAdmin in the future)In other IDEs, you can run the PackageRegistryService
project directly or adjust the stack, but you will need to either set up a local postgres database and configure the connection string in appsettings.json
accordingly or fine-tune the existing docker-compose file..
Currently, any change in src/PackageRegistryService
will trigger a release to the production registry. This is done by the CI/CD pipeline, which builds and pushes a docker image to the registry on every relevant commit to the main
branch.
This will move to a versioned release process in the future.
The PackageRegistryService
has a built-in Swagger UI endpoint for API documentation. It is served at /swagger/index.html
.
There are 2 solutions that contain test projects:
arc-validate-package-registry.sln
contains the test projects for the AVPRIndex
and AVPRClient
libraries, as well as future API and integration tests located in /tests
.PackageStagingArea.sln
contains the tests and sanity checks for all packages in the staging area.Run the tests with dotnet test
in the respective test project folders or on the respective solution.