search

IATI / js-validator-api

Pure JavaScript IATI validator implementation
GNU Affero General Public License v3.0
2 stars 1 forks source link

readme

Summary

Product Validator API Endpoints
Description Node.JS app that provides API end points to validate IATI XML files, is used by https://validator.iatistandard.org/
Website https://developer.iatistandard.org/
Related IATI/validator-services, IATI/validator-web
Documentation https://developer.iatistandard.org/
Technical Issues https://github.com/IATI/js-validator-api/issues
Support https://iatistandard.org/en/guidance/get-support/

IATI JavaScript Validator API

Deploy_To_Dev_Function_On_Push

Endpoints

See OpenAPI specification postman/schemas/index.yaml. To view locally in Swagger UI, you can use the 42crunch.vscode-openapi VSCode extension.

Prerequisities

Getting Started

  1. Clone this repository
  2. Follow instructions for nvm/node prerequisties above
  3. Setup the environment variables using the .env file (instructions below)
    • If running directly (option 1), this will mean setting up a redis database.

Option 1: running directly

  1. Run npm i to install dependencies
  2. Run npm start to run the Function locally using the Azure Functions Core Tools
  3. The API end points will be available on localhost:7071

Option 2: using docker

  1. Run npm run docker:start to run the Function inside a Docker container using docker-compose.yml to create a Redis instance.
  2. The API end points will be available on localhost:8080

Test setup

To test the basic setup, you can run the following command:

curl http://localhost:8080/api/pub/version

This should return a version number (change port if running directly instead of via docker).

To test the validation is working, obtain an IATI XML file, and run the following command (this is if running via the docker setup; change port to 7071 if running directly):

curl -X POST --data-binary @PATH_TO_IATI_XML http://localhost:8080/api/pub/validate

You should see something like (if the file is a valid IATI file):

{
  "valid": true,
  "fileType": "iati-activities",
  "iatiVersion": "2.03",
  "rulesetCommitSha": "2cd1a14f6c584000ae6d76e708c73ddcc12b6c40",
  "codelistCommitSha": "cf571799b909eb894c071dc364cd2a4dfbcd5d7c",
  "orgIdPrefixFileName": "org-id-fa56731af7.json",
  "apiVersion": "2.3.2",
  "summary": {
    "critical": 0,
    "error": 0,
    "warning": 0
  },
  "errors": []
}

If it has warnings or errors, you'll see them listed.

Environment Variables

Set Up

cp .env.example .env

Description

APPLICATIONINSIGHTS_CONNECTION_STRING

BASIC_GITHUB_TOKEN

REDIS_PORT=6379
REDIS_HOSTNAME=redis

VALIDATOR_SERVICES_URL=https://func-validator-services-dev.azurewebsites.net/api
VALIDATOR_SERVICES_KEY_NAME=x-functions-key
VALIDATOR_SERVICES_KEY_VALUE=

DATASTORE_SERVICES_URL=https://dev-func-datastore-services.azurewebsites.net/api DATASTORE_SERVICES_AUTH_HTTP_HEADER_NAME=x-functions-key DATASTORE_SERVICES_AUTH_HTTP_HEADER_VALUE= DATASTORE_SERVICES_IATI_IDENTIFIERS_EXIST_MAX_NUMBER_OF_IDS=5000

App config defaults (set in config/config.js)

   `APP_NAME`: IATI Validator API
   `VERSION`: process.env.npm_package_version
   `NODE_ENV`: process.env.NODE_ENV
   `APPLICATIONINSIGHTS_CONNECTION_STRING`: process.env.APPLICATIONINSIGHTS_CONNECTION_STRING
   `NS_PER_SEC`: 1e9
   `VERSIONS`: process.env.VERSIONS || ['2.01', '2.02', '2.03']
   `MAX_FILESIZE`: process.env.MAX_FILESIZE || 60

Adding New

Add in:

  1. .env.example
  2. .env
  3. /config/config.js

Import

import config from "./config.js";

let myEnvVariable = config.ENV_VAR

Linting and Code Formatting

Prerequisities

Info

Creating a new route

func new --name <routename> --template "HTTP trigger" --authlevel "Function"

AppInsights SDK

Unit Tests

There is a large set of Mocha unit tests for the Rulesets logic in ruleset-unit-tests.

Integration Tests

Running

Locally

In Docker container

Modifying/Adding

Integration tests are written in Postman v2.1 format and run with newman Import the integration-tests/dev-func-validator.postman_collection.json into Postman and write additional tests there Then once you are happy, export the collection and overwrite the file, then check into git.

Using files:

Deployment / Release / Version Management

https://github.com/IATI/IATI-Internal-Wiki#development-process

Customised Dependencies

xpath

IATI Resource Dependencies

Schemas

Schemas are stored statically in the application source code as they are versioned and rarely change.

./schemas/2.0X/*

The source is IATI-Schemas

Codelists

This repo provides the "Codelist Rules" for evaluating whether the codelists used by an IATI file being validated are correct. This application dynamically loads the codelist_rules.json file from this repo for each version-2.0X branch. This is also stored in Redis for caching.

Rulesets

This repo provides the "Rulesets" for various conditional and restraint rules for IATI data. This application dynamically loads the rulsets/standard.json file from this repo for each version-2.0X branch. This is also stored in Redis for caching.

Organisation IDs

The rulesets contain rules that require evaluation against a list of Publisher organisation identifiers, and valid organisation identifier prefixes. link

These are sourced from the following locations:

Known Issues

Line numbers for Schema Errors in activities longer than 65535 lines

Due to a known limitation in the core library libxml2 used to perform schema validation, if an activity has a schema error at a line greater than 65535, the value will not be accurate in the context.

If this is the case for your validation report, a message like so will be displayed in the context of the schema error:

At line greater than: 65537. Note: The validator cannot display accurate line numbers for schema errors located at a line greater than 65537 for this activity.