YANG Catalog

Overview

This repository contains YANG Catalog's REST API and the bulk of its internal infrastructure for processing YANG modules and extracting their properties. It also serves information to YANG Catalog's frontend and implements the functionality of the Admin UI.

YANG Module Processing

The scripts in this repository serve as a backend to add, update, remove and manage YANG module files in YANG Catalog. It is composed of:

scripts that run every day as a cron job
an API which lets users add, remove or find the modules they expect
scripts that parse new/updated modules from different sources and populate them into the YANG Catalog database

This repository works directly with the yangModels/yang repository. That repository contains modules structured by vendors (Cisco, Huawei and others) and SDOs (IETF, IEEE, MEF, BBF and others).

Parse and Populate

The most important directory in this repository is parseAndPopulate. This directory contains parsing scripts to parse all the modules of given directories. This gives us all the metadata of the modules according to draft-clacla-netmod-model-catalog-03. Parsed metadata is subsequently populated to Redis and ConfD databases. These databases are used for the search functionality of yangcatalog.org.

We can parse modules either with the --sdo option, which will go through a given directory and parse all of its YANG modules one by one, or without this option, which will try to find a platform-metadata.json file in the directory which contains paths to capability.xml files and parse all the modules according to those files with vendor metadata added.

To find all the modules with missing or wrong revisions, namespaces, imports, includes or modules that according to a capability.xml file should be in the folder but are missing, we can use the integrity script.

API

The API module runs as a UWSGI emperor vassal (using the yang-catalog.ini file) and contains several endpoints. Most of the endpoints serve to find modules in different ways. This is described deeper in the API documentation. If the user is registered, she/he can add, modify or delete modules under a pre-approved path. Once a user has filled in the registration form, one of YANG Catalog's admin users needs to approve the user using the Admin UI and give the user specific rights, so he is able to add, remove or update only certain modules.

Some requests may take a longer period of time to process. Because of this some endpoints will give the user a job-id. The user can check this job at any time to see if the job has been completed or is still in progress or failed during execution by using the job-id.

The YANG Catalog API is also used by some automated external jobs. When new vendor modules are merged into the YangModels/yang repository along with corresponding platform-metadata.json files, a job is triggered to populate them into the YANG Catalog database.

The backend also receives IETF Yang models every day and if there are any new drafts it will automatically populate the YANG Catalog database and update the YangModels/yang repository with all the new IETF modules if GitHub Actions pass successfully.

Jobs

There are several cron jobs that run every day.

statistics job which goes through all the modules that are in YANG Catalog and generates an HTML file which has information about what vendors' and SDOs' modules we have and the number of modules that we have.
resolve_expiration job that checks all the IETF draft modules and their expiration dates and updates its metadata accordingly.
remove_unused job that removes data on the server that is not used anymore.
user_reminder script that will be triggered once a month to show us approved user accounts and users waiting for approval.
In the ietfYangDraftPull directory there is one job.
1. pull_local clones the https://github.com/YangModels/yang repo, updates RFCs data with the latest data from the YANG-RFC.tgz, which is created during running of this script: https://github.com/YangCatalog/module-compilation/blob/develop/ietf_modules_extraction/run_ietf_module_extraction.sh. Then goes through RFC, experimental, IANA and OpenConfig modules data and populates YANG Catalog.
In the automatic_push directory there are two jobs.
1. ietf_push retrieves and adds new IETF RFC and draft modules to the https://github.com/yang-catalog/yang repository if there are any new/updated modules.
2. iana_push rsyncs modules from rsync.iana.org::assignments/yang-parameters/ and pushes new IANA modules to the https://github.com/yang-catalog/yang repo.
recovery script which pulls all the data from Redis and creates a json file which is saved on the server as a backup. If we lose all the data for some reason we can use this script to upload it back with no loss of data.
revise_tree_type reevaluates the tree type for modules that were previously of type nmda-compatible.
reviseSemver reevaluates the derived semantic versions of modules.

Messaging

Yang admin users are informed about every new module added to the YANG Catalog database in a Cisco Webex teams room and by email. Also, there are other emails and Webex messages sent for different events, they can be found in the message_factory module.

Installing

Pre-requisites

ConfD Premium has to be accessible

Backend Container

Since this is just a small part of the whole functional environment, you need to build using the docker-compose file from the deployment folder. Then the catalog_backend_api:latest image can be used to run a docker container where everything will start as it is supposed to.

API Documentation

See the README.md file in the documentation/ directory.

Fill the Redis database

Using backend/recovery/recovery.py --load --file /var/yang/cache/redis-json/<specific>.json for loading some specific backup, or using backend/recovery/recovery.py --load to load the latest backup.

NGINX Configuration

To be localized to your configuration.

        location /doc {
            alias /usr/share/nginx/html/slate/build;
        }

        location /api {
            rewrite /api(/.*)$ $1 break;
            include uwsgi_params;
            uwsgi_pass 127.0.0.1:8443;
            uwsgi_read_timeout 900;
        }

YangCatalog / backend

readme