AIPP

Parser for aeronautical information available online.

This gem incluces executables to download and parse aeronautical information (HTML, PDF, XSLX, ODS and CSV), then build and export is as AIXM or OFMX.

Thank you for supporting free and open-source software by sponsoring on GitHub or on Donorbox. Any gesture is appreciated, from a single Euro for a ☕️ cup of coffee to 🍹 early retirement.

Install
Usage
Regions
Storage
References
Development

Install

Security

This gem is cryptographically signed in order to assure it hasn't been tampered with. Unless already done, please add the author's public key as a trusted certificate now:

gem cert --add <(curl -Ls https://raw.github.com/svoop/aipp/main/certs/svoop.pem)

Standalone

Make sure to have the latest version of Ruby and then install this gem:

gem install aipp --trust-policy MediumSecurity

Bundler

If you're familiar with Bundler powered Ruby projects, you might prefer to add the following to your Gemfile or gems.rb:

gem 'aipp'

And then install the bundle:

bundle install --trust-policy MediumSecurity

Usage

AIPP parses different kind of information sources and converts them to different output formats depending on which executable you use:

Executable	Output Format
`aip2aixm`	AIXM
`aip2ofmx`	OFMX

The parsers are organized in three levels:

region            ⬅︎ aeronautical region such as "LF" (France)
└── scope         ⬅︎ scope such as "AIP" or "NOTAM"
    └── section   ⬅︎ section of the scope such as "ENR-2.1" or "aerodromes"

The following scopes are currently available:

Scope	Content	Cache
AIP (default)	aeronautical information publication	by AIRAC cycle
NOTAM	notice to airmen	by effective date and hour

To list all available scopes, regions and sections:

aip2aixm --list

See the built-in help for all options:

notam2aixm --help

Example: You wish to build the complete OFMX file for the current AIRAC cycle AIP of the region LF:

aip2ofmx -r LF

You'll find the OFMX file in the current directory if the binary exits successfully.

Regions

To implement a region, you have to create a directory lib/aipp/regions/{REGION}/ off the gem root and then subdirectories for each scope as well as for support files. Here's a simplified overview for the region "LF" (France):

LF/                         ⬅︎ region "LF"
├── README.md
├── aip                     ⬅︎ scope "AIP"
│   ├── AD-2.rb             ⬅︎ section "AD-2"
│   └── ENR-4.3.rb          ⬅︎ section "ENR-4.3"
├── notam                   ⬅︎ scope "NOTAM"
│   ├── AD.rb               ⬅︎ section "AD"
│   └── ENR.rb              ⬅︎ section "ENR"
├── borders
│   ├── france_atlantic_coast.geojson
│   └── france_atlantic_territorial_sea.geojson
├── fixtures
│   └── aerodromes.yml
└── helpers
    ├── base.rb
    └── surface.rb

⚠️	All paths from here on forward are relative to the region directory.

Parsers

Say, you want to parse AIP ENR-4.1. You have to create the file aip/ENR-4.1.rb which defines the class ENR41 as follows:

module AIPP::LF::AIP
  class ENR41 < AIPP::AIP::Parser
    depends_on :ENR21, :ENR22   # declare dependencies to other parsers
    (...)
  end
end

Another parser might target en-route NOTAM and therefore has to go to notam/ENR.rb like so:

module AIPP::LF::NOTAM
  class ENR < AIPP::NOTAM::Parser
    (...)
  end
end

⚠️	Parser files and classes may follow AIP naming conventions such as `ENR-4.1.rb`. However, you're free to use arbitrary naming for parser files like `navaids.rb` (e.g. if you're working with one big data source which contains the full AIP dataset you'd like to split into smaller parts).

The class has to implement some methods either in the class itself or in a helper included by the class.

Mandatory `parse` Method

The class must implement the parse method which contains the code to read, parse and write the data:

module AIPP::LF::AIP
  class ENR41 < AIPP::AIP::Parser

    def parse
      html = read             # read the Nokogiri::HTML5 document
      feature = (...)         # build the feature
      add(feature: feature)   # add the feature to AIXM::Document
    end

  end
end

Some AIP may be split over several files which require a little more code to load the individual HTML source files:

module AIPP::LF::AIP
  class AD2 < AIPP::AIP::Parser

    def parse
      %i(one two three).each do |part|
        html = read("#{aip}.#{part}")   # read with a non-standard name
        support_html = read('AD-0.6')   # maybe read necessary support documents
        (...)
      end
    end

  end
end

The parser has access to the following methods:

Method	Description
`section`	current section (e.g. `ENR-2.1` or `aerodromes`)
`read`	download, cache and read a document from source
`add`	add a `AIXM::Feature`
`find`	find previously written `AIXM::Feature`s by object
`find_by`	find previously written `AIXM::Feature`s by class and attribute values
`unique`	prevent duplicate `AIXM::Feature`s
`given`	inline condition for assignments
`link_to`	optionally checked Markdown link

Equally available is the current runtime environment. All of the following objects behave like OpenStruct:

Method	Description
`AIPP.cache`	cache to make transient objects available across AIPs
`AIPP.borders`	borders of the current region
`AIPP.fixtures`	fixtures of the current region
`AIPP.options`	arguments read from `aip2aixm` or `aip2ofmx` respectively
`AIPP.config`	configuration read from `config.yml`

To make the parser code more readable, a few core extensions are provided:

Mandatory `origin_for` Method

The class must implement the origin_for method which returns an origin object describing how to download the source data (e.g. an AIP file or NOTAM message):

module AIPP::LF::AIP
  class AD2 < AIPP::AIP::Parser

    def origin_for(document)
      # build and return the origin object
    end

  end
end

Return any of the following origin objects best explained by example:

AIPP::Downloader::File.new(
  file: "file.dat",   # relative path to file
  type: :pdf          # optional: file type if different from extension
)

AIPP::Downloader::File.new(
  archive: "foobar.zip",     # relative path to archive
  file: "subdir/file.dat",   # file to extract from archive
  type: :pdf                 # optional: file type if different from extension
)

See Downloader for more on recognised file and archive types.

AIPP::Downloader::HTTP.new(
  file: "https://example.com/foobar.zip",   # URL where the file is located
  type: :pdf,                               # optional: file type if different from extension
  headers: "Cookie: name=value",            # optional: additional headers e.g. for session
)

AIPP::Downloader::HTTP.new(
  archive: "https://example.com/foobar.zip",   # URL where the archive is located
  file: "subdir/file.dat",                     # file to extract from archive
  type: :pdf,                                  # optional: file type if different from extension
  headers: "Cookie: name=value",               # optional: additional headers e.g. for session
)

The excon gem is used to perform HTTP requests.

AIPP::Downloader::GraphQL.new(
  client: MyAPI::Client,       # GraphQL client class
  query: MyAPI::Name::Query,   # GraphQL query class
  variables: {                 # dynamic query parameters
    first_name: 'Geronimo',
    age: 50
  }
)

For this GraphQL downloader to work, you have to declare a GraphQL client class beforehand. See the graphql-client gem documentation for details, the following example fits the downloader above:

module MyAPI
  HttpAdapter = GraphQL::Client::HTTP.new(ENV['MY_API_URL']) do
    def headers(context)
      { "Authorization": "Bearer #{ENV['MY_API_AUTHORIZATION']}" }
    end
  end
  Schema = GraphQL::Client.load_schema(HttpAdapter)
  Client = GraphQL::Client.new(schema: Schema, execute: HttpAdapter)

  class Name
    Query = Client.parse <<~END
      query ($first_name: String!, $age: Int!) {
        queryNOTAMs(
          filter: {first_name: $first_name, age: $age}
        ) {
          name
        }
      }
    END
  end
end

For performance, all downloads are cached and subsequent runs will use the cached data rather than fetching the sources anew. Each scope defines a cache time window, see the table of scopes above. You can discard existing and rebuild caches by use of the --clean command line argument.

Optional `setup` Method

The class may implement the setup method. If present, it will be called when this parser is instantiated:

module AIPP::LF::AIP
  class AD2 < AIPP::AIP::Parser

    def setup
      AIXM.config.voice_channel_separation = :any
      AIPP.cache.setup_at ||= Time.now
    end

  end
end

Helpers

Helpers are mixins defined in the helpers/ subdirectory. All helpers are required automatically in alphabetic order.

Borders

AIXM knows named borders for country boundaries. However, you might need additional borders which don't exist as named borders.

You can define additional borders as AIPP::Border objects in two ways.

From GeoJSON

Create simple GeoJSON files in the borders/ subdirectory, for example this my_border_1.geojson:

{
  "type": "GeometryCollection",
  "geometries": [
    {
      "type": "LineString",
      "coordinates": [
        [6.009531650000042, 45.12013319700009],
        [6.015747738000073, 45.12006702600007]
      ]
    },
    {
      "type": "LineString",
      "coordinates": [
        [4.896732957592112, 43.95662950764992],
        [4.005739165537195, 44.10769266295027]
      ]
    }
  ]
}

⚠️	The GeoJSON file must consist of exactly one `GeometryCollection` which may contain any number of `LineString` geometries. Only `LineString` geometries are recognised! To define a closed polygon, the first coordinates of a `LineString` must be identical to the last coordinates.

From Coordinates

It's also possible to create a AIPP::Border objects on the fly:

my_border_2 = AIPP::Border.from_array(
  [
    ["6.009531650000042 45.12013319700009", "6.015747738000073 45.12006702600007"],
    ["4.896732957592112 43.95662950764992", "4.005739165537195 44.10769266295027"]
  ]
)

The coordinate pairs must be separated with whitespaces and/or commas. If you want to use this border everywhere, make sure you add it to the others:

  borders["my_border_2"] = my_border_2

Usage in Parsers

In the parser, the borders method gives you access to all borders read from GeoJSON files:

borders   # => { "my_border_1" => #<AIPP::Border>, "my_border_2" => #<AIPP::Border> }

The border object implements simple nearest point and segment calculations to create arrays of AIXM::XY which can be used with AIXM::Component::Geometry.

See AIPP::Border for more on this.

Fixtures

Fixtures are static YAML data files in the fixtures/ subdirectory. All fixtures are read automatically, e.g. the contents of the lib/aipp/regions/{REGION}/fixtures/aerodromes.yml will be available from AIPP.fixtures.aerodromes.

Read on for how to best use fixtures.

Patches

When parsed data is faulty or missing, you may fall back to fixtures instead. This is where patches come in. You can patch any AIXM attribute setter by defining a patch block inside the AIP parser and accessing the static data via parser.fixture:

module AIPP::LF::AIP
  class AD2 < AIP

    patch AIXM::Feature::Airport, :z do |object, value|
      throw(:abort) unless value.nil?
      throw(:abort, 'fixture missing') unless z = AIPP.fixtures.aerodromes.dig(object.id, 'z')
      AIXM.z(z, :qnh)
    end

  end
end

The patch receives the object and the value which is about to be assigned. It should implement something along these lines:

If the value is okay, throw(:abort) to leave the patch block without touching anything.
Otherwise, try to fetch a better value e.g. from the fixtures. If no better value can be found (e.g. outdated fixtures), throw(:abort, "reason") to leave the patch block and fail with a useful error message which contains the reason thrown.
At last, build and return the value object which will be assigned instead of the original value.

Source File Line Numbers

In order to reference the source of an AIXM/OFMX feature, it's necessary to know the line number where a particular node occurs in the HTML source file. You can ask any HTML element as follows:

tr.line

Errors

You should fail on fatal problems which must be fixed. The --debug-on-error command line argument will open a debug session when such an error occurs. Issue errors as usual:

fail "my message"

Warnings

You should warn on non-fatal problems which should be fixed (default) or might be ignored (severe: false). The --debug-on-warning ID command line argument will open a debug session when then warning with the given ID occurs. To issue a warning:

warn("my message")

Less important warnings are only shown if the --verbose mode is set:

warn("my message", severe: false)

Messages

info

Use info for essential info messages:

info("my message")                  # displays "my message" in black
info("my message", color: :green)   # displays "my message" in green

verbose info

Use verbose_info for in-depth info messages which are only shown if the --verbose mode is set:

verbose_info("my message")   # displays "my message" in blue

debug

The default Ruby debugger is enabled by default, you can add a breakpoint as usual with:

debugger

Storage

AIPP uses a storage directory for configuration, caching and in order to keep the results of previous runs. The default location is ~/.aipp, however, you can pass a different directory with the --storage argument.

You'll find a directory for each region and scope which contains the following items:

sources/
ZIP archives which cache all source files used to build.
builds/
ZIP archives of successful builds containing:
- the built AIXM/OFMX file
- build.yaml – context of the build process
- manifest.csv – diffable manifest (see below)
config.yml
This file contains configuration which will be read on subsequent runs, most notably the namespace UUID used to identify the creator of OFMX files.

The manifest is a CSV which lists every feature on a separate line along with its hashes, AIP and comment. You can diff or git diff two manifests:

$ git diff -U0 2019-09-12/manifest.csv 2019-10-10/manifest.csv

--- a/2019-09-12/manifest.csv
+++ b/2019-10-10/manifest.csv
@@ -204 +204 @@ AD-1.3,Ahp,9e9f031e,d6f22057,Airport: LFLJ COURCHEVEL
-AD-1.3,Ahp,9f1eed18,37ddbbde,Airport: LFQD ARRAS ROCLINCOURT
+AD-1.3,Ahp,9f1eed18,f0e60105,Airport: LFQD ARRAS ROCLINCOURT
@@ -312 +312 @@ AD-2,Aha,4250c9ee,04d49dc7,Address: RADIO for LFHV
-AD-2,Aha,6b381b32,fb947716,Address: RADIO for LFPO
+AD-2,Aha,6b381b32,b9723b7e,Address: RADIO for LFPO
@@ -664 +663,0 @@ AD-2,Ser,3920a7fd,4545c5eb,Service: AFIS by LFGA TWR
-AD-2,Ser,39215774,1f13f2cf,Service: APP by LFCR APP
@@ -878 +876,0 @@ AD-2,Ser,bb5228d7,7cfb4572,Service: TWR by LFMH TWR
-AD-2,Ser,bc72caf2,0a15b39c,Service: FIS by LFCR FIC
(...)

The advantage of git diff is its ability to highlight exactly which part of a line has changed. Check out this post to learn how.

References

Development

To install the development dependencies and then run the test suite:

bundle install
bundle exec rake    # run tests once
bundle exec guard   # run tests whenever files are modified

Please submit issues on:

https://github.com/svoop/aipp/issues

To contribute code, fork the project on GitHub, add your code and submit a pull request:

https://help.github.com/articles/fork-a-repo

License

The gem is available as open source under the terms of the MIT License.

svoop / aipp

readme

AIPP

Table of Contents

Install

Security

Standalone

Bundler

Usage

Regions

Parsers

Mandatory `parse` Method

Mandatory `origin_for` Method

Optional `setup` Method

Helpers

Borders

From GeoJSON

From Coordinates

Usage in Parsers

Fixtures

Patches

Source File Line Numbers

Errors

Warnings

Messages

info

verbose info

debug

Storage

References

Development

License

svoop / aipp

readme

AIPP

Table of Contents

Install

Security

Standalone

Bundler

Usage

Regions

Parsers

Mandatory parse Method

Mandatory origin_for Method

Optional setup Method

Helpers

Borders

From GeoJSON

From Coordinates

Usage in Parsers

Fixtures

Patches

Source File Line Numbers

Errors

Warnings

Messages

info

verbose info

debug

Storage

References

Development

License

Mandatory `parse` Method

Mandatory `origin_for` Method

Optional `setup` Method