rda-image-archive

This repository has contains developmental code that has been obsoleted by https://github.com/NCAR/rda-image-archive/.

overview

The software in this repository has three main tasks to accomplish:

1. To gather images into (at least) one repository.
2. To establish a common description framework for image metadata.
3. To provide bulk, programmatic access to image subsets.

To this end, this repository contains

• Rough instructions for image providers to "bundle" metadata with their images.
• A MySQL metadata schema and makefile.
• Jupyter notebooks to provide minimal working examples for both providers and archivists.

This repository does not yet contain the following features:

• A MySQL and PHP configuration to access images at rda.ucar.edu/i/<uuid>.
• A MySQL and PHP configuration to query images at rda.ucar.edu/i/<query>.
• Functionality to detect duplication images.
• A RELAX NG metadata validator.

documentation

I wrote this schema to capture the (rich) meteorological metadata available for (19th and 20th century) marine logbooks and to support (at least) two users: Kevin Wood (NOAA/UW.) and Philip Brohan (UK Met Office).

We'll now minimize details and abstract that schema I prepared for Wood and Brohan into a "lite" version.

arc | archive
pla | platform
doc | document
img | image
obs | observation

There are these (functional) dependencies

arc   pla
   \ /
   doc
    |
   img
    |
   obs

This Hasse diagram indicates, e.g.,

1. The table arc provides a required foreign key for records of the table doc.
2. The table img has a required foreign key for records of the table obs.
3. Each record in the table obs requires the foreign key for its unique parent record in img, and so too any unique record in img requires the foreign key for its parent in pla.

In the context of marine logbooks, the rules above specify to these assumptions and constraints:

1. Each document has an archive.
2. Each observation is from an image file.
3. Each image file is a scan of a meteorological document from some platform.

local installation

First, if one has a local mysql installation, I suggest initializing a test database images.

1. Clone the repository and create a new database.

   git clone https://github.com/coltongrainger/rda-image-archive.git 
   # option to pull from the master: sub `ncar` for `coltongrainger`
   cd rda-image-archive 
   mysql -e "create database images"

2. Create a configuration file mysql_args

   vi mysql_args

   with these three lines:

   [client]
   user=<my_user>
   password=<my_password>

   (These will be passed as arguments to mysql, see the Makefile.)

3. Now make the database.

   make

Outline of database schema (as of 2019-07-05)

We expect (at minimum) the following fields to be defined (or NULL, where appropriate) in order to ingest images into the images database.

• archive.host_country
• document.contact_person
• archive.notes
• platform.name
• document.id_within_archive
• document.id_within_archive_type
• document.record_type
• document.standardized_region_list
• document.colloquial_region_list
• document.start_date
• document.end_date
• document.rights_statement
• document.notes

Run make describe for a fuller description of each field. As of 2019-07-08, the console output reads:

$ mysql --defaults-extra-file=$ mysql_args images -e "describe archive;"
Field   Type    Null    Key     Default Extra
archive_id      smallint(6)     NO      PRI     NULL
name    varchar(100)    NO              NULL
host_country    char(3) NO              NULL
search_url      varchar(255)    YES             NULL
search_documentation    varchar(255)    YES             NULL
api_url varchar(255)    YES             NULL
api_documentation       varchar(255)    YES             NULL
notes   varchar(1000)   YES             NULL
$ mysql --defaults-extra-file=$ mysql_args images -e "describe platform;"
Field   Type    Null    Key     Default Extra
platform_id     smallint(6)     NO      PRI     NULL
name    varchar(255)    NO              NULL
$ mysql --defaults-extra-file=$ mysql_args images -e "describe document;"
Field   Type    Null    Key     Default Extra
document_id     smallint(6)     NO      PRI     NULL
platform_id     smallint(6)     NO      MUL     NULL
archive_id      smallint(6)     NO      MUL     NULL
id_within_archive       varchar(255)    NO              NULL
id_within_archive_type  varchar(255)    NO              NULL
start_date      date    NO              NULL
end_date        date    NO              NULL
contact_person  varchar(255)    YES             NULL
standardized_region_list        set('north_atlantic','south_atlantic','north_pacific','south_pacific','north_indian','south_indian','antarctic','arctic','mediterranean','black_sea','baltic_sea','persian_gulf','red_sea')     YES             NULL
type_of_record  varchar(255)    YES             NULL
rights_statement        varchar(255)    YES             NULL
colloquial_region_list  varchar(1000)   YES             NULL
notes   varchar(1000)   YES             NULL
$ mysql --defaults-extra-file=$ mysql_args images -e "describe image;"
Field   Type    Null    Key     Default Extra
image_id        char(32)        NO      PRI     NULL
document_id     smallint(6)     NO      MUL     NULL
relative_order  int(4)  NO              NULL
media_type      enum('image/bmp','image/gif','image/jp2','image/jpeg','image/png','image/tiff') NO              NULL
filesize        smallint(6)     YES             NULL
$ mysql --defaults-extra-file=$ mysql_args images -e "describe observation;"
Field   Type    Null    Key     Default Extra
observation_id  int(11) NO      PRI     NULL    auto_increment
image_id        char(32)        NO      MUL     NULL
is_cited_by_dataset_id  char(255)       NO              NULL
is_cited_by_dataset_id_type     char(32)        NO              NULL

Outline of database schema (as of 2019-06-25)

Second, to get a handle on the schema designed (as of 2019-06-25) for images, it is perhaps useful to see the output of make describe (one should run this command on their own to see updates to the schema since the time of this document's writing).

Printed here in gory detail, that reads:

mysql --defaults-extra-file=mysql_args images -e "describe archive;"

mysql --defaults-extra-file=mysql_args images -e "describe platform;" (truncated for space ...)

mysql --defaults-extra-file=mysql_args images -e "describe document;"

mysql --defaults-extra-file=mysql_args images -e "describe image;"

mysql --defaults-extra-file=mysql_args images -e "describe observation;"

Parent-child dependencies

Now, the required fields in the 5 tables above are based on my interpretation of the following parent-child dependencies.

archive   platform
    \     /
    document
       |
    image
       |
    observation

In the partially-ordered diagram of tables above, an entry in a lower table depends on metadata from its parent entry in each upper table. For example, an entry in the document table makes foreign key references to parent entries who are defined (at minimum) by the fields

• in archive, namely,
  • archive_id
  • name
  • host_country; and
• in platform, namely,
  • platform_id
  • name.

As a result (and of relevance to our immediate use case), to insert metadata for one (binary) image file in the table image (formerly called page), the following parent fields will have had to be previously defined.

Initial database ontology

The assumptions underlying the ontology so far (that is, all tables above and including the table image) are:

• Documents produce images;
• Platforms produce documents;
• Archives produce documents;
• Documents from a single platform could be housed in distinct archives;
• A single archive could be responsible for documents from distinct platforms.

The motivations underlying the ontology so far are:

• We aim to preserve metadata from an image's archive in order to maintain continuity of record.
• We want to document platform metadata as a means to qualify the instruments and methods of observation which determine an image's meteorological content.
• (And these two motivations are independent.)

I am open to feedback and criticism with respect to this ontology.

Minimal metadata

Having outlined a suggested installation, and some preliminary justification in support of dependencies chosen for the rda-image-archive's database schema, I would like now to propose a subset of fields that should be required as minimal metadata. We'll start with image files.

Image files

Ignoring surrogate keys (these are generated incrementally), I propose that an image file should be accompanied by, at minimum, the following metadata.

Observations

My initial impression is that an observation record (e.g., a transcription) should be accompanied by the following metadata.

Understandably, this requirement may be too much to ask for: time_after_image_start could be ill-defined, illegible, incorrect, or only available from deduction. I would appreciate feedback with respect to this level of temporal granularity for the observation table in the database ontology.

Recommended metadata

We proceed to describe the recommended metadata, beginning from the tables archive and platform and working down the child tables.

Archives

For archives, given that a document will have an identifier relative to its parent archive, I recommend providing links to the archive's catalog search and the archive's API (if one exists).

Platforms

For platforms, my initial recommendation is to include descriptions of meteorological instruments along with unit conventions. However, if either the unit conventions (e.g., degrees vs. cardinal directions) or the instrumentation (e.g., pre- vs. post- gyrocompass) vary with time, then these fields may need to be demoted to the document table.

Documents

For documents, I recommend metadata that refers to what NARA calls the "parent series", i.e., the collection-level class to which a given document belongs. This field is broken into the item and the value, to accommodate for different names^[Perhaps there is a SKOS concept to describe the parent-child relationship at the collection-level?] for the "parent series". I also recommend including a list of place names. This field would enable one to perform a reverse dictionary (fuzzy) search against the http://marineregions.org/gazetteer.php?p=webservices REST API, in order to associate documents with boundary polygons. The goal here is to enable queries for records by (fuzzy) georeference (e.g., within a certain radius, or a certain region). Lastly, I recommend a create_date indicating when a document file was created/imported/scanned into its parent archive.

In summary, it is recommended that a document includes metadata for:

• create_date as the date the document was moved an archive.
• relative_parent_item as the "type of parent object" relative to an archive.
• relative_parent_value as the "value of the parent object" within an archive.
• marine_region_description as a comma separated list of colloquial marine region names. (Could include seas, sandbanks, seamounts, ridges, bays, sampling stations, or ports.)

Images

For images, I highly recommend including file_size (in bytes) for the sake of providing metadata for users who would like to download images in bulk; this field is not required because it is trivial to deduce file_size programmatically.

Optionally, I recommend the following "fine resolution" metadata. (It has been mentioned that these fields are perhaps "out of scope" of metadata, and should properly be classified as data; nonetheless, I would prefer to provide fields at a fine enough granularity to enable georeferencing.)

The logic behind this section of the image table schema is to deduce the "virtual" field ut1_start_datetime from the MySQL command date_sub(local_start_date, interval local_time_zone hour_second), which gives the UT1 (solar) datetime at image start.

Observations

Because observations have fine (perhaps as fine as ~30 minute) time resolution, georeferencing metadata may be too much to ask a provider for. Again, however, my philosophy here is to provide fields at a fine enough granularity to enable time-series analyses.

The following fields are boolean indicators and are recommended.

• atmospheric pressure indicators

  • atmospheric_pressure_indicator

• temperature indicators

  • dry_bulb_temperature_indicator
  • wet_bulb_temperature_indicator
  • unspecified_air_temperature_indicator
  • sea_temperature_indicator

• wind speed indicators

  • wind_direction_indicator
  • wind_speed_indicator

• cloud indicators

  • cloud_form_indicator
  • cloud_direction_indicator
  • cloud_amount_indicator

Other recommended indicators might include

• humidity indicators
• weather indicators
• visibility indicators
• precipitation indicators
• ocean sea waves and swell indicators

but, as of 2019-06-25, these fields have not yet been implemented into the schema.

The finest level of granuality would be obtained from the following optional fields (with the end goal of determining a platform's location).

This concludes the discussion of required and recommended metadata. Again, I am open to feedback and criticism with respect to any part of the database ontology.

sources

The software in this repository was designed with reference to the following:

• https://github.com/riceissa/aiwatch/. Issa Rice.
• DataCite Metadata Schema Documentation for the Publication and Citation of Research Data. Version 4.2. DataCite e.V. https://doi.org/10.5438/bmjt-bx77
• US National Archives Catadocument API. Dominic Byrd-McDevitt.