Rewrite import script

[stefandesu]

The current import process is inefficient, complicated, and not well documented (#24). A full rewrite of the import script is necessary. It consists of the following parts:

1. Good documentation!
2. A bash script for batch imports (similar to the current scripts/import.sh).
3. One Node script to import a file.
4. One Node script to (re)create database indexes (only has to run once in the beginning and if indexes are changed).

The import Node script could have parameters similar to the current import.js (but everything is in one file):

• --terminologies or -t:

  • imports a single file of terminologies
  • sets concepts and topConcepts properties (by searching for concepts of that terminology) either to [null] (if concepts/top concepts exist) or to undefined (see #49)

• --concepts or -c:

  • imports a single file of concepts
  • question: should this remove all existing concepts for that scheme before import? (would mean that concepts for a scheme could only be imported all at once)
  • adjusts entry in terminologies collection to include concepts: [null] and (if applicable) topConcepts: [null] (see #49)
  • post-import adjustments necessary
  • set narrower property to either [] or [null]
  • set properties for text indexes

• --concordances or -k:

  • imports a single file of concordances
  • removes all existing mappings of a given concordance
  • imports mappings for that concordance via URLs in distribution property
  • post-import adjustments necessary
  • adjust distribution property to jskos-server API URLs
  • recalculate extend of concordances (i.e. number of mappings)

• --mappings or -m:

  • imports a single file of mappings
  • pre-import adjustments necessary
  • calculate mapping identifiers
  • set concordance URL in partOf if necessary
  • set creator to creator of concordance if no creator exists

• --annotations or -a:

  • imports a single file of annotations

• --index or -i:

  • (re)creates indexes (if type is given, for that type; if not, for all types)

• Each object should be verified before import. Question: Should that object not be imported if verification failed?

• Also, each import is handled as a stream. While the stream is read, objects are written to the database in batches (e.g. of 10000 at a time) to reduce necessary RAM.

• Also, the script should provide meaningful output, i.e. progress indication as well as error messages (e.g. "validation failed for object in line X, not imported").

Any thoughts or improvements?

[nichtich]

Pleas have a look at jskos-cli. The import script should provide a similar calling syntax:

jskos-import [options] type [file]

where type is given by string, detected with guessObjectType:

jskos-import schemes terminologies.ndjson
jskos-import mappings mappings.ndjson

...

We only need to add object type Annotation, not detected yet.

question: should this remove all existing concepts for that scheme before import? (would mean that concepts for a scheme could only be imported all at once)

No, better provide an additional command to purge selected parts of the database.

Each object should be verified before import. Question: Should that object not be imported if verification failed?

In general never import invalid records. Question is whether to abort and possibly reset the import if one record fails. This would require to pre-validate the whole file before import. I added option --validate to jskos-convert, maybe it could be handles similar?. Short answer: by default just omit faulty records and emit an error message.

[stefandesu]

Thanks for the hint about jskos-cli, it makes sense to use a similar calling syntax!

Annotations are difficult because they aren't part of JSKOS and we're currently using the Web Annotation Data Model. Should annotations be added to the JSKOS spec, maybe implementing only the parts of the Web Annotation Data Model that are relevant to us?

[nichtich]

A small subset of Annotations should be added to JSKOS: https://github.com/gbv/jskos/issues/73

[stefandesu]

TODO:

• [x] Remove old import scripts
• [x] Clean up scripts directory
• [x] Add script for bulk import multiple files
• [x] Add option --format to import from an API

[stefandesu]

Except that Travis is currently failing (although the tests work locally), I'd consider the import script done for now. Note that to use the command jskos-import, you need to link it to your path (npm link) which only works for one jskos-server instance (also mentioned in README).