Archipelago Roadmap 2019: Beta1

[DiegoPino]

Archipelago 2019 Roadmap

This is our working enumeration of concrete tasks, per Component and Service, for public evaluation (and comments).

All tasks listed here are leading our full feature release end of 2019. Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.

There are a lot more things involved and many are already done and coded, but i wanted to have a list that was closer to features checklist than the actual code milestones we are managing internally. Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).

Strawberryfield

• Field Property exposure to Drupal strategies
  • [x] JSON KEY Provider (flattener)
  • [x] JSON Flatten Keys
  • [x] JSONPATH/JMESPATH
  • [ ] Entity Reference Casting Provider (Using UUID loading and configurable entity type)
  • [x] JSON stored Service Endpoints with extended logic (e.g HOCR)
  • [ ] Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.

JSON representation and enrichment

• Better File management (Better than Drupal)
  • [x] File referencing via UUID instead of via Entity ID
  • [x] Handle temporary files when moving from TEMP storage to PERMANENT
  • [x] Increment file usage count on new versions
  • [ ] Decrement file usage count on version removal
  • [ ] Change file usage on Delete, EDIT on existing active content and versions
  • [ ] Add Webform based UI managment (reorder, replace, delete) for files
  • [ ] File based Plugins callable by Webhooks
    • [ ] TECHMD
    • [ ] ZIP/UNZIP
    • [ ] Derivative for larger MEDIA (video and Sound)
    • [ ] Pronom Service/Preservation
  • [x] New JSON Service Architecture reference
  • [x] Deposit/save on Node save whole, selfs sustainable Strawberry JSON blob in S3/Minio/FileSystem
• [x] Keep track of Service and action on Ingest/edit using Activity Streams
• [ ] Add more agent information on our activity streams for provenance and tracking.

Webforms integration

• [x] Webform Driven UI Ingest with custom handler and widget
• [ ] Create a set of Demo Webforms that cover base of our GLAM source data needs
• [ ] Allow Webform Field Widget selection be driven by RDF type and permissions.
• [ ] Create new, better, LoD Webform elements
  • [x] WIKIDATA
  • [x] LoC
  • [x] WIKIDATA Agents with LD Roles
  • [ ] Viaf
  • [x] Getty
• [ ] Create Stub (temporary) WIKIDATA entities if query shows desired WIKIDATA entity does not exist upstream.
  • [ ] "publish" to wikibase functionality
  • [ ] Replace repo wide stub uri with official one once pushed.
  • [ ] Keep track on the stub who is referencing it is (bidirectional reference?)
• [x] Move Strawberryfield harvest Webform handler's logic to plugins
  • [x] Deal with as:images
  • [x] Deal with as:documents, as:video, as:sound, as:dataset elements
  • [x] Deal with as:models
• [ ] Allow anonymous submits to be converted into proper Nodes by Admin (Self deposit, crowd sourced metadata)
• [ ] Make Webform API Interaction more versatile for our use. Use as schema validator.
• [ ] Add JS to avoid main node CRUD to submit/validate embeded Webform as widget

Media Displays

• [x] Add expected mime/type output to Media displays. Allows to tag media displays as JSON, XML or HTML only.
  • [x] React to mime type to allow JSON or XML output to be downloaded too.
• [ ] Add new Data Views Plugin integration to allow Media Displays to preprocess values on views exposed as API endpoints
• [ ] Version Media Displays
• [X] Provide example Twig templates for
  • [x] MODS
  • [ ] DC and
  • [ ] JSON-LD

Field Formatters

• [x] Static IIIF Images
• [x] Open Seadragon IIIF Images
  • [x] Add thumbnail navigation
• [x] IABookreader IIIF Images
  • [ ] Integrate Flavor based Solr Search
• [x] Panorama via IIIF
• [x] Metadata up-casters
• [x] Metadata up-casters with download endpoint
• [x] Video (HTML5)
• [x] Audio (HTML5)
• [ ] Web annotations (IIIF)
• [ ] Complex nested structures (Whole graphs)
• [x] 3D! (Three + JSM)
• [x] Expose View Mode to JSON Type value mapping that triggers automatic View Mode Selection

API Ingest, Migration and backup

• [x] Strawberryfield Normalizer: expands JSON string as a JSON when exporting
• [x] Strawberryfield denormalizer: string-ify JSON when importing
• [ ] Wrap JSONAPI on a set of Drush script to (Strawberry Seeds)
  • [ ] Allow Single command line invoke files and node ingest
  • [ ] Create virtual field Entity "buckets" to allow Media to be ingested into those as links and routed to internal Strawberryfield elements (utility methods for ingest)
• [ ] AMI (Archipelago Multi Import) First iteration
  • [ ] API Source (Other repos, ContentDM, Solr)
  • [ ] Google Spreadsheets (same as IMI)
  • [ ] Complete Drush 9 integration
• [ ] Filesystem drop-and-forget ingest. You save a JSON file into S3, Archipelago creates entities and relationships.

Service Architecture (Strawberry Runners)

• [x] Develop webhook driven notification service for derivatives
• [ ] Document/deploy webhook triggers for minio S3 per mimetype
• [ ] Document/deploy webhook triggers for AWS S3 (via lambda) per mimetype
• [ ] Develope Shell processing using Symfony callables and user configurable for each case (rule system)
• [ ] Generate JSON reference-able Services for complex non descriptive metadata and data
  • [x] HOCR
  • [ ] TECHMD
  • [ ] Web Annotations
  • [ ] Tabular datasets
  • [ ] Transcripts (similar to Web Annotations, mostly dependant)
  • [x] Build slim Content entity that can be used to index natively that content into Solr via search API
  • [ ] Allow Services to be self explaining of its capabilities. only GET will be allowed
  • [x] Allow Drupal to discover hierarchically via EDISMAX

SEO and API

• [ ] Allow Media displays output to be embeded in HTML head for SEO
• [ ] Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
• [x] Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs. UPDATED
  • [x] Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

• [ ] Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
• [ ] Same but needs better UI for referenced Services and Media
• [ ] Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
• [ ] Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

• [ ] Sync Configurations and remove non used ones for minio branch / periodic for each Drupal release
• [x] Site-build and remove orphan blocks
• [ ] Add more utility views
• [x] Enable JSONAPI by default on minio branch
• [x] Create jsonapi user with jsonapi credentials for minio branch
• [x] Create basic scripts to automate Docker/Bash operations
• [x] Update AWS deployer to match minio including docs and Cloud Services integration

Batch Operations

• [ ] Bulk Batch Views JSONPATH plugin to
  • [ ] Replace existing JSON values
  • [ ] Add to existing Values
  • [ ] Respect data type casted values, (entities, file references)
• [ ] Bulk Batch Views MEDIA plugin to
  • [ ] Replace Media
  • [ ] Add Media
• [ ] Bulk Batch Views ACL plugin to
  • [ ] Replace ACL and inheritance
  • [ ] Replace ACL individual Control List Elements
  • [ ] Add ACL individual Control List Elements
• [ ] Integrate into Solr Results and Strawberryfield Taxonomy Term pages

Future roadmap

• [x] D9 readyness 😄
• [x ] Solr Cloud/ Consortial ensemble
• [ ] Native Wikibase/Wikidata publishing

Documentation:

• [ ] Devops and new repository deployers
• [ ] Migration to and from.
• [ ] Backup and restoring
• [ ] Permissions, access and ACLs.
• [ ] Metadata Professionals, JSON schema and schema-less. AS, DR and AP internal ontologies. UPDATED
• [ ] Metadata Professionals, Key concepts of Archipelago
• [ ] Metadata, Ingest and edit workflows.
• [ ] Displays, Formatters and Media Plugins (Twig)
• [ ] Views Integration (Solr and Blocks)
• [ ] Strawberry Field Exposed Keys and Plugins
  • [ ] Property Exposing strategies and configs
• [ ] Media Management
• [ ] Solr and Discovery
• [ ] Extending and Coding
• [ ] SEO

[DiegoPino]

Note: This is an update from our original https://github.com/esmero/archipelago-aws-demo/issues/6 which we will keep for sanity/not-breaking links purpouses there too. That repo (not this) will contain specific to AWS deployment strategies, not version specific, like specially production almost-ready and performance optimized configurations and some demos on Multi tenant, Solr Cloud based deployments too. More on that later.

[giancarlobi]

@DiegoPino really really an intensive good work! thanks a lot for this schema and for all your work.

[DiegoPino]

@giancarlobi team effort! I totally feel worn out but happy we have our horizons clear. What i would like to see/make/have someone make is to create an ISSUE in each repository for each of these tasks to link this list to our work. It can be multiple issues, or larger ones, but they should explain the extend of the work required and where we are at, so we can mention those in each pull and advance. Once that is done i can update this description and add the links. Also, i will classify each of this tasks according to the effort or complexity. I need to figure a good label system. Means i need help here.