Archipelago Roadmap 2020: Beta2

[DiegoPino]

Archipelago 2020 Roadmap

See also #5

This is our working enumeration of concrete tasks for this year, per Component and Service, for public evaluation (critics and comments welcome).

All tasks listed here are leading our full feature release somewhere this first half of the year. We went more ambitious (we planned for end of 2019) so code has been moving along and we feel we are quite there! Checked tasks are ready, unchecked are in progress or planned. Priority is not given by this order.

Please feel free to comment, request more info or ask for clarification. Feature requests are also highly appreciated and taken in account (always, please!).

Strawberryfield

• Field Property exposure to Drupal strategies
  • [x] JSON KEY Provider (flattener)
  • [x] JSON Flatten Keys
  • [x] JSONPATH/JMESPATH
  • [ ] Entity Reference Casting Provider (Using UUID loading and configurable entity type)
  • [x] JSON stored Service Endpoints with extended logic (e.g HOCR)
  • [ ] Multi Map/ join: many properties to single. e.g All keys - Authorities- referring to creators, contributors etc unified as Agents keys. This leads to Fractal Ontologies and our Buckets approach.

JSON representation and enrichment

• Better File management (Better than Drupal)
  • [x] File referencing via UUID instead of via Entity ID
  • [x] Handle temporary files when moving from TEMP storage to PERMANENT
  • [x] Increment file usage count on new versions
  • [ ] Decrement file usage count on version removal
  • [x] Change file usage on Delete, EDIT on existing active content and versions
  • [ ] Add Webform based UI managment (reorder, replace, delete) for files
  • [ ] File based Plugins callable by Webhooks
    • [ ] TECHMD
    • [ ] ZIP/UNZIP
    • [ ] Derivative for larger MEDIA (video and Sound)
    • [ ] Pronom Service/Preservation
  • [x] New JSON Service Architecture reference
  • [x] Deposit/save on Node save whole, selfs sustainable Strawberry JSON blob in S3/Minio/FileSystem
• [x] Keep track of Service and action on Ingest/edit using Activity Streams
• [ ] Add more agent information on our activity streams for provenance and tracking.

Webforms integration

• [x] Webform Driven UI Ingest with custom handler and widget
• [ ] Create a set of Demo Webforms that cover base of our GLAM source data needs
• [ ] Allow Webform Field Widget selection be driven by RDF type and permissions.
• [x] New Solr Aware Entity Select Views (with code code to handle Solr to Entity) which allows
  • [x] Complex autocomplete elements (like get me all Digital Objects of Type Book with a green Cover the user can see
  • [x] New Fine grained Entity (node to node) reference possible through this.
• [ ] Create new, better, LoD Webform elements
  • [x] WIKIDATA
  • [x] LoC (with support for any Suggest endpoint)
  • [x] WIKIDATA Agents with LD Roles
  • [ ] WIKIDATA using custom SPARQL
  • [ ] Viaf
  • [x] Getty
  • [x] Nominatim Geo reconciliation. Normal and Reverse.
  • [x] Panorama Tour Building App (like 1200 lines of code, gosh!)
  • [x] Image and EXIF extraction on upload for U/facing previews.
• [ ] Create Stub (temporary) WIKIDATA entities if query shows desired WIKIDATA entity does not exist upstream.
  • [ ] "publish" to wikibase functionality
  • [ ] Replace repo wide stub uri with official one once pushed.
  • [ ] Keep track on the stub who is referencing it is (bidirectional reference?)
• [x] Move Strawberryfield harvest Webform handler's logic to plugins
  • [x] Deal with as:images
  • [x] Deal with as:documents, as:video, as:sound, as:dataset elements
  • [x] Deal with as:models
• [ ] Allow anonymous submits to be converted into proper Nodes by Admin (Self deposit, crowd sourced metadata)
• [x] Make Webform API Interaction work with States(JS) by removing one From wrapper.
• [ ] Make Webform API Interaction more versatile for our use. Use as schema validator.
• [x] Add JS to avoid main node CRUD to submit/validate embedded Webform as widget

Media Displays

• [x] Add expected mime/type output to Media displays. Allows to tag media displays as JSON, XML or HTML only.
  • [x] React to mime type to allow JSON or XML output to be downloaded too.
  • [x] Native/self rendering and Content-Type tagging.
  • [x] Automatic extraction From template of required/used variables (context). Not front facin yet but for sure useful for building a Pick-and-chose (or Data color picker) to aid in Twig Template building
• [ ] Add new Data Views Plugin integration to allow Media Displays to preprocess values on views exposed as API endpoints
• [ ] Version Media Displays
• [X] Provide example Twig templates for
  • [x] MODS
  • [ ] DC and
  • [ ] JSON-LD
  • [x] IIIF Manifest 2.1
  • [x] IIIF Manifest 3.0 (DRAFT)

Field Formatters

• [x] Static IIIF Images
• [x] Open Seadragon IIIF Images
  • [x] Add thumbnail navigation
• [x] IABookreader IIIF Images
  • [ ] Integrate Flavor based Solr Search
• [x] Panorama via IIIF
• [x] Panorama Tours via other Panorama Objects and IIIF, including Hotspots of many types
• [x] Metadata up-casters
• [x] Metadata up-casters with download endpoint
• [x] Video (HTML5)
• [x] Audio (HTML5)
• [x] PDF (custom, derived from the base PDF.js library. Not fancy. But Mozilla asks people to NOT use their fancy one directly and we agreed.
• [x] Web annotations (IIIF)
• [ ] Complex nested structures (Whole graphs)
• [x] 3D! (Three + JSM)
• [x] Minio 3.0 (With Resource comparison and multi sourced IIIF manifests)
• [x] Expose View Mode to JSON Type value mapping that triggers automatic View Mode Selection
• [ ] Webrecorder.io native player (WARC replay)

API Ingest, Migration and backup

• [x] Strawberryfield Normalizer: expands JSON string as a JSON when exporting
• [x] Strawberryfield denormalizer: string-ify JSON when importing
• [ ] Wrap JSONAPI on a set of Drush script to (Strawberry Seeds)
  • [ ] Allow Single command line invoke files and node ingest
  • [ ] Create virtual field Entity "buckets" to allow Media to be ingested into those as links and routed to internal Strawberryfield elements (utility methods for ingest)
• [ ] AMI (Archipelago Multi Import) First iteration
  • [ ] API Source (Other repos, ContentDM, Solr)
  • [ ] Google Spreadsheets (same as IMI)
  • [ ] Complete Drush 9 integration
• [ ] Filesystem drop-and-forget ingest. You save a JSON file into S3, Archipelago creates entities and relationships.

Service Architecture (Strawberry Runners)

• [x] Develop webhook driven notification service for derivatives
• [x] Custom, user facing Plugins. Build your own derivative workflows (system calls, JSON processing, etc)
• [ ] Document/deploy webhook triggers for minio S3 per mimetype
• [ ] Document/deploy webhook triggers for AWS S3 (via lambda) per mimetype
• [ ] Develope Shell processing using Symfony callables and user configurable for each case (rule system)
• [ ] Generate JSON reference-able Services for complex non descriptive metadata and data
  • [x] HOCR
  • [ ] TECHMD
  • [ ] Web Annotations
  • [ ] Tabular datasets
  • [ ] Transcripts (similar to Web Annotations, mostly dependant)
  • [x] Build slim Content entity that can be used to index natively that content into Solr via search API
  • [ ] Allow Services to be self explaining of its capabilities. only GET will be allowed
  • [x] Allow Drupal to discover hierarchically via EDISMAX

SEO and API

• [ ] Allow Media displays output to be embeded in HTML head for SEO
• [ ] Test/Develop nested DATA VIEWS integration for OAI-ORE and OAI-PMH
• [x] Create (TWIG, metadata displays) and expose as endpoints full set of IIIF API JSON outputs.
• [x] Add helper methods and twig extensions to allow Metadata displays to access pre existing views (like object listings for a collection) to help build those lists.

ACL / Permissions

• [ ] Integrate custom ACL with JSON Paths into per NODE ACL. Allowing this way to apply permissions to individual metadata elements/paths.
• [ ] Same but needs better UI for referenced Services and Media
• [ ] Allow Metadata (rule) to trigger ACL permissions. e.g if embargo_date == bla bla = remove public access
• [ ] Allow for ACL inheritance (from parent, recursive) without hard copies.

Deployment and DevOPS

• [ ] Sync Configurations and remove non used ones for minio branch / periodic for each Drupal release
• [x] Site-build and remove orphan blocks
• [x] Add more utility views
• [x] Enable JSONAPI by default on minio branch
• [x] Create jsonapi user with jsonapi credentials for minio branch
• [x] Create basic scripts to automate Docker/Bash operations
• [x] Update AWS deployer to match minio including docs and Cloud Services integration
• [x] XDEBUG integration. 2 PHP 7.3 Containers, Cookie based, routed by NGINX
• [x] Update all Strawberryfield modules script.
• [x] Drupal 8.8.1 and bumps on every module

Batch Operations

• [ ] Bulk Batch Views JSONPATH plugin to
  • [ ] Replace existing JSON values
  • [ ] Add to existing Values
  • [ ] Respect data type casted values, (entities, file references)
• [ ] Bulk Batch Views MEDIA plugin to
  • [ ] Replace Media
  • [ ] Add Media
• [ ] Bulk Batch Views ACL plugin to
  • [ ] Replace ACL and inheritance
  • [ ] Replace ACL individual Control List Elements
  • [ ] Add ACL individual Control List Elements
• [ ] Integrate into Solr Results and Strawberryfield Taxonomy Term pages

Future roadmap

• [x] D9 readyness 😄 -- we need second pass
• [x ] Solr Cloud/ Consortial ensemble
• [ ] Native Wikibase/Wikidata publishing

Documentation:

• [ ] Devops and new repository deployers
• [ ] Migration to and from.
• [ ] Backup and restoring
• [ ] Permissions, access and ACLs.
• [ ] Metadata Professionals, JSON schema and schema-less. AS, DR and AP internal ontologies. UPDATED
• [ x] Metadata Professionals, Key concepts of Archipelago
• [ x] Metadata, Ingest and edit workflows.
• [x ] Displays, Formatters and Media Plugins (Twig)
• [ ] Views Integration (Solr and Blocks)
• [ ] Strawberry Field Exposed Keys and Plugins
  • [ ] Property Exposing strategies and configs
• [ ] Media Management
• [ ] Solr and Discovery
• [ ] Extending and Coding
• [ ] SEO

[marlo-longley]

@DiegoPino Thank you for this update. Your leadership and and super hard work is what brought the code to this point! We were "more ambitious" and adding extra features as you said in 2019. Many of these features were in response to user needs and reflection which feels really good.

Ready for what's next!

[mitchellkeaney]

hey @DiegoPino we had talked about this IRL (remember those days...?) a while ago: the order of "Next" and "Previous" for PDF Viewer is in a counterintuitive order. Will swapping the order of these buttons be included in the Field Formatter section of this roadmap? Happy to open a new issue otherwise.

[DiegoPino]

@mitchellkeaney, i normally don't keep track of single UI or smaller front facing improvements in the general roadmap. The way i use Roadmap is for larger features and new functionality.

So, yes, i do remember and plan to have it for beta3, but please open an issue in github format_strawberryfield to keep track of that particular UI improvement and to make sure i do not forget. Thanks