IIIF Viewers with IPFS

[daviddias]

IIIF (International Image Interoperability Framework) is hosting a conference at the Vatican on June 5-9. We (@diasdavid and @flyingzumwalt) will be present at that conference to talk about the benefits of using decentralized Web technologies that institutions like IIIF can benefit from.

We will be collaborating with @edsilv and @aeschylus to prepare a demo to showcase at the conference, of both Mirador and UniversalViewer using IPFS to exchange and update annotations made on the images. This issue will be used to coordinate this effort as a main central hub.

Mirador and UniversalViewer are both viewers of IIIF data that follow IIIF defined data model, they are created by independent researchers, but there is a lot of collaboration and idea sharing.

From the design thinking we've done, we know that we need:

• A way to propagate changes on the Annotation List
• A way to synchronize the state of the Annotation List for each document.

This converts into roughly the following milestones:

1. Mechanism to propagate changes on the annotation list (this can be using libp2p pubsub directly)
2. Synchronize the Annotation list through IPFS directly (using orbit-db or another CRDT library that is mounted on top of IPFS
3. Load the entire collection of images into the viewer through IPFS itself
4. Load and run the entire viewer using only js-ipfs, requiring no plugins installed.

This suggested that we just need to create primitives for get and put with the ability to follow changes or updated in a document, once that part is done, it can be fully integrated.

Other important pieces of context:

• Annotation list is a simple JSON document
• Web Annotation Data Model - https://www.w3.org/TR/annotation-model/
• @dignifiedquire's quick experiment done for a previous conference (might not be working at the moment) - https://github.com/dignifiedquire/anno
• Viewdir - a collection of modules shared between both viewers https://github.com/viewdir
• UV uses https://svelte.technology/
• More context on the origins of the proposed demo https://github.com/orbitdb/orbit-db/issues/213
• CRDT Research repo https://github.com/ipfs/research-CRDT

[daviddias]

alright, just had a good chat with @pgte on this endeavor, he is in!

To move forward:

• @edsilv and @aeschylus, could you provide current Annotation List examples, changes, updates and so on.
• There is a lot of room to explore here, but I feel that the milestones described above are the best to start with, let me know if you feel something should be added from the start.
• We should set a weekly standup to make sure this demo is progressing and needs are met. @edsilv, @aeschylus, @flyingzumwalt would you be available to do this? I can set up a Doodle.

Let's try to all meet this week :)

[edsilv]

Great! Here's an example of an AnnotationList:

https://wellcomelibrary.org/annoservices/search/b18035723?q=gene

(http://iiif.io/model/shared-canvas/1.0/#AnnotationList)

This is the format Mirador uses to store annotations and the UV uses to display search results.

This will eventually be replaced with the Web Annotation Data Model as part of IIIF Presentation API 3.0, but this is still some way off:

https://github.com/IIIF/iiif.io/issues/496

(https://www.w3.org/TR/annotation-model/#annotation-collection)

Milestone 3 states that the images should be loaded over IPFS. Currently we only have this very simple fully-IPFS-enabled IIIF manifest: http://universalviewer.io/examples/?manifest=https%3A%2F%2Fipfs.io%2Fipfs%2FQmYomyCpT1vKNovYvTjAStYq31JoaH8xdivjmWcDj5mpM1

Milestone 4 would require some work from @aeschylus to port Mirador to be able to be loaded over IPFS. Currently the UV can be loaded using the ipfs.io gateway. Are you suggesting that we bypass the gateway and use js-ipfs to load all of the viewer assets?

Perhaps in order to keep things manageable initially we could focus on enabling serverless communication between the viewers over js-ipfs, with the viewers themselves and IIIF manifests being loaded over HTTP. We could then move on to fully IPFS-ifying the viewers and IIIF manifests + assets?

I need to complete the work that will allow me to update the UV search results programmatically as opposed to on user interaction. This will be necessary to be able to respond to the IPFS annotation endpoint. I'm working on giving the UV an external API and expect this to be ready in the next few weeks.

We will also need to be able to specify the annotation endpoint to subscribe to somehow when starting the viewers. Perhaps a text field that we paste the name of the endpoint into?

A weekly standup sounds good!

[edsilv]

At the bottom of https://wellcomelibrary.org/annoservices/search/b18035723?q=gene you'll see a hits array. This is extra data to provide context to search results. We don't need to worry about this as the UV doesn't require it.

[daviddias]

Let's find the time for that weekly standup

@edsilv @pgte @flyingzumwalt @aeschylus please fill: https://doodle.com/poll/bqpzs9xxzfde25pc

Everyone else is welcome to add their availability if they are interested in actively participating in this endeavor, otherwise, we will have to favor the schedule of the people contributing. Thank you :)

[daviddias]

\o/ woot, we have a schedule! :)

Every friday at:

Zoom URL is: https://zoom.us/j/131133322

[aeschylus]

Great!

I'll look forward to speaking to everyone on the 28th. For reference, here is the link to a fork of orbitdb we've made where the main example has been tossed up to work as a textarea.

https://github.com/viewdir/orbit-db/tree/initial-hacking

More context is in https://github.com/orbitdb/orbit-db/issues/213, noted above.

[flyingzumwalt]

Standup Friday April 21

Agenda

• Welcome
• Round of Intros
• Define target for next week

Misc:

Recent Article about using pubsub on IPFS

Notes

• Participants: @edsilv, @diasdavid, @flyingzumwalt, @pgte, @aeschylus

There are 2 main vierwers in the IIIF space

• Mirador (Stanford, Mellon-funded), which is more of a scholarly workbench and a tool for authoring annotations.
• Universal Viewer, aka "UV"(Wellcome Trust, British Library) - more of a generic viewer for many data formats (Image, PDF, Video, 3D objects...)

Main IIIF APIs:

• Images api - way to display images from an image service
• Presentation api - way of listing canvases (ie. a series of pages)

Other Apis: search, annotation, etc.

• new annotations API will use the WebAnnotations Model (w3c spec)
  • lets you create AnnotationList objects

The Demo: Serverless IIIF Image Annotations

• mirador producing annotations, writing them ot IPFS & UV showing those annotations -- completely serverless collaborative editing of the annotations

Work Done so Far

Drew and Ed got a basic version of UV running on orbit-db. they got it working but ran into troubles with orbit and confusion about what they should do

Questions:

• Will the IPFS daemon run in the browser? Will people have to run a go-ipfs daemon in order to use IPFS pubsub?
  • 2 attempts: one in november 2016 with @dignifiedquire. Then again in March 2017 using the latest orbit-db (hoping that they would be able to to pure js implementation)

Q: can we use js-ipfs? A: you can use either. go-ipfs has some features that js-ipfs doesn't have

Current Work

NEED for demo: Easy way for everyone in the room to get the demo code, including a working IPFS node

• needs ipfs-download.js, or something similar

Theatrics of the demo:

• everyone downloading ipfs binaries over HTTP will be slow
• everyone downloading IIIF tiles over HTTP will be slow
• everyone downloading IIIF tiles over IPFS ifs they have ipfs running will (should?) be much faster

Where the work will happen:

• @pgte will create a repo for this annotations-pubsub work in github.com/pgte/ ... we might create an "ipfs-labs" github org for this kind of work to live
• the first-pass version of this demo only relies on ipfs, which includes ipfs-pubsub. it doesn't need CRDTs, so doesn't need orbit-db
• if we have time before the demo, we will add CRDT support.

Action Items:

• [ ] figure out what the annotations pubsub API should look like
• [ ] get basic annotations on pubsub working in both viewers
  • updates are the whole blob of annotations (refresh the whole list on every change)
• [ ] prepare the theatrics of the demo
• [ ] Add CRDTs (deal with chronological order of annotations with many editors)

What Ed & Drew need:

• (pubsub consumer) a way for mirador & UV to listen for an event telling you that something has changed
• (pubsub producer) a way to send updates when you have changed something
• a way to declare a pubsub topic/channel and subscribe to it

[pgte]

Created repo, can move it later to org: https://github.com/pgte/ipfs-iiif-annotations

[edsilv]

Here's the event on sched: https://2017iiifconferencethevatican.sched.com/event/AChW/presentation-interoperable-peer-to-peer-research-with-iiif-and-ipfs-room-5

[daviddias]

Standup Friday April 28

Agenda

• Standup

Notes

• Participants: @edsilv, @diasdavid, @flyingzumwalt @Brendam O'Brian
  • Pedro won't be able to make it today
• iiifDB https://github.com/pgte/ipfs-iiif-db
• Testing
  • [ ] Write down test cases for iiif-db with potential scenarios of how it is going to be used by uv and mirador (ed)
  • [ ] Use JEST (https://facebook.github.io/jest/) to test UV
• Explore how we can achieve pagination of annotations using IPLD
• Food for thought - namespace of the publisher instead of just contentId
• [ ] Ship an example of iiif-db using browserify (@pgte)
• Goal: Have a thing that @edsilv can use by next Tuesday //cc @pgte

Video: it is being rendered and uploaded

[edsilv]

Some more notes from the 28/04/17 standup:

We are currently using the Open Annotation spec which is what the UV and Mirador both support and is perfect for an MVP. However, IIIF will use the Web Annotation Data Model in future. This has support for AnnotationPages which would allow us to break up the data into more manageable chunks instead of a single large json blob.

It also supports agents which would allow us to identify and filter by user.

The model uses IRIs for ids, which means fs:/ipfs/<hash> is a valid identifier if we want to permit content addressing.

Also worth noting that AnnotationPages are based upon Activity Streams Collections. Activity Streams is a more generic specification and can be applied to many other types of actions.

Create a front end test harness? Use Jest: https://facebook.github.io/jest/

[daviddias]

@pgte has finished the first milestone, you can see it here: https://github.com/pgte/ipfs-iiif-db/issues/1#issuecomment-299268499

@edsilv @aeschylus can I get your ack? The example includes a demo video

[edsilv]

Hey,

Looks great! I'll have a play with it before the standup :-)

On 4 May 2017 8:23 pm, "David Dias" notifications@github.com wrote:

@pgte https://github.com/pgte has finished the first milestone, you can see it here: pgte/ipfs-iiif-db#1 (comment) https://github.com/pgte/ipfs-iiif-db/issues/1#issuecomment-299268499

@edsilv https://github.com/edsilv @aeschylus https://github.com/aeschylus can I get your ack? The example includes a demo video

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/240#issuecomment-299284129, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRceUwzHYN_O0C4eH6MkBWiJkysRO1aks5r2iWegaJpZM4M-UZr .

[aeschylus]

Having a look now and getting our example put up.

[edsilv]

@flyingzumwalt @diasdavid Here's the designer/illustrator I mentioned: http://whitevinyldesign.com I worked with Luke on https://blokdust.com

[edsilv]

Also, here's the go-iiif lib: https://github.com/thisisaaronland/go-iiif#iiif-tile-seed

[daviddias]

Standup Friday May 5

Agenda

• standup: updates from everyone
• first milestone finished - get+put primitives are done, propagates data
• validate the api - can we build on the interface
• set up next milestone
• generating tiles

Notes

• Participants: @edsilv, @diasdavid, @flyingzumwalt, @aeschylus

First milestone finished - get + put

• doesn't use CRDTs but supports basic pubsub for propagation of changes

drew:

• created IPFS experiments repo in IIIF github
• how will we serve the code? npm link from github? publish on npm?
  • npm link locally and then browserify
• seems to have everything we need
• future feature: would be nice to be able to look back in history
  • A: there will be a proper CRDT structure to show log pedro:
• done with 1st iteration of the library. nothing IIIF specific, just a JSON blob going back & forth
• next: CRDTs and make the libary better at handling concurrency ed:
• working on UV for stanford
  • plays into what we need for the demo, smoothing off rough edges around supporting IIIF v3

Validating the API

• looks right. next validation: actually build the demo with it
• how does it find peers?
  • one way nodes find peers: auto-discover using WebRTC signaling
  • another way: websockets, using a preconfigured bootstrap list
    • we will probably use this for the demo so the nodes connect with peers that have the IIIF data loaded

What about preservation? Can we set up a listener node that's archiving the annotations?

• yes. we can create a listener node that listens to the topic an pins all of the annotations on that topic

every time someone makes an annotation, the annotation goes on IPFS and the node publishes a link to that annotation on the pubsub topic. There is not a global log for all annoations

Next Milestone:

Drew: get mirador using pedro's new pubsub library, using browserify version. would like to also incorporate into electron version of mirador Ed: integrate into UV Pedro: support Ed + Drew, integrate proper CRDT-based log Matt: think about structure of demo, theatrics and visuals/illustrations

Next Next Milestone:

Add in CRDTs: Swap out swarmjs or yjs in place of current simple pubsub library

[daviddias]

I've created a roadmap to help us have a notion of time we have left and the sync the things we have planned for each week. I'll keep updating this roadmap as things progress and we set new goals. Right now it clearly lists the goals for this week and how much time we have till the conf.

ROADMAP

Here is a generated ROADMAP with what we have planned so far to help us be on the same page.

Week May 8

• publish iiif-db to npm
• Integrate Mirador with iiif-db
• Integrate Universal Viewer with iiif-db
• sync annotations with both viewers
• upgrade iiif-db to support custom IPFS configs
• upgrade iiif-db to support switching between go-ipfs and js-ipfs nodes (see how it is done in js-ipfs tests)
• create two branchs of iiif-db, one using swarmjs, other using yjs offering the same API
• create an outline of the demo + presentation
• add namespace support (extra channel) to support pinning.

Week May 15

• (@aeschylus) - Implementing endpoint for Mirador
• (@flyingzumwalt) - Design the final demo and write an outline for it
• (@pgte + @edsilv ) Figure out how to shard the annotations list
• (@pgte) pick one of the CRDT libraries and invest the time of changing its internals to use js-ipfs directly

Week May 22

Week May 29

Week May 5

• The conference

[pgte]

Published on NPM: https://www.npmjs.com/package/ipfs-iiif-db

[pgte]

• [x] Support custom IPFS configs
• [x] Released ipfs-iiif-db v0.1.0.

[daviddias]

How is the integration going @aeschylus @edsilv ? Need any help?

[edsilv]

Hi David,

Here's the repo:

https://github.com/viewdir/ipfs-experiments

I've set up the UV page, but it isn't hooked up to ipfs-iiif-db yet.

UV, Mirador, and the contents of ipfs-iiif-db/examples/simple-producer-consumer/public/js are being copied out of node_modules on npm start.

I've included app.js in uv.html, but am getting Bignumber is not a constructor. Not sure why as it seems to be the same setup as in the example.

app.js contains some UI-specific code ($startButton, $stopButton, etc). Presumably we'll need to include the same UI elements in mirador.html and uv.html?

I'm going to be at this for the rest of the week: https://www.rijksmuseum.nl/en/2and3dphotography

I might be able to get away on Friday though to look more at this and join the standup.

On 9 May 2017 10:57 pm, "David Dias" notifications@github.com wrote:

How is the integration going @aeschylus https://github.com/aeschylus @edsilv https://github.com/edsilv ? Need any help?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/240#issuecomment-300299164, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRceWQstYGmhSdxInllwKl6fbJH_Hmoks5r4NNSgaJpZM4M-UZr .

[pgte]

Hi Ed,

The bundle node_modules/ipfs-iiif-annotations/examples/simple-producer-consumer/public/js/app.js is specific to that example. Instead, I guess you could try to somehow use the library from within the app:

1. If you're going to use a bundler like browserify or webpack, you could use it like a node.js or an ES6 module from inside your app.
2. Alternatively, the ipfs-iiif-db library could provide a bundle that you could include on a script tag.

Which pattern(s) suits your needs?

[edsilv]

I think #2 (the script tag) would be great in terms of reducing the amount of overhead required to getting something up and running.

I was thinking that perhaps we could create a http://svelte.technology component at some point to wrap up any reusable code and UI.This could be used as a custom element in your page and provide a nice interface with inline attributes and events you could listen for. You could imagine having a topic attribute and onchanged event. Also, optional start/stop IPFS buttons.

It could be a drop-in way to 'ipfs-ify' your application. I have some experience with svelte and could help with such an effort.

On 10 May 2017 10:22 am, "Pedro Teixeira" notifications@github.com wrote:

Hi Ed,

The bundle node_modules/ipfs-iiif-annotations/examples/simple- producer-consumer/public/js/app.js is specific to that example. Instead, I guess you could try to somehow use the library from within the app:

1. If you're going to use a bundler like browserify or webpack, you could use it like a node.js or an ES6 module from inside your app.
2. Alternatively, the ipfs-iiif/db library could provide a bundle that you could include on a script tag.

Which pattern(s) suits your needs?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/240#issuecomment-300411062, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRceRpAi2tGCn54hUmTJFLoaGU9zeVbks5r4XPYgaJpZM4M-UZr .

[pgte]

@edsilv : Just published version 0.2.0, which now includes a packaged dist.

Here are the instructions for dropping in a script tag: https://github.com/pgte/ipfs-iiif-db/tree/master#using-a-script-tag-in-a-browser

[edsilv]

@pgte I've updated https://github.com/viewdir/ipfs-experiments to use the unpkg.com script instead. mirador.html has a button that submits the contents of a textfield and this is console.logged in uv.html. So the plumbing works :-)

[edsilv]

Hey, preservation didn't get mentioned on the standup. We should aim to make it trivially easy to have your ipfs node "watch" a topic and pin new annotations. Perhaps ipfs-iiif-db could have a cli like ipfs-iiif-db watch mytopic ?

[pgte]

@edsilv I think that's a great idea. @diasdavid?

[edsilv]

Also, just a suggestion - but what about calling it ipfs-annotation-db ? Bit more generic?

[flyingzumwalt]

a cli that follows the channel and outputs to stdout is a great idea for showing that a machine could follow the channel and archive/preserve it. On Sat, May 13, 2017 at 1:19 PM Edward Silverton notifications@github.com wrote:

Also, just a suggestion - but what about calling it ipfs-annotation-db ? Bit more generic?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ipfs/notes/issues/240#issuecomment-301261840, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIesoHP68NKAKhaoV2G5DpSUV38JGaDks5r5eYxgaJpZM4M-UZr .

[daviddias]

Sounds great to me ! :)

[pgte]

@edsilv @aeschylus Here is a proposal for the new API that supports CRDTs: https://github.com/pgte/ipfs-iiif-db/tree/yjs#instantiate

It's much more specific than the previous one, since I had to define a specific data structure, supported by shared CRDTs.

It should now be easier to integrate with a UI without the need to refresh the entire annotation list.

Here is an example app: https://github.com/pgte/ipfs-iiif-db/blob/yjs/examples/simple-producer-consumer/src/app.js

Here is a demo of it working: https://youtu.be/8xE1MEyZsbo

@edsilv @aeschylus Do you see any problem with this API, or some use case that you require that it doesn't support?

[pgte]

Now has own issue: https://github.com/pgte/ipfs-iiif-db/issues/4

[edsilv]

@pgte Looks good - essentially CRUD on top of a CRDT? I'll look at updating the demo app tomorrow to fit this new model. I'm sure this will uncover any potential problems.

[edsilv]

https://jsonfeed.org/version/1

[pgte]

@edsilv great! I haven't released a distribution yet, will do it first thing in the morning.

[BigBlueHat]

@edsilv curious about jsonfeed.org. The Web Annotation Protocol has a "feed" mechanism built in: https://www.w3.org/TR/annotation-protocol/#representations-with-annotation-descriptions

It's based on Activity Stream 2.0 collections.

If I've misunderstood the plans for this, let me know. 😃 Happy to help where I can!

[edsilv]

@BigBlueHat Hey! I just posted jsonfeed.org out of interest really. If we're making something here that's essentially a way to easily do CRUDable json arrays on top of a CRDT there are all kinds of possibilities. Activity Streams are super interesting. It would be great if you wanted to join our new regular IPFS for GLAM call. Perhaps you could give us the lowdown on Activity Streams?

https://github.com/ipfs-labs/ipfs-glam-community

[pgte]

@edsilv @aeschylus version 2.0.0 of ifs-iiif-db released.

Distribution URL to update the script tag: http://unpkg.com/ipfs-iiif-db@2.0.0/dist/index.min.js

(The previous version is still available through here: https://unpkg.com/ipfs-iiif-db@0.2.0/dist/index.min.js )

[pgte]

@edsilv @aeschylus beware of reconnection difficulties (almost fixed) reported here: https://github.com/pgte/ipfs-iiif-db/issues/6

[edsilv]

@pgte I've updated the example. Here's a video of it working: https://youtu.be/SMOiTZYeoG0

Right now I'm passing in the original param inline on both clients like this:

https://github.com/viewdir/ipfs-experiments/blob/master/viewer-annotations/mirador.html#L27

Perhaps we generate the @id by hashing the combined manifest URI and topic?

e.g.

"@id": "fs:/ipfs/<hash>",

[pgte]

@edsilv we can generate any kind of id if it's not defined, and we could create a random one. The first argument of the db.annotationList() is supposed to be an id, a unique identifier for that annotationList. Could be a URI, any string that has some identification meaning to you.

We can create an alternative API where the ID can (or not) be defined inside the originalValue, and if it's not defined, we create a random one. Something like:

db.annotationList({
    "@context": "http://iiif.io/api/search/0/context.json",
    "@id": "https://wellcomelibrary.org/annoservices/search/b18035723?q=gene",
    "@type": "sc:AnnotationList",
    "within": {
        "@type": "sc:Layer"
    },
    "startIndex": 0,
    "resources": []
})

(in this case we wouldn't create an id since it already has one).

Another option is to enforce an id.

Which API suites you best?

[edsilv]

@pgte Hmm, not sure. I think this is something we can discuss on the standup later.

[flyingzumwalt]

Here's my outline for the demo. We can discuss + refine on today's call. It's in this hackpad

Same info in slide deck form (slightly outdated): Serverless IIIF with IPFS.pdf ... and here's the Keynote file

[pgte]

New 2.1.0 version of the ipfs-iiif-db package just released. Supports new semantics for the @id attribute of an annotationList and passing in only the id or the initial annotationList. // @edsilv @aeschylus

[pgte]

@edsilv @aeschylus just released version 2.2.0. Relevant changes:

• code: If you're using the Githubissues.
• Githubissues is a development platform for aggregating issues.