distributed-text-services / specifications

Specifications for the DTS API
https://w3id.org/dts
28 stars 9 forks source link

ideas for an "indexes" endpoint #167

Open PietroLiuzzo opened 4 years ago

PietroLiuzzo commented 4 years ago

Dear all, I have been thinking about a possible additional API, which I will temptatively call indexes Endpoint which I am quite sure has been planned many times and perhaps discarded or deferred to the future. I am not sure and I thought it harmless in any case to copy here my thoughts.

I was thinking of a syntax like the following, to have an API which is index-like and allows a navigation of the text which is not only the structure of it, just like in most cases we would anyway have, and be generic enough not to restrict it self to any particular type of index.

/dts/api/indexes?{&id}{&name}{&page}

{
  "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
  },
  "@type": "IriTemplate",
  "template": "/dts/api/collections/?id={collection_id}&name={index_name}&page={index_page}",
  "variableRepresentation": "BasicRepresentation",
  "mapping": [
    {
      "@type": "IriTemplateMapping",
      "variable": "collection_id",
      "required": false
    },
    {
      "@type": "IriTemplateMapping",
      "variable": "index_name",
      "required": false
    },
    {
      "@type": "IriTemplateMapping",
      "variable": "index_page",
      "required": false
    }
  ]
}

/dts/api/indexes?{&id}

would lists the available supported indexes with their names, depending on what one has marked for such indexes, for example terms, persons, dates, whatever.

Here is a made-up example for a non-readable collection

/api/dts/indexes?id=urn:dts:betmas

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id":"urn:dts:betmas",
    "member": [
      {"name": "persons"},
      {"name": "places"}
    ],
"dts:collection": "/api/dts/collections?id=urn:dts:betmas"

}

and one for a readable collection, where additionally I would get direct access to other endpoints, and may find more indexes specific to that only.

/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id":"urn:dts:betmas:LIT3122Galaw",
    "member": [
      {"name": "persons"},
      {"name": "places"},
      {"name": "terms"}
    ],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT3122Galaw",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT3122Galaw",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT3122Galaw"

}

Once one has picked the wanted index, this would have the structure of a normal book index with a key and the passages it is found at, optionally also the actual string annotated, if relevant (it would not mean anything for an index of topics for example)

/dts/api/indexes?{&id}{&name}

would return the desired index. for example, something like

/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&name=persons

would have pagination has the collection endpoint and then a number of "attestations" one for each indexed thing. Each of these attestations would come with a series of members, like in the navigation endpoint, for each occurrence in the collection.

{
  "@context" : {
    "dts" : "https://w3id.org/dts/api#",
    "@vocab" : "https://www.w3.org/ns/hydra/core#",
    },
"view" : {
    "first" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=1",
    "next" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=2",
    "last" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=10",
    "previous" : null,
    "@type" : "PartialIndexView",
    "@id" : "/api/dts/indexes?id=urn:dts:betmas:LIT3122Galaw&page=1",
    "totalItems": 200,
    "title" : "index of persons",
 "name" : "persons"
    "description": "index of persons attested in collection",

  },
"dts:attestation" :[
{"@id" : "https://betamasaheft.eu/PRS2377Bakaffa",
"@value" : "Bakaffa"
 "member": [
      {"ref": "1.1", 
       "@value" : "መሢሕ፡ ሰገድ፡"},
      {"ref": "1.2", "@value" : "በካፋ፡"},
      {"start": "1.1.1", "end": "1.1.2"}
    ]
},
{"@id" : "https://betamasaheft.eu/PRS3827Ephrem",
 "member": [
      {"ref": "1.3"},
      {"ref": "1.4"},
      {"start": "1.1.1", "end": "1.1.2"}
    ]
}
],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT3122Galaw",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT3122Galaw",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT3122Galaw"
}

I am not sure if it would be useful to include also, &level=2 and &groupSize=2 to determin t the format of ref and return them as "dts:citeDepth" and "dts:level": 3.

In the special case of places, I think it would be also useful to link to pelagios annotations, with a property for that like in the following

/api/dts/indexes?id=urn:dts:betmas: LIT4633Chronicle&name=places

{
  "@context" : {
    "dts" : "https://w3id.org/dts/api#",
    "@vocab" : "https://www.w3.org/ns/hydra/core#",
    },
"view" : {
    "first" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=1",
    "next" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=2",
    "last" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=10",
    "previous" : null,
    "@type" : "PartialIndexView",
    "@id" : "/api/dts/indexes?id=urn:dts:betmas:LIT4633Chronicle&page=1",
    "totalItems": 200,
    "title" : "index of places",
 "name" : "places"
    "description": "index of persons attested in collection",

  },
"dts:pelagios" : "https://betamasaheft.eu/api/placeNames/works/LIT4633Chronicle",
"dts:attestation" :[
{"@id" : "https://betamasaheft.eu/LOC6157WahniA",
 "member": [
      {"ref": "2", 
       "@value" : "ደብረ፡ ወህኒ፡"}
    ]
},
{"@id" : "https://betamasaheft.eu/LOC6376Yebab",
 "member": [
      {"ref": "2", 
        "@value": "ይባባ፡"}
    ]
}
],
"dts:passage": "/api/dts/documents?id=urn:dts:betmas:LIT4633Chronicle",
"dts:references": "/api/dts/navigation?id=urn:dts:betmas:LIT4633Chronicle",
"dts:collection": "/api/dts/collections?id=urn:dts:betmas:LIT4633Chronicle"
}  

I think here @value could also be used also if the document api supports CTS like syntax to get passages like urn:dts:betmas:LIT4633Chronicle:2@ደብረ፡ ወህኒ፡[1], or eventually if one, starting from this wants to get directly to the forms attested. I have not added totals for the number of attestations, although it would be useful perhaps.

I think this type of API would allow a very flexible access to the text in one of the most common ways to access texts normally in books or web resources.

mromanello commented 4 years ago

Hi @PietroLiuzzo, many thanks for this proposal. As someone who has recently developed a completely custom JSON API for the index locorum (i.e. index of cited passages from primary sources) of a scholarly publication, I'm very interested in defining ways to standardize how indexes can be exposed and queried.

Here some initial thoughts/questions triggered while reading your ideas above:

PietroLiuzzo commented 4 years ago

Ideally I think indexes could be at any collection or passage level: index for the collection, for the entire document, for a section of if. However, if an index for a resource does need only dts:ref, what would then such a reference look like at collection level?

{
    "dts:attestations": [
        {
            "member": {
                "@lang": "gez",
                "@value": "ደብረ፡ ዳሞ፡",
                "dts:ref": "152"
            },
            "@id": "https://betamasaheft.eu/INS0105DD"
        },
        {
            "member": [
                {
                    "@lang": "gez",
                    "@value": "ደብረ፡ ሊባኖስ፡",
                    "dts:ref": "109"
                },
                {
                    "@lang": "gez",
                    "@value": "ደብረ፡ ሊባኖስ፡",
                    "dts:ref": "231"
                }
            ],
            "@id": "https://betamasaheft.eu/INS0341DL"
        }
    ]
}

in the above example I have in dts:ref the same I would have in the navigation endpoint for the deepest possible level of reference for the attestation. But if the request is for a collection, and I want to group by attestation, the dts:ref would need to be a full URI or I should we group by source within the attestation and specify a "@source" kind of thing?

{
    "dts:attestations": [
        {
            "@id": "https://betamasaheft.eu/INS0341DL",
            "source": [
                {
                    "@id": "https://betamasaheft.eu/LIT1234",
                    "member": [
                        {
                            "@lang": "gez",
                            "@value": "ደብረ፡ ሊባኖስ፡",
                            "dts:ref": "109"
                        },
                        {
                            "@lang": "gez",
                            "@value": "ደብረ፡ ሊባኖስ፡",
                            "dts:ref": "231"
                        }
                    ]
                },
                {
                    "@id": "https://betamasaheft.eu/LIT5678",
                    "member": [
                        {"dts:ref": "1.3.4"},
                        {"dts:ref": "2.3.1"}
                    ]
                }
            ]
        }
    ]
}

on your second point I fully agree, again I am not sure where that optional information may be better placed. skos:*Match ? or simply making an array of the value of @id

{
    "@id": "https://betamasaheft.eu/INS0341DL",
    "skos:exactMatch": "https://www.wikidata.org/entity/Q12345678",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}

on the index types defined in dts, I am not actually sure, but I think it can do no harm, especially if it does no mean that these are the only ones one can make.

I also agree on the last point, and I think I would opt for the same structure of the navigation api with citeType. if I understand well here we want to state the structure of the citation. let's say that, in an ideal word, the value of my index locorum is also a DTS uri, e.g. in my resource at 1.a and 2.c.i there is a citation of Gen. 1.1., I could have something like this

{
    "@id": "https://betamasaheft.eu/LIT1546Genesi&ref=1.1&level=2",   
    "label": "Gen. 1.1",
    "type": "index locorum",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}

but I would need to lookup in the collection API for dts:citeStructure, perhaps this could be served directly?

{
    "@id": "https://betamasaheft.eu/LIT1546Genesi",
    "dts:ref" : "1.1",
    "dts:level" : 2 ,
    "dts:citeDepth": 2,
    "dts:citeStructure": [
        {
            "dts:citeType": "chapter",
            "dts:citeStructure": [
                {"dts:citeType": "verse"}
            ]
        }
    ],
    "label": "Gen. 1.1",
    "type": "index locorum",
    "member": [
        {"dts:ref": "1.a"},
        {"dts:ref": "2.c.i"}
    ]
}
mromanello commented 4 years ago

Some additional thoughts from today's chat:

  1. need to specify the exact location of the index text within its context

Examples:

To complicate things, the same indexed entity (i.e. Homer) may be mentioned several times within the same context. The index will need to contain each individual attestation, and provide a way to unambiguously find this attestation in its context (use case: client application displays the highlighted indexed term in its context).

The solution to this depends on the type of content:

(@PietroLiuzzo can you mock up an example for the xpath solution based on what you already have?)

  1. behaviour of index endpoint at collection vs document level

2a) One option that would work well at different levels (i.e. keeping same structure of the response) is to move @id inside members like in the following

{
    "dts:attestations": [
        {
            "member": {
                "@lang": "gez",
                "@value": "ደብረ፡ ዳሞ፡",
                "dts:ref": "152",
                "@id": "https://betamasaheft.eu/INS0105DD"
            }
        }
       ... here the list continues ...
    ]
}

2b. The other option, sketched by @PietroLiuzzo above, is to use @source inside attestations to group indexed elements by containing document. The "problem" with this, is that the response structure changes depending on the index level and the client will have to guess which one it received (a document index or a collection index?).

This way, the response structure does not change when the index level changes (collection vs document).

  1. embed dts:citeStructure information in the index endpoint response

From an implementation point of view this seems desirable, as to limit the number of calls to diff. endpoints that the client will have to make. However, this is in conflict with one of the two options for point 2 above. It can work with 2b (@source used to group attestations at collection level) but it will become super redundant with 2a (@id gets moved inside members)

PonteIneptique commented 4 years ago

While I am not against it in general, I would heavily recommend that we do not reinvent the wheel: I really think we should, if we do this, build onto the WebAnnotation framework : http://webannotation.org/

mromanello commented 4 years ago

@PonteIneptique yes, Web Annotation was exactly what we were thinking of concerning point #1 above. Or is it a more general point (along the lines of "an index is just a set of annotations with the purpose of indexing")?

PonteIneptique commented 4 years ago

It's kinda both, I have to admit. I really think we should try to frame this functionality in the context of both Hydra and WebAnnotation, except if there is a spec for a webannotation API.

But it looks to me like Index is a collection of annotations, so you could have, in terms of APIs:

AnnotationCollection -(n)> WebAnnotation where things are derived from both. The nice thing about it, is that it would go beyond indexes...

PonteIneptique commented 4 years ago

And I'd probably allow @source in target of WebAnnotation to be a Navigation object, ie {ref, start, end, documentid, level?, citationtype?}

PietroLiuzzo commented 4 years ago

Sorry @PonteIneptique could I please ask you to unpack some of this with an example? thanks!

PonteIneptique commented 4 years ago

@PietroLiuzzo I'll try to do that in the following days, please remind me if I don't L(

balmas commented 4 years ago

I'm not sure but it could also be useful to pay attention to the work of the LD4LT group here, it seems they are also thinking about issues of interoperability of linguistic annotations https://github.com/ld4lt/linguistic-annotation

PonteIneptique commented 4 years ago

Example of navigation:

foobar.com/dts/annotations

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id": "//foobar.com/foo/annotations",
    "@type": "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "title": "Annotations Root Collection",
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
             "@id" : "//foobar.com/foo/annotations/indexes",
             "title" : "Index collection",
             "description": "Collection of Index",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        },
        {
             "@id" : "//foobar.com/foo/annotations/linguistic_annotation",
             "title" : "Linguistic Annotations Collection",
             "description": "Collection of Linguistic Annotations",
             "@type" : "Collection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        }
    ]
}

foobar.com/dts/annotations?id=//foobar.com/foo/annotations/indexes

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id" : "//foobar.com/foo/annotations/indexes",
    "title" : "Index collection",
    "description": "Collection of Index",
    "@type" : "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
             "@id" : "//foobar.com/foo/annotations/indexes/texts",
             "title" : "Texts Indexes",
             "description": "Collection of indexes for each book",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        },
        {
             "@id" : "//foobar.com/foo/annotations/indexes/general",
             "title" : "General Index",
             "description": "General Index",
             "@type" : "AnnotationCollection",
             "totalItems" : 2,
             "dts:totalParents": 1,
             "dts:totalChildren": 2
        }
    ]
}

foobar.com/dts/annotations?id=//foobar.com/foo/annotations/indexes

{
    "@context": {
        "@vocab": "https://www.w3.org/ns/hydra/core#",
        "dc": "http://purl.org/dc/terms/",
        "dts": "https://w3id.org/dts/api#"
    },
    "@id" : "//foobar.com/foo/annotations/indexes/general/general", 
    "title" : "General Index",
    "description": "General Index",
    "@type" : "AnnotationCollection",
    "totalItems": 2,
    "dts:totalParents": 0,
    "dts:totalChildren": 2,
    "dts:dublincore": {
        "dc:publisher": ["École Nationale des Chartes"],
    },
    "member": [
        {
          "@context": "http://www.w3.org/ns/anno.jsonld",
          "id": "http://example.org/anno3",
          "type": "Annotation",
          "creator": {
            "id": "http://example.org/user1",
            "name": "Foo",
            "nick": "Bar"
          },
          "body": [
            {
              "role": "identifying",
              "text": "Place1",
              "id": "http://example.org/place1"
            }
          ],  
          "target": {
            "source": {
                // Optional level?, citationtype?
                "type": "Resource",
                "ref": "Passage1",
                "id": "foobar.com/Document2",
                "passage" : "foobar.com/dts/document?id={id}&ref={ref}"
            },
            "selector": {
              "type": "TextQuoteSelector", // Or any other selection
              //....
            }
          }
        }
        // Other Annotation
    ],
    "view": {
        // Pagination information
    }
}
PietroLiuzzo commented 4 years ago

thanks @PonteIneptique ! I think I understand a bit more. I will try it out and see how far I get in managing my indexes. basically what I had as named indexes will be annotation collections has you already said and the web annotation will be there to model each single member. grouping I guess would be for the client to do?

PonteIneptique commented 4 years ago

I think it can be up to the client, but the provider should probably, if he is doing indexes, have multiple target with the same body. Note that this is an idea, and in no way vouched by the Technical Committee.

Le ven. 19 juin 2020 à 17:22, Pietro Liuzzo notifications@github.com a écrit :

thanks @PonteIneptique https://github.com/PonteIneptique ! I think I understand a bit more. I will try it out and see how far I get in managing my indexes. basically what I had as named indexes will be annotation collections has you already said and the web annotation will be there to model each single member. grouping I guess would be for the client to do?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/distributed-text-services/specifications/issues/167#issuecomment-646695160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOXEZTES4A7ZQV4HHAZ2VDRXN7ETANCNFSM4JCQIONA .

PietroLiuzzo commented 4 years ago

webannotation Technical Committee or DTS?

PonteIneptique commented 4 years ago

DTS :)

Le ven. 19 juin 2020 à 17:27, Pietro Liuzzo notifications@github.com a écrit :

webannotation Technical Committee or DTS?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/distributed-text-services/specifications/issues/167#issuecomment-646697456, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOXEZRQ5KTRWSLF6QOHKI3RXN7WLANCNFSM4JCQIONA .

PietroLiuzzo commented 4 years ago

Soooo I have given it a go with my DTS allocated time, and I think it does work decently.

/api/dts/annotations

this fetches the top collection of annotations, which I limit to three which are related to collections of TEI files + one which has them all, i.e. all our annotations for any given indexed thing.

{
  "dts:totalChildren": 4,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations Root Collection",
  "member": [
    {
      "dts:totalChildren": 7,
      "title": "Annotations of works Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of mss Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/mss",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of narr Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/narr",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7,
      "title": "Annotations of all Root Collection",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/all",
      "dts:totalParents": 1,
      "totalItems": 7
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations",
  "dts:totalParents": 0,
  "totalItems": 4
}

/api/dts/annotations/works

this has a collection of annotation collections, one for each available index, which I also decide ahead of time. beside these another collection of annotation collection, /items/ is just organized differently, and has a collection of annotation collections one for each item. this way one can drill down to a more specific context.

{
  "dts:totalChildren": 6,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of works Root Collection",
  "member": [
    {
      "dts:totalChildren": 154,
      "title": "Index of persons for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/persons",
      "dts:totalParents": 1,
      "totalItems": 154
    },
    {
      "dts:totalChildren": 25,
      "title": "Index of places for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/places",
      "dts:totalParents": 1,
      "totalItems": 25
    },
    {
      "dts:totalChildren": 7249,
      "title": "Index of keywords for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/keywords",
      "dts:totalParents": 1,
      "totalItems": 7249
    },
    {
      "dts:totalChildren": 3,
      "title": "Index of loci for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/loci",
      "dts:totalParents": 1,
      "totalItems": 3
    },
    {
      "dts:totalChildren": 7,
      "title": "Index of works for works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/works",
      "dts:totalParents": 1,
      "totalItems": 7
    },
    {
      "dts:totalChildren": 7262,
      "title": "Annotations of each item in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items",
      "dts:totalParents": 1,
      "totalItems": 7262
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works",
  "dts:totalParents": 1,
  "totalItems": 6
}

/api/dts/annotations/works/items

While the previous ones do not really need to be paginated, there are always less than 8 annotation collections, there are a lot of items in a resource collection, thus, here they are paginated.

{
  "view": {
    "name": "items",
    "first": "/api/dts/annotations/works/items?page=1",
    "next": "/api/dts/annotations/works/items?page=2",
    "last": "/api/dts/annotations/works/items?page=727",
    "previous": null,
    "title": null,
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/items?page=1",
    "totalItems": 7262
  },
  "dts:totalChildren": 7262,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of each item in works",
  "member": [
    {
      "dts:totalChildren": 1,
      "title": "Annotations of فصل في أصول علم الرمل in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3551IHA",
      "dts:totalParents": 3,
      "totalItems": 1
    },
    {
      "dts:totalChildren": 1,
      "title": "Annotations of Collection of Praises of the Prophet in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT1527IHA",
      "dts:totalParents": 3,
      "totalItems": 1
    },
...
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items",
  "dts:totalParents": 3,
  "totalItems": 7262
}

/api/dts/annotations/works/places

will return the index of places for the resource collection works and will return one annotation collection for each referenced entity

{
  "view": {
    "name": "places",
    "first": "/api/dts/annotations/works/places?page=1",
    "next": "/api/dts/annotations/works/places?page=2",
    "last": "/api/dts/annotations/works/places?page=91",
    "previous": null,
    "title": "Index of places",
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/places?page=1",
    "totalItems": 902
  },
  "dts:totalChildren": 25,
  "@context": {...},
  "dts:dublincore": {...}
    ]
  },
  "title": "Index of places for works",
  "member": [
    {
      "dts:totalChildren": "127",
      "title": "Annotations of Goǧǧām in places index.",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/LOC3549Gojjam",
      "dts:totalParents": 3,
      "totalItems": "127"
    },
    {
      "dts:totalChildren": "59",
      "title": "Annotations of ʾAbbāy in places index.",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/LOC1022Abbay",
      "dts:totalParents": 3,
      "totalItems": "59"
    },
..... 
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/places",
  "dts:totalParents": 1,
  "totalItems": 25
}

/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam

the id here can be used as a parameter (so that it need not to be hierarchical) and finally we get some annotations!

{
  "view": {
    "name": "places",
    "first": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=1",
    "next": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=2",
    "last": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=13",
    "previous": null,
    "title": "Index of places",
    "@type": "PartialIndexView",
    "@id": "/api/dts/annotations/works/places?id=https://betamasaheft.eu/LOC3549Gojjam&page=1",
    "totalItems": 127
  },
  "dts:totalChildren": 127,
  "@context": {...},
  "dts:dublincore": {....},
  "title": "Annotations of Goǧǧām in places index.",
  "member": [
    {
      "body": [
        {
          "text": "ጐዣም፡",
          "@lang": "gez",
          "role": "placeName",
          "id": "https://betamasaheft.eu/LOC3549Gojjam"
        }
      ],
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "target": {
        "source": {
          "ref": null,
          "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11",
          "type": "Resource",
          "link": "https://betamasaheft.eu/LIT3951ChronSusenyos.11",
          "id": "https://betamasaheft.eu/LIT3951ChronSusenyos",
          "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11"
        },
        "selector": {
          "@value": "TEI/text/body/div[2]/div[11]/ab/placeName[1]",
          "type": "XPath"
        }
      },
      "type": "Annotation"
    },
    {
      "body": [
        {
          "text": "ጐዣም።",
          "@lang": "gez",
          "role": "placeName",
          "id": "https://betamasaheft.eu/LOC3549Gojjam"
        }
      ],
      "@context": "http://www.w3.org/ns/anno.jsonld",
      "target": {
        "source": {
          "ref": null,
          "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11",
          "type": "Resource",
          "link": "https://betamasaheft.eu/LIT3951ChronSusenyos.11",
          "id": "https://betamasaheft.eu/LIT3951ChronSusenyos",
          "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT3951ChronSusenyos&ref=11"
        },
        "selector": {
          "@value": "TEI/text/body/div[2]/div[11]/ab/placeName[3]",
          "type": "XPath"
        }
      },
      "type": "Annotation"
    },
    .... other annotations
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/LOC3549Gojjam",
  "dts:totalParents": 3,
  "totalItems": 127
}

/api/dts/annotations/works/items/LIT3122Galaw

if I go down the /items/ road, I can then have the list of indexes for one work (at this point I am not supporting editions for this, but it would just be yet another level of collections)

{
  "dts:totalChildren": 4,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Annotations of Chronicle of Galāwdewos in works",
  "member": [
    {
      "dts:totalChildren": 60,
      "title": "Index of persons for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/persons",
      "dts:totalParents": 1,
      "totalItems": 60
    },
    {
      "dts:totalChildren": 181,
      "title": "Index of places for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/places",
      "dts:totalParents": 1,
      "totalItems": 181
    },
    {
      "dts:totalChildren": 3,
      "title": "Index of keywords for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/keywords",
      "dts:totalParents": 1,
      "totalItems": 3
    },
    {
      "dts:totalChildren": 1,
      "title": "Index of works for Chronicle of Galāwdewos in works",
      "@type": "AnnotationCollection",
      "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw/works",
      "dts:totalParents": 1,
      "totalItems": 1
    }
  ],
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT3122Galaw",
  "dts:totalParents": 3,
  "totalItems": 4
}

/api/dts/annotations/works/items/LIT3122Galaw/persons

and

/api/dts/annotations/works/items/LIT3122Galaw/persons?id=https://betamasaheft.eu/PRS1522Ahmadb

would be then the same, but at a deeper level of context (the most obvious one, the one of the indexes for one edition).

the index of loci is a special beast I think here an example of what I have tried

{
  "dts:totalChildren": 2,
  "@context": {...},
  "dts:dublincore": {...},
  "title": "Index of loci for Zenā mǝssāleyāt za-ṭabib Fisǝʾalgos in works",
  "member": {
    "body": [
      {
        "text": "ተመሰልኩ፡ ጰልቃን፡ ጳልቃን፡ ዘገዳም።",
        "@lang": "gez",
        "role": "ref",
        "id": "betmas:LIT2000Mazmur.101.7"
      }
    ],
    "@context": "http://www.w3.org/ns/anno.jsonld",
    "target": {
      "source": {
        "dts:citeType": "sentence",
        "dts:level": "1",
        "dts:citeDepth": 2,
        "dts:passage": "/api/dts/document?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2",
        "type": "Resource",
        "link": "https://betamasaheft.eu/LIT4916PhysB.4.2",
        "id": "https://betamasaheft.eu/LIT4916PhysB",
        "dts:ref": "4.2",
        "dts:references": "/api/dts/navigation?id=https://betamasaheft.eu/LIT4916PhysB&ref=4.2"
      },
      "selector": {
        "@value": "TEI/text/body/div[2]/div/div[2]/ab/ref",
        "type": "XPath"
      }
    },
    "type": "Annotation"
  },
  "@type": "AnnotationCollection",
  "@id": "https://betamasaheft.eu/api/dts/annotations/works/items/LIT4916PhysB/loci",
  "dts:totalParents": 4,
  "totalItems": 2
}

in the navigation object I am not doing any start-end, because luckily enough I do not have any of those things to index, only things which would be within a referenceable unit. I also make up a link property in source and a type property in selector, and certainly some other stuff and I certainly have abused a couple of things here and there. I needed to try it out. At least for my needs this would serve well, and I think it does have what we discussed above, and improves on the fact that the annotations are not different from one level to the other, they will always and consistently be annotation collections at any level.

PonteIneptique commented 4 years ago

Regarding your last comment, I'd recommend to check the XPathSelection type : https://www.w3.org/TR/annotation-model/#xpath-selector

Note that I don't know if, for an index, I'd repeat the target multiple time or have separate Annotation (Ie one annotation targeting all different presence of specific places vs. multiple annotation with a single target vs. one annotation with multiple target in the same document).

What you experiment shows though is the important need of a filtering system (and believe me, I don't want to, but I feel we'll have to finally address this...)

balmas commented 4 years ago

There is already a Web Annotation API protocol: https://www.w3.org/TR/annotation-protocol/

I think we have to have a compelling reason to implement our own

PietroLiuzzo commented 4 years ago

My best reason I think is just: I like DTS better! My real reason would be that I want it tightly linked with the other DTS APIs, to the navigation and document especially, because after all, with an index what I want to do is simply go back to that fragment of text where the indexed term occurs and navigate from there. In my head, and I may be simply failing to abstract enough, an index, whatever its format is, offers a navigation and access feature as much as the structure of passages which I my have and offer in my navigation API, it actually uses that system of reference and has its scope only there. It does not tell me "this is an annotation with this target and this body", it tells me "this thing is present at this passage" or for loci "this passage of text x is cited at this passage of text y". I would love to also have, as there are dts:passage and dts:references a dts:annotations or dts:indexes that would allow me to jump to different set of annotations referring to an id or available at that level of context, i.e. what is indexed here and where is this passage indexed. It would greatly help with cross quotations of ancient texts. As I said, I see how this can be abstracted to fit the same annotation model, but I feel it is much closer and tightly connected to the DTS navigation API than to a tag or a post on a blog. If the same can be used with profit, I am ok with it, I do not have any problem, all on the contrary, however my feeling is that this is actually a different thing. This is probably not compelling, but it is part of my reasons for doing this. On the filtering I agree, and I could easily add a parameter for a search term which would allow also for facets values to drill further down on the results and narrow down collections this way to have an index of persons in passages in the collection matching the query string and with a specific value for a facet.

PonteIneptique commented 4 years ago

There is already a Web Annotation API protocol: https://www.w3.org/TR/annotation-protocol/ I think we have to have a compelling reason to implement our own

I agree with @balmas , I just could not find it again. Looking at it from afar, I think we can reuse the container system with an AnnotationCollection DTS system for discovery, but this would create an hybrid system, and I am not sure I like this idea (like member DTS property + annotation Protocol Items property ? No way it's gonna be easy to parse).

Maybe we should look again at https://iiif.io/api/presentation/3.0/#55-annotation-page : Annotations are members of the canvases and served through the Annotation Protocol with specific pointers.

@balmas I don't think I have seen any discoverability system in the WebAnnotation Protocol. Any pointers ? Because for indexes at least, that's what people are gonna want.

PonteIneptique commented 4 years ago

Given that I am developing my though as I write (which is probably a bad idea ?):

Maybe we should look again at https://iiif.io/api/presentation/3.0/#55-annotation-page : Annotations are members of the canvases and served through the Annotation Protocol with specific pointers.

  1. Definitely, I can imagine having a dts:annotations + dts:indexes (which would differentiate both kind of annotations ?) in Collection, Resource and Reference (#184) object.
  2. If there is no discoverability / curating system for annotations in the WebAnnotation protocol, we should either: propose one to them (Not sure I want to go there, but you know the XKCD comics about creatings standards...) or build onto what DTS does to provide one, which would remove the need for a specific item browser.
balmas commented 4 years ago

@balmas I don't think I have seen any discoverability system in the WebAnnotation Protocol. Any pointers ? Because for indexes at least, that's what people are gonna want.

There has been some discussion on the w3 annotation list about this recently but I don't think there is any specific progress on it. See the thread at https://lists.w3.org/Archives/Public/public-openannotation/2020May/0003.html

It might be worth getting in touch with the folks working on Pundit to see if a collaboration is possible?

balmas commented 4 years ago

If we don't go the Web Annotation protocol route, then following the IIIF model does make a lot of sense. I get @PietroLiuzzo 's point about keeping the two connected.