Gather usage information

[acka47]

Questions we need to answer for lobid-gnd and preferably the other two lobid services:

• Who uses lobid?
• For that is lobid used? Which features are used?
  • Web UI
  • API
  • Bulk downloads
  • OpenRefine Reconciliation Endpoint

We should first take a deeper lool at the logs, I will provide the indicators for API vs. bulk vs. openrefine later. Furthermore, we should consider setting up a very short poll.

[acka47]

@dr0i, as I am not able to get this information from Matomo, can you please extract it from the logs directly? For all stats, please use the six months from 2018-08-01 to 2019-01-31.

lobid-resources

• 23.067: RSS: GET requests containing format=rss"or"Accept: application/rss+xml"`
• 5.304: Bulk downloads: GET requests containing format=jsonl, format=bulk, "Accept: application/x-jsonlines", or "Accept-Encoding: gzip"
• 479: other rdf formats: GET requests containing .rdf, .ttl, .nt
• 46: auto suggest: GET requests containing format=json: (The colon is important!)

lobid-gnd

• 178: Bulk downloads: GET requests containing format=jsonl, "Accept: application/x-jsonlines", or "Accept-Encoding: gzip"
• 61.896: auto suggest: GET requests containing format=json:
• 1.059:other rdf formats: GET requests containing .rdf, .ttl, .nt
• 126.611: OpenRefine reconciliation: POST with path /gnd/reconcile (EDIT dr0i: POST aren't logged. The path is just grepped (without word boundaries or so), so it may stand lonely, or completed with /properties.* (or anything else)).

lobid-organisations

• 3.107: csv Downloads: GET mit Header "Accept: text/csv", format=csv
• 1: Bulk downloads: GET requests containing format=bulk, "Accept: application/x-jsonlines", or "Accept-Encoding: gzip"
• 24: auto suggest: GET requests containing format=json:
• 330: other rdf formats: GET requests containing .rdf, .ttl, .nt
• 167: OpenRefine reconciliation: POST with path /organisations/reconcile

[acka47]

@fsteeg, please take a quick look whether i have forgotten anything.

[fsteeg]

Looks good, just two points:

other rdf formats: GET requests containing .rdf, .ttl, .nt

This works for all 3 services.

OpenRefine reconciliation: GET requests with path /gnd/reconcile

OpenRefine does indeed send a GET to that route to get the service metadata. However I think this is cached until the service in removed in OpenRefine. So this would indicate something like the unique users (without the non-OpenRefine users like coli-conc), not the number or frequency of usage. The actual reconciliation requests are POSTed to that route. This should include usage like in coli-conc.

[acka47]

THanks, @fsteeg. I updated https://github.com/hbz/lobid/issues/394#issuecomment-460613271 accordingly. Assigning @dr0i .

[dr0i]

Note that we don't log header in general. Et al no Accept-Header.

[dr0i]

@acka47 I added the numbers into your comment. See a note from me: the patterns you gave are just grepped by (so they may be complemented by whatever), and mind that we also don't log POSTs.

[acka47]

To get a feeling for the geographic distribution, here are the cities in the date range from 2018-08-01 to 2019-01-31 we had at least 100 visits from:

Frankfurt/Main Berlin Wien Kiel Leipzig Dortmund Hannover Hamburg Tübingen Braunschweig Graz Basel Marburg Dresden Stuttgart Mainz Starnberg Ashburn Winterthur Pinneberg

[acka47]

According to Matomo, nearly 200 .preview URLs were requested in /gnd from 2018-08-01 to 2019-01-31.

[acka47]

Closing this one. However, we should think about also logging POSTs to the /reconcile endpoint.