Directory Prototype V2

[jpcamara]

Based on a design meeting w/ @kayaelle

Using the badgelist structure referenced here: https://github.com/mozilla/openbadges-discussion/issues/1#issuecomment-27394992

Storing registered endpoints in a MySQL database. The thought is that an issuer that wanted badges in the directory would register with it (hit an api endpoint, submit through a form, or some such). They would register an endpoint that they'd expose their badge class lists from. We may want to offer both this functionality, and the ability to request that a badge be indexed by hitting a directory endpoint directly. This was proposed by both @kayaelle and @stenington. @kayaelle and I discussed it and thought that a badge listing on an issuers site would probably be the easier lift for people, but maybe allowing for both is a good compromise for sites that want to push individual badges directly.

Storing indexing information in ElasticSearch. Experimenting with checking elastic search for previously indexed badges so we don't re-index, although to start it will probably just index multiple times (I think it's smart enough to know there's no difference).

There will be two parts to the directory: the indexer and the api. For now the indexer will probably just be one of the cluster workers and the api will use the rest, since heroku would require an additional paid dyno to have a background worker.

[threeqube]

cc/ @cmcavoy

[jpcamara]

I emailed this info, but it should really be in here as well.

Directory Demo

Here is a working example of searching the directory. It’s using the actual working directory process, with discovery badges being the issuer. http://test-openbadges-directory.herokuapp.com/examples/browser/#/recent

More Details about Directory design

The directory is currently indexing on a setInterval schedule (so fancy) and using a combination of mysql and elastic search. Elastic search is being used for all searching and indexing, Mysql is being used to store issuer information (and technically api keys, though they’re dormant at the moment).

It checks the issuer table, sees if any issuers have not been indexed (based on an indexed_at timestamp and a configurable interval), then goes after the endpoint if they need some refreshing/indexing. It hits the endpoint, which exposes a listing of badges. Then it goes after each of the listed badges. As it’s indexing, it timestamps the badge and also validates it (using the openbadges-validator module). If the badge is invalid it still indexes it, but it marks it with some metadata (valid: false) in elastic search. Nothing that is valid: false can be returned from the search (but it gives the directory insight into valid and invalid badges being consumed).

The way this can work right now (considering no one is exposing the listing yet) is that the Directory service itself is exposing a listing right now. There’s an additional table in mysql called “listings”, which I am populating by running a job that pulls from the google spreadsheet and puts the json blobs in mysql. Then I expose the contents of that database. http://test-openbadges-directory.herokuapp.com/temp/discovery/listing

In the “issuers” table I have an entry for an issuer called “Discovery”, and the “endpoint” field is pointing to that url. Making directory like a snake that is devouring itself. Meta. Recursive. Scary!

cc/ @stenington @chloeatplay @kayaelle

[jpcamara]

I just realized that I think I put in some ORs on the tags, instead of ANDs. I'll create an issue. #21

[threeqube]

http://test-openbadges-directory.herokuapp.com/temp/discovery/listing

This generates the following error message for me: {"message":"Cannot enqueue Query after fatal error."}

[jpcamara]

Yea there's a connection leak in there - i'll trim that up. It's back again (for a little while). Won't impact the searching though!

[threeqube]

Thanks am I supposed to just see a long list of stuff without a UI component?

[jpcamara]

Yea - the link you posted is for the json badgelist that the directory indexer consumes.

It hits the endpoint, which exposes a listing of badges... The way this can work right now (considering no one is exposing the listing yet) is that the Directory service itself is exposing a listing right now... http://test-openbadges-directory.herokuapp.com/temp/discovery/listing

The actual UI is in the section "Directory Demo".

Here is a working example of searching the directory. It’s using the actual working directory process, with discovery badges being the issuer. http://test-openbadges-directory.herokuapp.com/examples/browser/#/recent

[threeqube]

Ah gotcha. Thanks!