Dashboard comparison of activities between endpoints

[matmaxgeds]

Rationale

It would be useful to easily see that the registry has X activities, DSv2 has y activities (no v1 or validator fails, no duplicates - just first instance, can be a short processing delay), datadump has Z (same as registry) activities, DSClassic has A activities (missing duplicate IDs etc), D-portal has C (no secondary publishers)....which ones have cached/historial data........this way users can judge which source best meets their needs - maybe as part of the guide to using IATI data - but ideally via an auto updated thing as it will go out of date fast.

Proposal

• Errrr........start with the basics e.g. total count of activities.
• Then can work out if there are any v1 activities - and report so (maybe with a link to the number of v1 registry activities not included)
• See if all include the same publishers.....report any that are in the registry but not in the tool
• Then see if they include a sample activity to test each file on the registry
• Finally/probably too intensive..........get a list of activities in each and cross reference

[notshi]

Here are some links for d-portal that will get you some of the data you need. These are generated directly from the database so they are always up to date.

1. Total unique activity count json
2. Total activity count json - includes duplicate ids
3. Total publisher count json
4. Total v1 count json - slow search as it's quite intensive ~18 seconds
5. List of publishers json
6. List of unique identifiers in csv
7. List of identifiers in csv - includes duplicate ids

Getting the list of identifiers in json is intensive and will need paging so have opted for csv which is relatively quick.

[matmaxgeds]

Thanks @notshi, also linking up to @andylolz's gist which has many of the others: https://gist.github.com/andylolz/18649e999f2bfca17174ae3e90bb37a0

[notshi]

Woah, that's great! Really interesting to see these numbers on the same page. Hopefully we can get a clearer picture of the IATI ecosystem.

[siemvaessen]

I like this idea. As part of our efforts for AIDA we are thinking about setting up a /stats page with all the endpoints etc crunching some basic stats. Could we create a useful doc of stats needed/handy?