Display counts on list pages for "Unblocks Requested", and "Unblocked"

[alexhaydock]

On list pages on the main Blocked site, it would be useful if totals were displayed next to the xx entries header for:

• Number of list entries for which an unblock has been requested.
• Number of list entries which have been unblocked.

Ideally this would be per-site figures and not per-request figures.

[dantheta]

Deployed.

[edjw]

@alexhaydock – Does this let you do what you wanted to? The stats are on this page (https://www.blocked.org.uk/lists) but not the individual list pages.

[JimKillock]

There's a slight comparability problem: the number of unblocked URLs needs to be compared with the total number of blocks, as well as the number of domains affected.

[JimKillock]

Reopening this: the stat columns need to show:

• Entry count
• Number of entries reported
• Number of ISP blocks found
• Number of ISP blocks lifted

Otherwise we can't compare like with like.

@edjw we think this is needed to help us finish the stats

[edjw]

@dantheta – Can you take a look at this?

[dantheta]

I've got a version of this ready to test, but it's awfully slow because we've added in the status from the probe result tests (previously it was only using the ISP reports). I can add caching, but it means the results won't be up to the second.

I'm assuming that the number of ISP blocks lifted should be driven only by the ISP reports verdict, and not the detected probe status as well. I can change this pretty quickly if needed.

[dantheta]

New version is deployed.

[JimKillock]

Thanks Dan, some delay or caching is fine: we're after a best effort comparability here, as the data is going into a static report.

[alexhaydock]

I think there might be some doubling of the statistics here. Looks like the lists seem to be detecting twice as many blocks per-list as there are sites in some of the lists.

"Weddings" has roughly 4500 entries but apparently 8600 blocked sites in it. Are the figures getting multiplied by 2 somewhere?

[JimKillock]

@alexhaydock The 8600 is a figure for all blocks over all ISPs as I understand it. That is why it is higher.

However there still seem to be some bugs:

https://www.blocked.org.uk/lists shows that cement has just two reports made and one unblock;

https://www.blocked.org.uk/list/Cement shows several reports and unblocks on the first page.

Also: I made an error in the columns needed, as we also need a figure for "Entry count still blocked".

[dantheta]

The reported badge on the list page is currently shown for items where an ISP report exists in any state, but the reports total on the lists summary page is for ISP reports that were actually sent.

All of the ISP reports in the cement category apart from two are in the cancelled state.

For the entry count still blocked column, should that be based on ISP reports submitted (but not unblocked) or the detection system's view of which sites are still blocked?

[JimKillock]

OK, in that case we should make the flags consistent with the summary.

I think it makes more sense to show reports that are sent, rather than ones that were raised but not sent. That said I am surprised it is only two, as I have sent at least a dozen relating to concrete, though perhaps that was not from this list.

For "entry count still blocked" this should be the total of sites actually blocked according to the detection system.

(The aim of these stats is to illustrate the scale of potential errors yet to be reported, beyond what has been reported to ISPs and analysed by us.)

[dantheta]

OK - I'll need to do some testing with that one. That flag is also used to decide whether to make the "report site" button available, and we don't currently support reporting a site which has already been reported but not user-verified. I don't think the list presentation has any impact on the report stats, so is it OK to write it up as an issue for the backlog?

Working on the entries still blocked now.

[JimKillock]

Yes, thanks Dan, we can resolve the flag issue later, it doesn't need fixing today. Let me know if we can help work out the issue with counts - I could for instance look through the reports we've classified as "Building and construction" and compare them with the concrete list.

[dantheta]

OK, I'll get that switched over.

[JimKillock]

Actually, could you replace it with (or add) a column for "Current ISP blocks"? This would be directly comparable with the total "ISP blocks found" which presumably is the total of blocks originally present.

[dantheta]

They're both done. I added the extra column, so the totals can be viewed for list items and network blocks.

[JimKillock]

That’s great Dan, I think it’s a lot clearer now.

[JimKillock]

Further thought: would it be possible to add a total row at the bottom?

[JimKillock]

I'd still like to total row, especially if it were dynamic, eg something we could combine with the ISP filtering in #241

[dantheta]

Totals row added to the lists view.