Feature:Integration with Fediseer.com

[db0]

What do you want to see? I have created https://fediseer.com which is a way to crowdsource the validity of servers and create a chain of trust.

It provides an open REST API and it would be great if this could be integrated into this explorer.

The way it could work is that one could see if an instance in this list has been guaranteed (and by whom) and how many endorsements they have from other instances

[db0]

This can also help with #77

[tgxn]

Perfect, I have to rewrite my scoring algo - I'll do something like a 12-hourly scrape of your API and factor those results into the algo... I might have some time this week sometime, but no promises :)

[db0]

No problem! If you need any specific feature/endpoint in the API, do let me know!

[tgxn]

Weird, but clicking the "Full API Documentation" link takes me to https://overctrl.dbzer0.com/api/

going to https://fediseer.com/api/ directly works, but not https://fediseer.com/api (without trailing slash) must be something up with your reverse proxy.

Annnyway....

After having a quick look, would publishing your database (like https://data.lemmyverse.net/) be possible? Ideally I'd like a way to do one (or a paged) call to get a list of all guarantees and endorsements for all instances? I could obviously do it domain by domain, but I'd rather not have to make ~4 requests for each instance. 🤷‍♂️

This service is meant to operate in much the way that PGP trust chains I assume, with instances vouching for each other?

Also, I think the list from /v1/instances only returns stuff that I already have:

• domain, local_posts, comment_counts, total_users, active_users_monthly, signup - crawled directly
• uptime_alltime - from fediverse explorer
• activity_suspicion & active_users_suspicion are calculated in the same way as you do 😎

[db0]

Oh yea that totally looks like a reverse proxy thing. I'll fix :D

After having a quick look, would publishing your database (like https://data.lemmyverse.net/) be possible? Ideally I'd like a way to do one (or a paged) call to get a list of all guarantees and endorsements for all instances?

the /whitelist endpoint should contain a full list of all instances and their guarantor so that should cover that area.

Getting the full list of endorsements for each instance is a bit trickier, as each instance can have multiple endorsements given and received. I could export a parquet files with the content of the endorsements table. Would a form of

endorsing domain | endorsed domain

work for you? You can just parse this parquet to gather the exact list.

This service is meant to operate in much the way that PGP trust chains I assume, with instances vouching for each other?

Yes indeed

Also, I think the list from /v1/instances only returns stuff that I already have:

Yes, this is not really useful to you. This is just compiling publicly available info and doing some calcs.

[tgxn]

the /whitelist endpoint should contain a full list of all instances and their guarantor so that should cover that area.

dang I didn't see this, that looks good, I'll see if I can hit it at the same time I hit fediexplorer (12 hours), could increase to 6 for yours.

Getting the full list of endorsements for each instance is a bit trickier, as each instance can have multiple endorsements given and received. I could export a parquet files with the content of the endorsements table. Would a form of

endorsing domain | endorsed domain

work for you? You can just parse this parquet to gather the exact list.

maybe, what other details would there be? is it just a binary endorse/not endorsed, or is there other metadata?

I'm struggling a bit here, I've read your devlog and had a look at the API responses, I'm not sure what the difference between these would be:

• approvals - displays "endorsements" given for a domain
• endorsements - displays "endorsements" received for a domain
• guarantees - displays all "guarantees" given by a specific domain
• whitelist - list of all instances with "endorsements" for all domains

What data should I use to drive my decisions on trust - Would I use endorsements from whitelist and rank instances based on that? Should I factor in "guarantees"?

I currently store every instances allow and block lists, which is how I sort the instance list (if you have a server that x other instances are de-federated with, you score lower, if you appear on x other servers allow list, then you score higher) so I need a way to "factor in" your data as well as other sources in order to make a single "score" in order to sort the instances/communities 🥴

---

I registered my instance, and got a PM, works great, but - if I were using this service to directly drive my instances whitelist, then wouldn't all instances currently in the trust network essentially be de-federated from me?

[tgxn]

looking at some of the data coming out of /whitelist

would it be possible to include "chain length"/ do you think that would be useful?

[db0]

OK so, a guarantee can only be received once per domain. Being guaranteed means that someone ensured you're not spam. If expanded enough, the guarantee whitelist can become the base allow list for any instance to avoid spam instances.

Ensorsements are subjective. Any instance can endorse any other instance for any reason. One can use the endorsements as a whitelist if they want to be very picky, or they could could merge the endorsements of two or more instances they trust to have a larger allow-list.

What data should I use to drive my decisions on trust - Would I use endorsements from whitelist and rank instances based on that? Should I factor in "guarantees"?

So about your trust, the most important thing is the guarantee. A non-guarantee instance means that they couldn't find anyone else to claim they're not spam, which is a very low barrier to pass.

Endorsements is a more fuzzy value. It just points to how many other instances "like" this instance. it doesn't necessarily mean its untrustworthy if it doesn't have any however. Likewise, a right-wing side wouldn't endorse a left-wing one and vice-versa. You can use endorsement to adjust your sorting, and let people make decisions based on it.

Approvals are not particularly useful for sortingh. It just shows how many endorsements an instance has given out. However it's useful if one wants to make an allow-list for their instance based on endorsements of instances they trust.

My suggestion is: use guarantees for the trust ranking. Show the endorsements as a number (which one can expand to see which ones),but don't use it for trust but allow it to be used for sorting.

Use approvals to allow people to see which instances any on specific instance or combination of instances "like", like a filter

[db0]

would it be possible to include "chain length"/ do you think that would be useful?

Adding a way to see an instances full chain of trust is something I want to add at some point, but I don't think seeing the chain length would be particularly useful, no

I registered my instance, and got a PM, works great, but - if I were using this service to directly drive my instances whitelist, then wouldn't all instances currently in the trust network essentially be de-federated from me?

If you used the guarantee=1&endorsements=0 value, then your whitelist would consist of all instances which have been guaranteed to not be spam by others. I wouldn't use this currently because we haven't onboarded enough instances. But once we do, it could be a useful way to avoid spam instances, and even if you don't want to use it permanently, could also serve as a "panic button" when the fediverse is under attack.

[tgxn]

Ok, thanks for helping me understand. I wrote it down like this for myself later :)

• guarantee - "single instance parent" - once per domain - "not spam" one to one mapping of an instance and a parent instance that has guaranteed it is not spam

• endorsements - instances can endorse many other instances "instance friends" - how many other friends this instance has

• approvals how many instances this one has endorsed

I can use the existence of a guarantor to exclude instances from being flagged as "sus", and then for each instance, I can use the endorsements value directly as an additional metric.

I currently plan on getting a full list (endorsements=0&guarantors=0) and then filtering it from there.

NPs on the chain of trust thing, it'd be cool to have an endpoint for example that showed the path this instance used to get guaranteed.

Can an instance revoke a guarantee? What happens if an instance higher up in the chain is "unguaranteed"?

[db0]

Yes you can revoke a guarantee. If the instance which guaranteed you is "orphaned" then your instance remains guaranteed for 24 hours, giving you or your guarantor a chance to find another guarantor. If your guarantor receives a guarantee from someone else, all instances below them will be fine. This "grace period" is to prevent your instance being defederated by others using the guarantee list as allow-list and refreshing it during that period.

When a chain of trust is broken, all orphaned instances receive a PM as well so that they are aware. When it's repaired, they also receive one. If your guarantor cannot find another one to guarantee for them, you can remove your own guarantor and look for a different one on your own.

Without a guarantee, you cannot guarantee or endorse others, but existing endorsements will be considered valid for 7 days

one to one mapping of an instance and a parent instance that has guaranteed it is not spam

Correction. It is one to many. A single instance can guarantee up to 20 others but each instance can be guaranteed only by one instance

I currently plan on getting a full list (endorsements=0&guarantors=0) and then filtering it from there.

Note that using guarantors=0 instead of guarantors=1 will give you a lot of instances without much useful info. The only info you can use from a non-guarantee'd instance if whether it's been claimed by its admins or not.

[tgxn]

I'm now using the data from fediseer to:

• affect overall score calculation for smart sort
• remove instances from sus if they have a guarantee

it's live now, so we'll see how it goes, and I'll work on the frontend stuff when i get some more time (Y)

[db0]

Noice! Would it also possible to see who guaranteed an instance, in an instance's info?

[tgxn]

[db0]

Cheers, your number seems to be wrong though. Where are you getting the endorsements amount?

[tgxn]

You're not wrong they are waay out of date, heck. I did some fixes, and now I'm doign pagination properly and using the url as follows: /whitelist?endorsements=1&guarantors=1&page=${page}&limit=${perPage} gives me about ~255 results, with tags. I'll look at using the tags data soon...

[tgxn]

Looks right now 👍