Stopped halfway due missing authorizations

[oculos]

Hi,

I have given my token the scopes as per instructions. Nevertheless, it breaks down after processing some domains:

Actually, it stopped now with an error after a dozen domains processed:

2023-01-13 11:39:26,260 ERROR Cannot fetch follow information for 10minutepleroma.com from mastodon.babb.be: b'{"error":"This action is outside the authorized scopes"}'
Traceback (most recent call last):

[jpwarren]

Ah, I think this might indicate that another scope is needed to fetch the follower statistics for a domain.

Can you please post your (tokens redacted) config for both the blocklist_instance_sources and blocklist_instance_destinations settings?

[jpwarren]

Ah, I see in the Mastodon code that the scope for /api/v1/admin/measures access is admin:read. It's in https://github.com/mastodon/mastodon/blob/v4.0.2/app/controllers/api/v1/admin/measures_controller.rb

Sorry, you'll need to give the app/token in the instance you push to admin:read access to it can check if your instance has people following accounts at the domain about to be blocked. I'll update the documentation.

Perhaps we could consider turning off the instance followers check if people want to keep the scopes locked down tighter?

Ideally the Mastodon scopes should be finer-grained so you can provide access to instance statistics without also enabling full admin:read access.

[oculos]

Hi! I tried with admin:read. I got the same error. Do you still wanna see the config?

[jpwarren]

Yes please! I see GitHub autoclosed the issue because I tagged it as 'Fixes:' in the merge comment.

[oculos]

Here is the full error:

(...)
2023-01-13 22:24:39,577 INFO Change detected. Need to update ['reject_media', 'reject_reports', 'obfuscate'] for domain block for social.quodverum.com
2023-01-13 22:24:39,577 INFO Old block definition: <DomainBlock {'domain': 'social.quodverum.com', 'severity': 'silence', 'public_comment': '', 'private_comment': '', 'reject_media': True, 'reject_reports': True, 'obfuscate': True, 'id': '24'}>
2023-01-13 22:24:39,578 INFO Pushing new block definition: <DomainBlock {'domain': 'social.quodverum.com', 'severity': 'silence', 'public_comment': '', 'private_comment': '', 'reject_media': False, 'reject_reports': False, 'obfuscate': False}>
2023-01-13 22:24:40,704 INFO Change detected. Need to update ['reject_media', 'reject_reports', 'obfuscate'] for domain block for toot.love
2023-01-13 22:24:40,704 INFO Old block definition: <DomainBlock {'domain': 'toot.love', 'severity': 'silence', 'public_comment': '', 'private_comment': '', 'reject_media': True, 'reject_reports': True, 'obfuscate': True, 'id': '25'}>
2023-01-13 22:24:40,705 INFO Pushing new block definition: <DomainBlock {'domain': 'toot.love', 'severity': 'silence', 'public_comment': '', 'private_comment': '', 'reject_media': False, 'reject_reports': False, 'obfuscate': False}>
2023-01-13 22:24:41,836 INFO Adding new block: <DomainBlock {'domain': '10minutepleroma.com', 'severity': 'suspend', 'public_comment': '', 'private_comment': '', 'reject_media': True, 'reject_reports': True, 'obfuscate': False}>...
2023-01-13 22:24:41,955 ERROR Cannot fetch follow information for 10minutepleroma.com from mastodon.babb.be: b'{"error":"This action is outside the authorized scopes"}'
Traceback (most recent call last):
  File "/home/mastodon/.local/bin/fediblock-sync", line 8, in <module>
    sys.exit(main())
  File "/home/mastodon/.local/lib/python3.10/site-packages/fediblockhole/__init__.py", line 661, in main
    sync_blocklists(args)
  File "/home/mastodon/.local/lib/python3.10/site-packages/fediblockhole/__init__.py", line 83, in sync_blocklists
    push_blocklist(token, domain, merged.values(), conf.dryrun, import_fields, max_followed_severity)
  File "/home/mastodon/.local/lib/python3.10/site-packages/fediblockhole/__init__.py", line 535, in push_blocklist
    newblock.severity = check_followed_severity(host, token, newblock.domain, newblock.severity, max_followed_severity)
  File "/home/mastodon/.local/lib/python3.10/site-packages/fediblockhole/__init__.py", line 407, in check_followed_severity
    follows = fetch_instance_follows(token, host, domain)
  File "/home/mastodon/.local/lib/python3.10/site-packages/fediblockhole/__init__.py", line 388, in fetch_instance_follows
    raise ValueError(f"Something went wrong: {response.status_code}: {response.content}")
ValueError: Something went wrong: 403: b'{"error":"This action is outside the authorized scopes"}'

(this was an attempt to sync from mastodon.social

And here is my config:

# List of instances to read blocklists from.
# If the instance makes its blocklist public, no authorization token is needed.
#   Otherwise, `token` is a Bearer token authorised to read domain_blocks.
# If `admin` = True, use the more detailed admin API, which requires a token with a
#   higher level of authorization.
# If `import_fields` are provided, only import these fields from the instance.
#   Overrides the global `import_fields` setting.
blocklist_instance_sources = [
   { domain = 'mastodon.social'}, # an instance with a public list of domain_blocks
  # { domain = 'jorts.horse', token = '<a_different_token>' }, # user accessible block list
  # { domain = 'eigenmagic.net', token = '<a_token_with_read_auth>', admin = true }, # admin access required
]

# List of URLs to read csv blocklists from
# Format tells the parser which format to use when parsing the blocklist
# max_severity tells the parser to override any severities that are higher than this value
# import_fields tells the parser to only import that set of fields from a specific source
blocklist_url_sources = [
  # { url = 'file:///home/daedalus/src/fediblockhole/samples/demo-blocklist-01.csv', format = 'csv' },
#  { url = 'https://raw.githubusercontent.com/eigenmagic/fediblockhole/main/samples/demo-blocklist-01.csv', format = 'csv' },

]

# List of instances to write blocklist to
blocklist_instance_destinations = [
   { domain = 'mastodon.babb.be', token = 'mytoken'},
]

## Store a local copy of the remote blocklists after we fetch them
#save_intermediate = true

## Directory to store the local blocklist copies
# savedir = '/tmp'

## File to save the fully merged blocklist into
# blocklist_savefile = '/tmp/merged_blocklist.csv'

## Don't push blocklist to instances, even if they're defined above
# no_push_instance = false

## Don't fetch blocklists from URLs, even if they're defined above
# no_fetch_url = false

## Don't fetch blocklists from instances, even if they're defined above
# no_fetch_instance = false

## Set the mergeplan to use when dealing with overlaps between blocklists
# The default 'max' mergeplan will use the harshest severity block found for a domain.
# The 'min' mergeplan will use the lightest severity block found for a domain.
# mergeplan = 'max'

## Set which fields we import
## 'domain' and 'severity' are always imported, these are additional
##
import_fields = ['public_comment', 'reject_media', 'reject_reports', 'obfuscate', 'private_comment']

## Set which fields we export
## 'domain' and 'severity' are always exported, these are additional
##
export_fields = ['public_comment']

[jpwarren]

Curious. Do you have access to the instance database? Can you run this SQL code so we can see what the scopes are?

SELECT token, scopes FROM oauth_access_tokens
WHERE token = '<mytoken>';

Again, redact the token before sharing the output.

[oculos]

Here it goes:

mastodon_production=> SELECT token, scopes FROM oauth_access_tokens
WHERE token = 'my token';
                    token                    |                       scopes
---------------------------------------------+----------------------------------------------------
mytoken | admin:read:domain_blocks admin:write:domain_blocks

It looks like the admin:read isn't being applied, no matter if it is selected on the gui. I then applied a " admin:write:domain_blocks admin:read:domain_blocks admin:read admin:write" via the UPDATE command, and it worked.

A bug with Mastodon?

[lnlyssg]

I then applied a " admin:write:domain_blocks admin:read:domain_blocks admin:read admin:write" via the UPDATE command, and it worked.

I had the exact same issue and can confirm that this has got it working for me too.

[jpwarren]

Ooh interesting!

I'd expect any changes you make in the GUI to overwrite anything done directly in the database, but it doesn't seem to do that. But the GUI says it has!

Yeah that smells like a bug.

[jpwarren]

Did some experiments, and the SQL hack is probably not the right way to do things.

Changing the app scopes updates the oauth_applications table, but doesn't change what's been authorised for the existing token, which is the right behaviour. The SQL hack I've suggested updates the token scopes directly, but not the app scopes. That's why the access doesn't change if you make changes in the GUI.

The tool should really only change the app scopes and you then need to regenerate the token in the GUI. I'll update the doco to say that.

The admin scopes have been added into Mastodon main, so this whole hack thing should go away in the next release.

[jpwarren]

Okay, use this SQL instead:

UPDATE oauth_applications as app
  SET scopes = '<scopes>'
  FROM oauth_access_tokens as tok
  WHERE app.id = tok.application_id
  AND app.name = '<the_app_name>'
;

and then this if you don't want to generate a new token:

UPDATE oauth_access_tokens as tok
  SET scopes = '<scopes>'
  FROM oauth_applications as app
  WHERE app.id = tok.application_id
  AND app.name = '<the_app_name>'
;

[jpwarren]

Fix is merged, so I'll close this issue now.