nanos / FediFetcher

FediFetcher is a tool for Mastodon that automatically fetches missing replies and posts from other fediverse instances, and adds them to your own Mastodon instance.
https://blog.thms.uk/fedifetcher?utm_source=github
MIT License
297 stars 215 forks source link

New version depends on endpoint "api/v1/admin/accounts" which is not available in Pleroma #50

Closed csolisr closed 1 year ago

csolisr commented 1 year ago

When attempting to run the latest version of FediFetcher (v4.3.0), using it on Pleroma/Akkoma fails as the new method depends on the endpoint "https://example.net/api/v1/admin" which gives the error

{"error":"Not implemented"}

Logs:

2023-06-13 10:44:37.970590 CST: Starting FediFetcher
2023-06-13 10:44:38.680607 CST: Job failed after 0:00:00.709996.
Traceback (most recent call last):
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):                                                                                                                                                                                 File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen                                                                                                              httplib_response = self._make_request(
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f12063f8d00>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/admin/.local/lib/python3.9/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/home/admin/.local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //social.example.net/api/v1/admin/accounts (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f12063f8d00>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/admin/FediFetcher/find_posts.py", line 860, in <module>
    reply_toots = get_all_reply_toots(
  File "/home/admin/FediFetcher/find_posts.py", line 310, in get_all_reply_toots
    reply_toots = list(
  File "/home/admin/FediFetcher/find_posts.py", line 311, in <genexpr>
    itertools.chain.from_iterable(
  File "/home/admin/FediFetcher/find_posts.py", line 278, in get_active_user_ids
    resp = get(url, headers={
  File "/home/admin/FediFetcher/find_posts.py", line 687, in get
    response = requests.get( url, headers= h, timeout=timeout)
  File "/home/admin/.local/lib/python3.9/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/home/admin/.local/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/admin/.local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/admin/.local/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/admin/.local/lib/python3.9/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //social.example.net/api/v1/admin/accounts (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f12063f8d00>: Failed to establish a new connection: [Errno 111] Connection refused'))
csolisr commented 1 year ago

I am yet to confirm whether this change was made upstream by Pleroma itself, as the documentation only shows information on "api/v1/admin/users" instead: https://docs-develop.pleroma.social/backend/development/API/admin_api/

nanos commented 1 year ago

hi @csolisr

thanks for your report.

In fact, the functionality is part of the very original functionality of FediFetcher.

However, only one small (in my opinion the least useful) part of FediFetcher relies on that API endpoint: If you set REPLY_INTERVAL_IN_HOURS / --reply-interval-in-hours to 0, FediFetcher won't use that API endpoint, and as far as I know every other part of its functionality is compatible with Pleroma. (Though I haven't personally tested it, so I'd love to know your feedback, please)

csolisr commented 1 year ago

Well now I'm getting a different error related to a different backend, this time api/v1/timelines/home which, curiously enough, does exist and returns data without needing a user identifier:

2023-06-14 15:00:17.950783 CST: Starting FediFetcher                                                                                                                                                   [13/97320]2023-06-14 15:00:18.054776 CST: Error getting timeline toots: HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //social.example.net/api/v1/timelines/home (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7feb923cd6a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2023-06-14 15:00:18.054923 CST: Job failed after 0:00:00.104140.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
                                                                                                                                                                                                                 During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 358, in connect
    self.sock = conn = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7feb923cd6a0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //social.example.net/api/v1/timelines/home (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7feb923cd6a0>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/admin/FediFetcher/find_posts.py", line 855, in <module>
    timeline_toots = get_timeline(arguments.server, token, arguments.home_timeline_length)
  File "/home/admin/FediFetcher/find_posts.py", line 218, in get_timeline
    response = get_toots(url, access_token)
  File "/home/admin/FediFetcher/find_posts.py", line 250, in get_toots
    response = get( url, headers={
  File "/home/admin/FediFetcher/find_posts.py", line 685, in get
    response = requests.get( url, headers= h, timeout=timeout)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //social.example.net/api/v1/timelines/home (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7feb923cd6a0>: Failed to establish a new connection: [Errno 111] Connection refused'))
csolisr commented 1 year ago

OK, I just found that I had to remove the "https://" part from the URL field.

nanos commented 1 year ago

Yeah, that's because you should provide the server name, instead of the URL 😉

You are not the first to make this mistake, though, so since v5.0.0 FediFetcher will accept the URL there too.

Is this otherwise working for you now @csolisr ?

csolisr commented 1 year ago

Sure it does, except for the part of requiring the --reply-interval-in-hours=0. I do have another peeve but that's for another issue. Thanks!