Firefish Github action error

[Y0ngg4n]

Run python find_posts.py --lock-hours=0 --access-token=*** -c="./config.json"
Traceback (most recent call last):
2023-09-13 12:50:27.033317 UTC: Starting FediFetcher
2023-09-13 12:50:27.659332 UTC: Job failed after 0:00:00.626002.
  File "/home/runner/work/FediFetcher/FediFetcher/find_posts.py", line 1362, in <module>
    reply_toots = get_all_reply_toots(
  File "/home/runner/work/FediFetcher/FediFetcher/find_posts.py", line 410, in get_all_reply_toots
    reply_toots = list(
  File "/home/runner/work/FediFetcher/FediFetcher/find_posts.py", line 411, in <genexpr>
    itertools.chain.from_iterable(
  File "/home/runner/work/FediFetcher/FediFetcher/find_posts.py", line 400, in get_active_user_ids
    raise Exception(
Exception: Error getting user IDs on server firefish.place. Status code: 404

[Teqed]

Problem

config.json referenced:

{
    "server": "...",
    "home-timeline-length": 40,
    "max-bookmarks": 40,
    "max-favourites": 40,
    "max-followings": 40,
    "max-followers": 40,
    "max-follow-requests": 40,
    "from-notifications": 7,
    "reply-interval-in-hours": 72,
    "remember-users-for-hours": 12,
    "log-level": 10
}

Because reply_interval_in_hours > 0, FediFetcher is attempting to get replies for users using get_reply_toots(), which expects a Mastodon API compatible user ID, fetched from the Mastodon accounts admin endpoint /api/v1/admin/accounts.

However, Firefish does not appear to implement this endpoint -- at the very least, it returns status code 404.

In particular, this endpoint is needed because by default it provides a list of all local users, their Mastodon-compatible ID, and the date of their last status post.

Firefish has a similar API endpoint at /api/admin/show-users, which you can provide { "origin": "local" } to and get a similar result:

However, to use Firefish's Mastodon-compatible API at /api/v1/accounts/:id/statuses for get_reply_toots(), a Mastodon-compatible ID for the user must be provided (numeric), and not their native Firefish database user ID (alphanumeric). It's possible to obtain these by using Firefish's Mastodon-compatible /api/v2/search endpoint...

...but this is starting to multiply requests for something that at this point would need to be solved individually for Firefish, possibly within its own API.

Solutions?

Set reply_interval_in_hours = 0. This will disable this particular feature, which currently relies on a Mastodon API not implemented by Firefish.

FediFetcher could do for more robust error handling that catches a 404 here and continues with execution.

A Firefish-specific implementation of FediFetcher might be able to circumvent this limitation.

[Y0ngg4n]

@Teqed so should we make a firefish issue for that?

I have set reply_interval_in_hours = 0 but it does not fix the problem.

[Teqed]

@Y0ngg4n Your most recent workflow run is a re-run of an old workflow that is based on commit abf8f3860f5487ac855856560f8fa27b3d706e9c. You'll need to issue a new workflow run, which should default to your latest commit.

so should we make a firefish issue for that?

Currently the recommended value for reply-interval-in-hours in the README.md is zero (disabled), but if you are personally motivated to make use of this feature, you could submit a PR to the Misskey family that implements the missing Mastodon API endpoint. I believe the Mastodon API implementation was done by the upstream Misskey project, but the Firefish project may also be interested in working on improving their Mastodon API.

[nanos]

Thanks for the report @Y0ngg4n

Can I just clarify if you are encountering this error because your own instance is a Firefish instance, or if you are encountering this error when you are pulling in remote posts from a Firefish instance?

[Y0ngg4n]

@Teqed ah yeah thank you. sorry that was dumb. Now i get this errors: https://github.com/Y0ngg4n/FediFetcher/actions/runs/6183764988/job/16786123104

@nanos my instance is a firefish instance

[nanos]

Thanks @Y0ngg4n

In that case I'd like to point you to #72 - FediFetcher doesn't currently have Firefish support, simply because I don't have a firefish instance. I'd happily accept PRs that fix this, though.

I'll therefore close this issue now, but feel free to continue discussion with @Teqed here, for as long as you feel it's helpful.

One final comment:

FediFetcher could do for more robust error handling that catches a 404 here and continues with execution.

IMO it's better to fail, so that users know there is a problem, than to continue as though there was no problem, and then leave people wondering why their replies aren't being fetched.

[Teqed]

Now i get this errors: Y0ngg4n/FediFetcher/actions/runs/6183764988/job/16786123104

Looking at the logs here, it appears that FediFetcher is successfully adding context URLs up until it suddenly starts returning status code 401 (Unauthorized). This looks like a rate limit (429) that is incorrectly returning unauthorized (401). Sorry to say that this re-confirms the original issue described on #72.

[Y0ngg4n]

@Teqed hm yeah thats a bummer. maybe because i am using the release version on the server and not the one from the development branch.

[Teqed]

FediFetcher could do for more robust error handling that catches a 404 here and continues with execution.

IMO it's better to fail, so that users know there is a problem, than to continue as though there was no problem, and then leave people wondering why their replies aren't being fetched.

Another way to tackle this design would be to catch exceptions at each feature module, and then exit with a failure status code if any of the modules returned an error. This would still alert users to problems without causing other working features from functioning, which may be useful in scenarios where one of our API calls gets an unexpected response from downstream Mastodon servers. -- But in this instance I agree that it's fine the way it is.

[Teqed]

@Teqed hm yeah thats a bummer. maybe because i am using the release version on the server and not the one from the development branch.

Firefish's default branch is develop, unless you deliberately checked out the main or beta branches.

Misskey / Firefish seems to return the status codes in the message body, and I believe the status code I saw earlier was actually from the body and not the actual status code.

[Y0ngg4n]

@Teqed yeah but maybe they did not made a release of the develop branch to the docker containers.