search

robertoszek / pleroma-bot

Bot for mirroring one or multiple Twitter accounts in Pleroma/Mastodon/Misskey.
https://robertoszek.github.io/pleroma-bot
MIT License
104 stars 18 forks source link

requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: #68

Closed vserbu closed 2 years ago

vserbu commented 2 years ago

For some reason, some of the twitter account result in error, and previously those accounts worked. Last one is @Ultrekillblast and before that it was official @startrek account. I got this:

... ℹ 2022-02-22 18:23:31,681 - pleroma_bot - INFO - tweets gathered: 27 ℹ 2022-02-22 18:23:34,148 - pleroma_bot - INFO - tweets to post: 27 ℹ 2022-02-22 18:23:34,150 - pleroma_bot - INFO - (1/27) ✖ 2022-02-22 18:23:34,402 - pleroma_bot - ERROR - Exception occurred (cli.py:502) Traceback (most recent call last): File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 103, in post_pleroma response.raise_for_status() File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/media

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 482, in main post_id = user.post_pleroma( File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 117, in post_pleroma response.raise_for_status() File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/media ...

robertoszek commented 2 years ago

Hi @vserbu ! What version of pleroma-bot are you using?

vserbu commented 2 years ago

I'm still on the 1.0.1 version. How do I update on newer version?

robertoszek commented 2 years ago

If you installed it using pip, you can upgrade to the latest version using this command:

pip install pleroma-bot --upgrade
vserbu commented 2 years ago

I did it, but the error is the same. It starts to post tweets and then blocks on one (in that example it was on second one) and that is it. Is there any cache to clear or something? I tried to limit the size of the tweet because I taught maybe it could be some big video or image, but it is not.

robertoszek commented 2 years ago

Would you mind sharing the output of your second try?

And maybe also your current config file (with the tokens or any sensitive info removed).

vserbu commented 2 years ago

Yes, I'm seeing it is somewhat different:

... ℹ 2022-02-23 08:53:03,014 - pleroma_bot - INFO - Processing user: 2139 ℹ 2022-02-23 08:53:04,529 - pleroma_bot - INFO - tweets gathered: 4 ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - tweets to post: 4 ℹ 2022-02-23 08:53:05,787 - pleroma_bot - INFO - (1/4) ✖ 2022-02-23 08:53:05,930 - pleroma_bot - ERROR - Exception occurred Unprocessable Entity Validation failed: File content type is invalid, File is invalid File: /home/mastodon/tweets/1496229640171630596/0.mp4 (_pleroma.py:127) ⚠ 2022-02-23 08:53:05,931 - pleroma_bot - WARNING - Error uploading media: 422 (_pleroma.py:134) ✖ 2022-02-23 08:53:05,984 - pleroma_bot - ERROR - Exception occurred (cli.py:536) Traceback (most recent call last): File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 513, in main post_id = user.post_pleroma( File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 167, in post_pleroma response.raise_for_status() File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses ...

robertoszek commented 2 years ago

It looks like the Mastodon instance is probably reporting the wrong type for this attachment: https://video.twimg.com/tweet_video/FMOs2sNXMAEWLxe.mp4

$ file 0.mp4 
0.mp4: ISO Media

Hence, failing the validation. The instance wrongly expects an ISO (.iso) file (or something else than an .mp4) to be uploaded based on running the file utility.

pleroma-bot can't do much here, the issue lies on the instance's file command reporting the wrong type. It could be that it's an old version and updating to a more recent version would fix it.

This shouldn't block you from using the bot normally, for these cases we skip the upload of the problematic media and try to publish the tweet's text anyway. But in this case, the tweet's text is empty and it only has the attachment, so it also fails to post an empty status.

robertoszek commented 2 years ago

I've published a release candidate version which won't fail due to the empty status (it will skip posting it), so at least it's not a fatal error and you could continue using the bot for publishing the rest of the tweets if you so choose. If you want to try it you can install it like so:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc2

However, the file issue will still occur, the problematic attachment won't be uploaded because the Mastodon instance won't recognize it as an mp4 media file, there's not much the bot can do about that.

vserbu commented 2 years ago

I'll give it a try, tnx :)

vserbu commented 2 years ago

It's working, but I have a new problem. Visibility is suddenly not public even dough I stated visibility: "public" already. Now the new tweets are not in LOCAL timeline, only in my HOME timeline.

robertoszek commented 2 years ago

Nice catch! I somehow missed an edge-case where the visibility is defined as a global value, sorry about that! Feel free to give it a try with 1.0.3rc3:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc3
vserbu commented 2 years ago

Still the same problem :(

robertoszek commented 2 years ago

Really? That's odd, I thought I had found the issue. Could you share your current config? (without the tokens or sensitive info)

vserbu commented 2 years ago 
pleroma_base_url: https://salocha.online
max_tweets: 40
twitter_token: xxxxx
users:
- twitter_username: TheSpaceshipper
  pleroma_username: 1771
  pleroma_token: xxx
- twitter_username: TrekNewsNet
  pleroma_username: 1792
  pleroma_token: xxx
- twitter_username: retroscifiart
  pleroma_username: 1808
  pleroma_token: xxx
- twitter_username: 70sscifi
  pleroma_username: 1812
  pleroma_token: xxx
- twitter_username: tosgifs
  pleroma_username: 1833
  pleroma_token: xxx
- twitter_username: ultrakillblast
  pleroma_username: 2139
  pleroma_token: xxx
visibility: "public"
include_replies: false
file_max_size: 5MB
include_rts: false
vserbu commented 2 years ago

It looks like it's working now!

robertoszek commented 2 years ago

Great! I was about to say that I could not seem to be able to reproduce the visibility issue on my side with your config on 1.0.3rc3 haha

Maybe I should've clarified that it only would apply to new posts published after upgrading to 1.0.3rc3 (the ones published already would've kept the wrong visibility).

I'll include these fixes on the next stable release as well. Have a nice day!

robertoszek commented 2 years ago

And thank you for finding these issues and reporting them as well!

vserbu commented 2 years ago

Thank you for quick response :)

vserbu commented 2 years ago

I think we have another Issue regarding this. The script stuck at:

ℹ 2022-02-23 20:03:35,112 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-23 20:03:35,113 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-23 20:03:35,119 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-23 20:03:35,120 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-23 20:03:38,831 - pleroma_bot - INFO - tweets gathered:        18 
Processing tweets...  |

And that is it....stuck.

When I removed that account (#) everything else went ok.

robertoszek commented 2 years ago

I may have an idea of what's happening there. Does running it with this version make any difference?

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc5
vserbu commented 2 years ago

It is not stucked any more but we are back with the similar error as on the beggining.


ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - config path: /home/mastodon/config.yml 
ℹ 2022-02-24 10:01:19,351 - pleroma_bot - INFO - tweets temp folder: /home/mastodon/tweets 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - ====================================== 
ℹ 2022-02-24 10:01:19,364 - pleroma_bot - INFO - Processing user:       1771 
ℹ 2022-02-24 10:01:21,926 - pleroma_bot - INFO - tweets gathered:        34 
ℹ 2022-02-24 10:01:24,416 - pleroma_bot - INFO - tweets to post:         7 
ℹ 2022-02-24 10:01:24,417 - pleroma_bot - INFO - (1/7) 
✖ 2022-02-24 10:01:24,871 - pleroma_bot - ERROR - Exception occurred (cli.py:537) 
Traceback (most recent call last):
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 514, in main
    post_id = user.post_pleroma(
  File "/home/mastodon/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/mastodon/.local/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://salocha.online/api/v1/statuses
robertoszek commented 2 years ago

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

Would you mind testing out my theory by running this version?:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc6
vserbu commented 2 years ago

It's working, you were absolutely - right! tnx!

robertoszek commented 2 years ago

Nice! I should be the one thanking you :) Thanks to your reporting we found these issues and the fixes will be included on the next stable release. Cheers!

nemobis commented 2 years ago

Hmm... I'm wondering if you're hitting Mastodon's character limit (500 per toot) with a quoted tweet.

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

I see that's actually what you were trying to do, not sure what went wrong: https://github.com/robertoszek/pleroma-bot/commit/8f44c53dbdf5535543adc5ec895d5cd8810bd5cd#diff-46774f4895c169807f7a3d85cc4ebc34a18ac37e7f4b2ed7306d5ef6977e59adR156

robertoszek commented 2 years ago

When truncating a toot, it might be worth truncating before the signature, to avoid breaking links: see https://respublicae.eu/@Europarl_DE/108068732034451913 for an example.

@nemobis Oh, right! I missed how this would happen when a signature and original date are both enabled. it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc8
nemobis commented 2 years ago

Il 03/04/22 20:23, robertoszek ha scritto:

it should be fixed on 1.0.3rc8, feel free to give it a spin and see if the changes solved it:

Yes, thank you! All good on 1.0.3rc9.

nemobis commented 2 years ago

I'm still seeing some HTTP 422. Consistently for this one account:

ℹ 2022-04-10 09:29:33,942 - pleroma_bot - INFO - Processing user:       108070048904140658
ℹ 2022-04-10 09:29:36,909 - pleroma_bot - INFO - tweets gathered:        49
ℹ 2022-04-10 09:29:44,172 - pleroma_bot - INFO - tweets to post:         49
ℹ 2022-04-10 09:29:44,173 - pleroma_bot - INFO - (1/49)
✖ 2022-04-10 09:29:56,582 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:556) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 536, in main
    tweet["possibly_sensitive"],
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 605, in post
    post_id = self.post_pleroma(tweet, poll, sensitive)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 170, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses

I think it might be one of these tweets (cw: controversial political content):

robertoszek commented 2 years ago

@nemobis For those specific 2 tweets you provided, I've not been able to reproduce the issue, they posted fine on a Mastodon instance on my testing (with or without signature/original date too).

Maybe the culprit is some other tweet. You can try running the bot in verbose mode and see if it helps to narrow it down.

nemobis commented 2 years ago

Il 10/04/22 23:40, robertoszek ha scritto:

Maybe the culprit is some other tweet.

Yeah I don't know, I'll add some logging locally to find out.

nemobis commented 2 years ago

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

robertoszek commented 2 years ago

I keep getting HTTP 422 for https://respublicae.eu/@EP_Defence possibly due to https://nitter.eu/EP_Defence/status/1513526853532897283#m . Could it be because there are 4+2 images there considering the quoted tweet?

@nemobis Right! Nice catch, Mastodon only allows 4 images per post, possibly that's the cause. (Misskey and Pleroma allow 16).

I've added the upper limit for Mastodon to 1.0.3rc13, feel free to try it out and see if you still get an HTTP 422 for the same tweet:

pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot==1.0.3rc13
nemobis commented 2 years ago

Some quote-tweets with 5+ photos are still getting an HTTP 422 on the most recent release, though I'm not quite sure why (I even hacked the Mastodon code to raise the limit to 8).

Culprit seems to be https://nitter.eu/andreykovatchev/status/1534110522722996224#m

ℹ 2022-07-30 01:02:44,385 - pleroma_bot - INFO - Processing user:       108069143892397143
ℹ 2022-07-30 01:02:46,464 - pleroma_bot - INFO - tweets gathered:        7
ℹ 2022-07-30 01:02:50,080 - pleroma_bot - INFO - tweets to post:         7
ℹ 2022-07-30 01:02:50,081 - pleroma_bot - INFO - (1/7)
✖ 2022-07-30 01:02:57,280 - pleroma_bot - ERROR - Exception occurred for user, skipping... (cli.py:574) 
Traceback (most recent call last):
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/cli.py", line 554, in main
    tweets_to_post["media_processed"]
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_utils.py", line 620, in post
    post_id = self.post_pleroma(tweet, poll, sensitive, media)
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/pleroma_bot/_pleroma.py", line 192, in post_pleroma
    response.raise_for_status()
  File "/home/7/federico/mastodon/lib/python3.7/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://respublicae.eu/api/v1/statuses
nemobis commented 2 years ago

On the other hand a bunch of posts which previously failed are now succeeding, like https://respublicae.eu/@Cedefop/108732982556276877 from https://nitter.eu/Cedefop/status/1542082206641840129 . :shrug:

nemobis commented 1 year ago

I'm getting an HTTP 422 while trying to post (I think) this https://nitter.it/NatalieAmiri/status/1586320608538902528#m , a tweet with a video quoting another tweet with a video.

It would be easier to debug if the exception printed the original tweet ID too.

nemobis commented 1 year ago

A RT https://nitter.it/EmmanuelMacron/status/1591869166046380035#m failed too, maybe any quote tweet with video fails?

robertoszek commented 1 year ago

Ah, for the first tweet you mentioned it's probably because I don't think Mastodon supports multiple videos on the same post: https://user-images.githubusercontent.com/1080963/202745994-93c67a8f-417e-465c-9f0a-0f69c61e7085.png

Hmm I need to think what's the best way to handle that. I assume it's better to drop the video on the quoted/retweeted tweet instead of the one attached to the user you're mirroring. You'll lose some context either way, which is a shame.

Ah, I see it's the same problem for the other tweet too (multiple videos): https://user-images.githubusercontent.com/1080963/202748467-169040d2-e560-4b2d-9a31-ff607e7035a7.png

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

nemobis commented 1 year ago

Il 18/11/22 18:06, robertoszek ha scritto:

I'll open a new issue to keep track of it, as the "422 Unprocessable Entity" error is too generic and could be multiple things causing it.

Thanks!