Closed jimmybrancaccio closed 3 years ago
Hi! Hmmm, could you provide me with the Twitter account's username so I can maybe try to reproduce the issue? (if it's a private account that you don't want to give away publicly, consider contacting me privately).
I think I have a hunch of what part of the code is misbehaving, but I'd have to debug it to make sure. Also, if you're comfortable with making changes locally, maybe consider adding a line right after this one: https://github.com/robertoszek/pleroma-twitter-info-grabber/blob/db9cf0ee424e00d18331e21ba39fede6bf4d7199/pleroma_bot/cli.py#L285 with the following content:
logger.info(f"tweets: \t {tweets}")
And paste here the output you get, that way I can analyze Twitter's API response and see the JSON myself. Thanks!
@robertoszek I've added that logger line to cli.py
. The error is a bit puzzling because it was posting tweets for this user:
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45792
pleroma_bot - INFO - tweets: {'meta': {'result_count': 0}}
pleroma_bot - INFO - Current pinned: 1345041202492526592
pleroma_bot - INFO - Previous pinned: 1345041202492526592
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45792
pleroma_bot - INFO - tweets: {'errors': [{'value': '15488316', 'parameter': 'id', 'resource_type': 'user', 'section': 'data', 'title': 'Authorization Error', 'detail': 'Sorry, you are not authorized to see the user with id: [15488316].', 'type': 'https://api.twitter.com/2/problems/not-authorized-for-resource'}]}
pleroma_bot - ERROR - Exception occurred
Traceback (most recent call last):
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 287, in main
if tweets["meta"]["result_count"] > 0:
KeyError: 'meta'
Is the account private by any chance? Do you have a bearer token able to consult their tweets?
You can try doing the request directly and see if you get the same result:
curl -H "Authorization: Bearer <your-token-here>" 'https://api.twitter.com/2/tweets/search/recent?max_results=40&query=from%3A<twitter-account-here>&poll.fields=duration_minutes%2Cend_datetime%2Cid%2Coptions%2Cvoting_status&media.fields=duration_ms%2Cheight%2Cmedia_key%2Cpreview_image_url%2Ctype%2Curl%2Cwidth%2Cpublic_metrics&expansions=attachments.poll_ids%2Cattachments.media_keys%2Cauthor_id%2Centities.mentions.username%2Cgeo.place_id%2Cin_reply_to_user_id%2Creferenced_tweets.id%2Creferenced_tweets.id.author_id&tweet.fields=attachments%2Cauthor_id%2Ccontext_annotations%2Cconversation_id%2Ccreated_at%2Centities%2Cgeo%2Cid%2Cin_reply_to_user_id%2Clang%2Cpublic_metrics%2Cpossibly_sensitive%2Creferenced_tweets%2Csource%2Ctext%2Cwithheld'
Replacing <your-token-here>
with your token and <twitter-account-here>
with the Twitter username.
The account isn't private (Twitter username is Cryo
). Running the cURL command spews out a bunch, so it seems to work fine there. I can send the output, or you could try the cURL command from your end with that username. Let me know if you want the output from my attempt!
Actually, I forgot, I changed it recently and this is the actual request now:
curl -H "Authorization: Bearer <your-token-here>" 'https://api.twitter.com/2/users/<twitter-id>/tweets'
If you don't know the Twitter's account ID, you can get it here using their Twitter handle: https://tweeterid.com
Either way, I have no problem getting the Twitter account tweets with my token and I can't reproduce the issue while debugging pointing to that Twitter account. This is very interesting. Do you mind pasting your config here (with the tokens redacted)?
Here ya go!
twitter_base_url: https://api.twitter.com/1.1
# Change this to your Fediverse instance
pleroma_base_url: https://mastodon.linuxbox.ninja
# How many tweets to get in every execution
# Twitter's API hard limit is 3,200
max_tweets: 40
# Twitter bearer token
twitter_token: **************
# List of users and their attributes
users:
- twitter_username: cryo
pleroma_username: 45792
pleroma_token: ***************
media_upload: true
nitter: false
visibility: "public"
sensitive: false
include_rts: false
include_replies: false
support_account: jimmyb
bio_text: "\U0001F916 BEEP BOOP \U0001F916 \n I'm a bot that mirrors {{ twitter_username }}'s Twitter's\
\ account. \n \n "
Huh, is this the full config?
I can see it tried to process 2 users in the log (with the same pleroma_username
), that's very odd:
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45792
pleroma_bot - INFO - tweets: {'meta': {'result_count': 0}}
pleroma_bot - INFO - Current pinned: 1345041202492526592
pleroma_bot - INFO - Previous pinned: 1345041202492526592
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45792
I tried it with your config to get the tweets from January and ran it multiple times but I wasn't able to reproduce the issue.
Ok, that tipped me off. That wasn't the full config. The issue is I used the same pleroma_username
value for the account right below that account. I fixed that and then I get this:
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45792
pleroma_bot - INFO - Current pinned: 1345041202492526592
pleroma_bot - INFO - Previous pinned: 1345041202492526592
pleroma_bot - INFO - ======================================
pleroma_bot - INFO - Processing user: 45794
pleroma_bot - WARNING - No posts were found in the target Fediverse account
pleroma_bot - INFO - How far back should we retrieve tweets from the Twitter account?
pleroma_bot - INFO - Enter a date (YYYY-MM-DD):
pleroma_bot - INFO - [Leave it empty to retrieve *ALL* tweets or enter 'continue'
pleroma_bot - INFO - if you want the bot to execute as normal (checking date of
pleroma_bot - INFO - last post in the Fediverse account)]
2020-09-01
pleroma_bot - ERROR - Exception occurred
Traceback (most recent call last):
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 286, in main
if tweets["meta"]["result_count"] > 0:
KeyError: 'meta'
So running the cURL command before, gave me this:
{"meta":{"result_count":0}}
The actual account having an issue IS private.
If I leave the date empty, and just hit enter at the prompt, I get this:
leroma_bot - INFO - Processing user: 45794
pleroma_bot - WARNING - No posts were found in the target Fediverse account
pleroma_bot - INFO - How far back should we retrieve tweets from the Twitter account?
pleroma_bot - INFO - Enter a date (YYYY-MM-DD):
pleroma_bot - INFO - [Leave it empty to retrieve *ALL* tweets or enter 'continue'
pleroma_bot - INFO - if you want the bot to execute as normal (checking date of
pleroma_bot - INFO - last post in the Fediverse account)]
pleroma_bot - ERROR - Exception occurred
Traceback (most recent call last):
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 283, in main
date_pleroma = user.get_date_last_pleroma_post()
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 44, in get_date_last_pleroma_post
date_pleroma = self.force_date()
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/_utils.py", line 421, in force_date
datetime.strptime(input_date, "%Y-%m-%d"),
File "/usr/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/usr/lib/python3.8/_strptime.py", line 349, in _strptime
raise ValueError("time data %r does not match format %r" %
ValueError: time data '' does not match format '%Y-%m-%d'
Oh, test release v0.7.3 fixed that empty date business, you could try installing it like this:
pip install --index-url https://test.pypi.org/simple/ pleroma-bot --upgrade
And then run it against the private account, see if you get the same result.
Hrmm, it won't let me install the test version for whatever reason:
ERROR: Requested pleroma-bot from https://test-files.pythonhosted.org/packages/14/7b/71260a0da2e952f2e41d1eb794e5978a672ec78c0a2f4757672226bac406/pleroma_bot-0.4.5-py3-none-any.whl#sha256=d9f1f6b8c6253419894d6df746b529203e1f00922e22eb9756375f9f64270a97 has different version in metadata: '0.4.0'
However even if I manually copy the Python files from the site it fails.
pleroma_bot - INFO - Processing user: 45794
pleroma_bot - WARNING - No posts were found in the target Fediverse account
pleroma_bot - INFO - How far back should we retrieve tweets from the Twitter account?
pleroma_bot - INFO - Enter a date (YYYY-MM-DD):
pleroma_bot - INFO - [Leave it empty to retrieve *ALL* tweets or enter 'continue'
pleroma_bot - INFO - if you want the bot to execute as normal (checking date of
pleroma_bot - INFO - last post in the Fediverse account)]
pleroma_bot - ERROR - Exception occurred
Traceback (most recent call last):
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 294, in main
if tweets["meta"]["result_count"] > 0:
KeyError: 'meta'
Well, looking more into it, the root of the issue is that OAuth 2.0 Bearer tokens do not have access to protected tweets. Source: https://developer.twitter.com/en/docs/authentication/overview
The only way to retrieve those is to obtain OAuth 1.0a Consumer API Keys and Access Tokens performing a 3-legged OAuth so you can act on behalf of a Twitter account (which has to be following or be the owner of the protected account).
I'll try generating them and see if we can adapt the code to somehow support those, it's not a high priority right now to be honest. Marking the issue as an enhancement, as it never was designed to use OAuth 1.0a.
Hello again, it wasn't as painful to integrate as I initially thought, I've uploaded a new test release (v0.7.8), try installing it like so:
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot --upgrade
If you have trouble getting it installed, maybe try uninstalling it first:
pip uninstall pleroma-bot
After upgrading, you'll need to add to the protected user section in the config the following:
consumer_key: xxxxxxxxxxxxxxxxxxxxxxxxx
consumer_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
access_token_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
access_token_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
To obtain the consumer_*
you need to go to your app's keys and tokens (https://developer.twitter.com)
And for the access_*
, you can run this to obtain them by yourself:
https://github.com/joestump/python-oauth2/wiki/Twitter-Three-legged-OAuth-Python-3.0
I was able to reproduce the issue with a Twitter account with protected tweets and this fixed it for me. Tell me if you're able to get it working. Cheers!
Hey @robertoszek! It looks like it's working! The first time it errored out because of a 404 error (may want to consider some further error trapping/catching. But that didn't surprise me since I selected to get all tweets and the account hasn't had much activity since like 2017 and before (one of the images had a 404).
It looks like a lot of the users I have setup hadn't ever been synced which is odd because I thought I ran an initial sync when I got this all setup at the beginning of the month. They all seem to be working now though so no big deal!
I really appreciate the help and work to get this going. For what it's worth it looks like you can get all the necessary tokens/keys from the Twitter developer portal UI:
Excellent, good to hear it worked! You raise a good point about old attachments that may have been deleted, I've made some changes to be more permissive in case of 404's. 90d441af48d950463765c7dce8ed662ff3d446ae I completely missed the option in Twitter's developer portal to generate access tokens and ended up creating them myself 😅 haha Oh, well. Live and learn.
This issue will be automatically closed when a pull request merges the changes into master, don't hesitate to add additional comments if you find any issues with the implementation. Cheers!
The only other issue I seemed to have come across is accounts with no tweets cause an error. Perhaps catching that and just moving on would be good there. It seems silly to "sync" an account with no tweets, but for the sake of having "good" code, might as well as try to avoid dying and just showing a warning or something and moving on to the next account. 😁
I've not come across an error with Twitter accounts that have no tweets. It didn't fail for me when the account had no new tweets (since last Fediverse post) nor when an account had no tweets whatsoever (tried both with protected and public). And I wasn't able to make it happen either no matter if it was a first run or not.
What error gets raised? And did you happen to save the log?
Ok, it seems to only happen during the cronjob, but not when running the script manually.
ℹ 2021-01-30 12:00:54,975 - pleroma_bot - INFO - Processing user: 45841
⚠ 2021-01-30 12:00:55,496 - pleroma_bot - WARNING - No posts were found in the target Fediverse account (_pleroma.py:42)
ℹ 2021-01-30 12:00:55,497 - pleroma_bot - INFO - How far back should we retrieve tweets from the Twitter account?
ℹ 2021-01-30 12:00:55,497 - pleroma_bot - INFO - Enter a date (YYYY-MM-DD):
ℹ 2021-01-30 12:00:55,498 - pleroma_bot - INFO - [Leave it empty to retrieve *ALL* tweets or enter 'continue'
ℹ 2021-01-30 12:00:55,498 - pleroma_bot - INFO - if you want the bot to execute as normal (checking date of
ℹ 2021-01-30 12:00:55,498 - pleroma_bot - INFO - last post in the Fediverse account)]
✖ 2021-01-30 12:00:55,499 - pleroma_bot - ERROR - Exception occurred (cli.py:373)
Traceback (most recent call last):
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/cli.py", line 328, in main
date_pleroma = user.get_date_last_pleroma_post()
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/_pleroma.py", line 44, in get_date_last_pleroma_post
date_pleroma = self.force_date()
File "/home/jimmy/.local/lib/python3.8/site-packages/pleroma_bot/_utils.py", line 254, in force_date
input_date = input()
EOFError: EOF when reading a line
Perhaps because when the cron runs it doesn't get any input from the user for the date?
Perhaps because when the cron runs it doesn't get any input from the user for the date?
Yeah, that looks like it's exactly what's happening. The first time you add a new user you should run it manually at least once, so it doesn't require your input anymore in following executions.
That, or you can run your cron job with the -s
flag, which will ignore first run checks (not requiring your input) and continue as usual.
Bot for mirroring one or multiple Twitter accounts in Pleroma/Mastodon.
optional arguments:
-h, --help show this help message and exit
-n, --noProfile skips Fediverse profile update (no background image,
profile image, bio text, etc.)
--forceDate [FORCEDATE]
forces the tweet retrieval to start from a specific
date. The twitter_username value (FORCEDATE) can be
supplied to only force it for that particular user in
the config
-s, --skipChecks skips first run checks
--version show program's version number and exit
The first time you add a new user you should run it manually at least once, so it doesn't require your input anymore in following executions.
So I did it manually and gave it a date and it was fine. But now during cron runs it still has an issue. Would it be best to just add the -s
parameter to my cronjob(s) at this point?
Would it be best to just add the -s parameter to my cronjob(s) at this point?
That's what I'd try next, yes. Did the Fediverse account had any posts/toots published after the manual run or did it remain empty?
Oh! I forgot to add earlier that there have been a couple of minor fixes since the last test release, if you're keen of giving it a spin and see if it solves any of the issues you're experiencing (v0.7.9
)
pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple pleroma-bot --upgrade
@robertoszek It remained empty. After looking at the account it actually does have some tweets, which are pretty old but I guess it should still fetch them when I tell the script to get everything.
https://twitter.com/cowardswayout
So there may still an issue here.
I am giving 0.7.9 a shot now.
Actually, Twitter's API only lets you go back to '2010-11-06T00:00:00Z
', correct me if I'm wrong but seems there are no tweets newer than that date in this Twitter account.
I think you're definitely right. It looks like the newest Tweet is from Jul 16, 2010. Alright so, I think it would be best if I just ran my cron(s) with the -s
flag for now. Sound right?
It's safer that way, as it should never ask for your input.
However, this is an special case in my opinion, normally once you run it manually the first time (if you want to retrieve older tweets) you should be good to go as the Fediverse will have some posts/toots and the -s
flag won't be really necessary.
Your call, really. It won't hurt to include the --skipChecks
flag on your cron job anyway. 👍
@robertoszek Awesome, sound good. I really appreciate all your quick communication in getting me all setup. I think we can consider this/these issues resolved!
No worries, thank you for your patience and sticking through with the troubleshooting! haha I'll just leave the issue open as it will automatically close when the changes merge into the master branch, have a nice day!
Hiyo!
First off apologies on not getting around to testing 0.7.0 until now (#28, #29). Things were working initially for the first run it seems, but I am now getting an error:
It appears to be getting "stuck" at the same Twitter account during each run. Is there anything else I can provide to help troubleshoot?