Only works sometimes on Twitter

[Ereshkigal101]

Hey again! I've let it run for almost a day now and for some reason, it's just picking and choosing some images to give the source to. I've set it on a 5 minute delay in between checks so it doesn't eat up all the API checks. When it does skip a tweet, however, it says that "there's no downloadable media" when there's clearly an image there on the tweet.

[rainyroads]

Hey @Ereshkigal101,

Twitter's API is.. weird, sometimes. It shouldn't happen often, but sometimes the API will not return that a tweet has media content/images uploaded with it when it clearly does. I have no idea why, but this should be rare.

Can you provide some tweets where you've seen this error as a reference? If this is a frequent issue for you I'll see if I can escalate this as a bug report to Twitter and see why it's happening.

[Ereshkigal101]

https://twitter.com/WaifusCentral/status/1271694165643071494?s=20 https://twitter.com/WaifusCentral/status/1271663940028846080?s=20 https://twitter.com/WaifusCentral/status/1271656340260339713?s=20 https://twitter.com/WaifusCentral/status/1271648786310365186?s=20

There's a few that haven't had any reply. (Heads up btw, most are NSFW)

[rainyroads]

Yeah, Twitter isn't returning any media content for those tweets.

{
  "created_at": "Sat Jun 13 06:41:30 +0000 2020",
  "id": 1271694165643071494,
  "id_str": "1271694165643071494",
  "text": "//   New Hentai Order (18+)   //\\n\\ud83c\\udf39https://t.co/0Zl3OUmnEF\\ud83c\\udf39\\n#Hentai #Lewd #NSFW\\n\\ud83c\\udf3a@JevanaRena\\n\\ud83d\\udc9e@KittyPervert\\n\\ud83c\\udf3a\\u2026 https://t.co/QRuy53Yxc0",
  "truncated": true,
  "entities": {
    "hashtags": [
      {
        "text": "Hentai",
        "indices": [
          59,
          66
        ]
      },
      {
        "text": "Lewd",
        "indices": [
          67,
          72
        ]
      },
      {
        "text": "NSFW",
        "indices": [
          73,
          78
        ]
      }
    ],
    "symbols": [

    ],
    "user_mentions": [
      {
        "screen_name": "JevanaRena",
        "name": "\\u304a\\u7a7a \\u2603\\ufe0fOkuu\\ud83c\\udf38\\ud83c\\udf84",
        "id": 720348889857953793,
        "id_str": "720348889857953793",
        "indices": [
          80,
          91
        ]
      },
      {
        "screen_name": "KittyPervert",
        "name": "*.\\u00b0\\ud83d\\udd1e\\u210d\\u0454nt\\u03b1i Kitty \\u2741  \\ufeec",
        "id": 704627899127635968,
        "id_str": "704627899127635968",
        "indices": [
          93,
          106
        ]
      }
    ],
    "urls": [
      {
        "url": "https://t.co/0Zl3OUmnEF",
        "expanded_url": "http://discord.gg/rmDwe2m",
        "display_url": "discord.gg/rmDwe2m",
        "indices": [
          34,
          57
        ]
      },
      {
        "url": "https://t.co/QRuy53Yxc0",
        "expanded_url": "https://twitter.com/i/web/status/1271694165643071494",
        "display_url": "twitter.com/i/web/status/1\\u2026",
        "indices": [
          110,
          133
        ]
      }
    ]
  },
  "source": "<a href=\\"https://twitter.com/QueenIllyasviel\\" rel=\\"nofollow\\">Normal Tweets</a>",
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "in_reply_to_screen_name": null,
  "user": {
    "id": 1043440161466605569,
    "id_str": "1043440161466605569",
    "name": "\\ud83c\\udf3a Illya \\ud83c\\udf3a",
    "screen_name": "WaifusCentral",
    "location": "J-List Affiliate",
    "description": "18+ ONLY! The best account on Twitter for waifus! | No pics/vids are mine unless stated! | DM for Image Removal! Contact via Discord: Illya#0001",
    "url": "https://t.co/WWTr10Yi1m",
    "entities": {
      "url": {
        "urls": [
          {
            "url": "https://t.co/WWTr10Yi1m",
            "expanded_url": "http://www.discord.gg/PYumCDR",
            "display_url": "discord.gg/PYumCDR",
            "indices": [
              0,
              23
            ]
          }
        ]
      },
      "description": {
        "urls": [

        ]
      }
    },
    "protected": false,
    "followers_count": 56046,
    "friends_count": 1276,
    "listed_count": 428,
    "created_at": "Sat Sep 22 10:01:35 +0000 2018",
    "favourites_count": 19278,
    "utc_offset": null,
    "time_zone": null,
    "geo_enabled": false,
    "verified": false,
    "statuses_count": 43504,
    "lang": null,
    "contributors_enabled": false,
    "is_translator": false,
    "is_translation_enabled": false,
    "profile_background_color": "000000",
    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_tile": false,
    "profile_image_url": "http://pbs.twimg.com/profile_images/1269901955658395648/bYw2wX-q_normal.jpg",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/1269901955658395648/bYw2wX-q_normal.jpg",
    "profile_banner_url": "https://pbs.twimg.com/profile_banners/1043440161466605569/1590868855",
    "profile_link_color": "E81C4F",
    "profile_sidebar_border_color": "000000",
    "profile_sidebar_fill_color": "000000",
    "profile_text_color": "000000",
    "profile_use_background_image": false,
    "has_extended_profile": false,
    "default_profile": false,
    "default_profile_image": false,
    "following": false,
    "follow_request_sent": false,
    "notifications": false,
    "translator_type": "none"
  },
  "geo": null,
  "coordinates": null,
  "place": null,
  "contributors": null,
  "is_quote_status": false,
  "retweet_count": 1,
  "favorite_count": 10,
  "favorited": false,
  "retweeted": false,
  "possibly_sensitive": true,
  "possibly_sensitive_appealable": false,
  "lang": "en"
}

There should be a media dict in the entities or extended_entities property here that contains the image included in the Tweet, but there isn't anything returned by the official API.

I'm not sure why this happens but I'll look into it.

[rainyroads]

Actually.. this was pretty easy to figure out, lol. https://stackoverflow.com/a/40753183

Turns out this is a compatibility issue, and I just need to update the application to pass tweet_mode='extended' to mitigate the issue.

I believe the reason this normally is a rare issue is because the bug only presents itself when tweets are over 140 characters in length, as that's when the default "compatibility" mode kicks in, and apparently this breaks media embed for some reason when that happens. Even though media embeds have nothing to do with Tweet length, lol. http://docs.tweepy.org/en/latest/extended_tweets.html

I'll have this fixed in a commit shortly.

[Ereshkigal101]

Hey once again, it still seems to be having this issue, it's been doing it for EVERY single tweet that's been put out on my twitter page since launching it an hour and a half ago (4 in total). I'll obviously keep it going but just wanted to let you know that it's still happening.

[rainyroads]

@Ereshkigal101 Sorry about that! Can you update to the latest commit and try again?

[Ereshkigal101]

Of course, thanks for being awesome by the way! Lemme try it right now

[Ereshkigal101]

Alright... well the media bug is gone supposedly, but now there's a new one to replace it lol. This time, it's giving me the error:

[rainyroads]

Alright... well the media bug is gone supposedly, but now there's a new one to replace it lol. This time, it's giving me the error:

Alright, so when using extended_mode, tweet.text is replaced by tweet.full_text. The Twitter API annoys me, lol.

Patched this now, hopefully that's the last thing.

[Ereshkigal101]

Let's hope so! Thanks again for all the help, I'll follow up on how it's doing when I wake up.

[Ereshkigal101]

Looks like it worked pretty well while I slept! It's not giving sauce to like 1 tweet every here and there, but for the most part it's working! Thanks for the help! ^.^

[rainyroads]

Awesome, I'm glad to hear it! Let me know if any other issues pop up