API doesn't work with some twitter videos

[Pimentoso]

Bug Report

Current Behavior

Trying to call API using some twitter posts URL and specifying video = true results in an error.

I'm using the PRO endpoint

sample url: https://twitter.com/k_k_k_k_k_k_k_l/statuses/1215411759537889280

API request: https://pro.microlink.io/?url=https%3A%2F%2Ftwitter.com%2Fk_k_k_k_k_k_k_l%2Fstatuses%2F1215411759537889280&video=true

result using video = true:

{:data=>
  {:title=>"1215411759537889280",
   :description=>
    "Something went wrong, but don’t fret — let’s give it another shot.",
   :lang=>"en",
   :author=>nil,
   :publisher=>"Twitter",
   :image=>nil,
   :url=>"https://twitter.com/k_k_k_k_k_k_k_l/statuses/1215411759537889280",
   :date=>"2020-01-23T10:52:37.000Z",
   :logo=>
    {:url=>"https://logo.clearbit.com/twitter.com",
     :type=>"png",
     :size=>2089,
     :height=>128,
     :width=>128,
     :size_pretty=>"2.09 kB"},
   :video=>nil},
 :status=>"success"}

result without params (works as expected)

{:data=>
  {:title=>"jak on Twitter",
   :description=>
    "“@realDonaldTrump Iranian media publishes a video of the terrorist Abu Mahdi Al-Muhandis who was killed accompanied by Soleimani speaking in the Persian language to one of the channels of the Iranian Revolutionary Guards and says:I wish to see the destruction of Saudi Arabia before I die\n" +
    "#أرامكو https://t.co/FyDD5rM64c…",
   :lang=>"en",
   :author=>"k_k_k_k_k_k_k_l",
   :publisher=>"Twitter",
   :image=>
    {:url=>
      "https://pbs.twimg.com/ext_tw_video_thumb/1215411616390344704/pu/img/gO7fduP-u4Q0I8fL.jpg",
     :type=>"jpg",
     :size=>27718,
     :height=>360,
     :width=>640,
     :size_pretty=>"27.7 kB"},
   :url=>"https://twitter.com/k_k_k_k_k_k_k_l/status/1215411759537889280",
   :date=>"2020-01-23T10:48:28.000Z",
   :logo=>
    {:url=>"https://logo.clearbit.com/twitter.com",
     :type=>"png",
     :size=>2089,
     :height=>128,
     :width=>128,
     :size_pretty=>"2.09 kB"}},
 :status=>"success"}

Expected behavior/code

I expect to have a video entity in the response, or at least not to receive a 'success' response code.

[Kikobeats]

Hello,

This is a known issue we've for a long time and unfortunately, we don't have an immediate solution.

The first problem here that Twitter doesn't have a formal API for getting the video. The extraction is done on browser-side, following the same way as a user can see the video from his Twitter account.

Most of the time this is working fine, it randomly fails: it doesn't return anything for a short period of time, and after that, the video is available an again.

Although The twitter video URL is not present on the response payload, the rest of the data is correct. That's why the API returns a 200 OK.

Let me ask something: Do you need the video for creating a link preview, or just for consuming it from your backend side?

[Pimentoso]

Hi, I understand your point. The response of your API gets stored on our backend and is later used to create a link preview. It would be nice to have the API fallback to the behavior it would have without the video=true parameter, so at least we get a video thumbnail in the 'image' entity (like in my second example)

[Kikobeats]

We created Microlink SDK for consuming Microlink API in a way you can create beauty links previews without worries 🙂

It supports to setup media fallbacks:

import Microlink from '@microlink/react'

export default () => (
  <Microlink
    url='https://twitter.com/k_k_k_k_k_k_k_l/statuses/1215411759537889280'
    media={['video', 'image']}
  />
)

In that code snippet, if the video isn't available, the image will be used.

Recently, we added support for iframe. That's the preferred method for services like Twitter, where they don't have a formal way for retrieving the video, but using the iframe you can confident the video will be there.

You can use it as media as well:

import Microlink from '@microlink/react'

export default () => (
  <Microlink
    url='https://twitter.com/k_k_k_k_k_k_k_l/statuses/1215411759537889280'
    media={['iframe', 'video', 'image']}
  />
)

Microlink SDK is available as React or Vanilla bundle. is it suitable for your codebase?

If you prefer to continue with your backend approach, that's okay but be careful about the values stored in your DB: Eventually, the links can be expired.

For example, an Instagram image is served from Instagram CDNs and they have a TTL of 4 hours; after that the change, so you need to pass the URL against Microlink API for getting the new URL.

That's why we created Microlink SDK that makes this mechanism transparent

[Pimentoso]

Yeah the whole embed part of our project would surely benefit from a refactor from the ground up, but sadly it's not a priority at the moment. I'll talk to the frontend guys if the microlink react library is applicable. Thanks!