Unauthorized(401) when using tweets/search/recent

[rn4n]

Prerequisites

• [x] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filed: open issues
  • Checked the bug report guidelines

Description

I'm trying to get the recent tweets from an user excluding the retweets. The documentation says that I can use -is:retweet on my query and I tried this:

const client = new Twitter({
  consumer_key: process.env.TWITTER_API_KEY,
  consumer_secret: process.env.TWITTER_API_SECRET,
  access_token_key: process.env.TWITTER_ACCESS_TOKEN,
  access_token_secret: process.env.TWITTER_ACCESS_TOKEN_SECRET,
});

const getLatestTweets = async (limit = 20) => {
  const { data } = await client.get('tweets/search/recent', {
    query: 'from:levxyca -is:retweet',
    'tweet.fields': 'created_at,public_metrics,entities',
    max_results: limit,
  });

  if (!data) { /* Not important for this issue. :D */ }

  return data;
};

getLatestTweets();

The point is on query: 'from:levxyca -is:retweet'. If a change it to query: 'from:levxyca', I can get the user tweets but I need to remove the retweets manually. When I add the -is:retweet I get the error specified below.

Do I need to do any dealings with my query?

Actual behavior:

I'm getting a Unauthorized error:

:\Users\Me\Desktop\Repositories\pandadomalbot\node_modules\twitter-v2\build\TwitterError.js:23
        return new module.exports(json.title, json.status, json.detail);
               ^

TwitterError: Unauthorized
    at Function.module.exports.fromJson (C:\Users\Me\Desktop\Repositories\pandadomalbot\node_modules\twitter-v2\build\TwitterError.js:23:16)
    at Twitter.get (C:\Users\Me\Desktop\Repositories\pandadomalbot\node_modules\twitter-v2\build\twitter.js:43:49)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async getLatestTweets (C:\Users\Me\Desktop\Repositories\pandadomalbot\utils\twitter.js:35:20)
    at async getTodaysLiveAnnouncement (C:\Users\Me\Desktop\Repositories\pandadomalbot\utils\twitter.js:70:18)
    at async Object.exports.default (C:\Users\Me\Desktop\Repositories\pandadomalbot\commands\rt.js:9:19) {
  code: 401,
  details: 'Unauthorized'
}

Versions

1.1.0

[edsu]

Thanks for reporting this. I've noticed that the 401 gets returned if the query is anything but a simple word. For example changing query to obama biden causes it too.

[edsu]

I've also noticed that it will work if you don't configure your client with access_token_key and access_token_secret? This is what the twitter-v2 unit test is doing which passes with the complex query 'url:"https://medium.com" -is:retweet lang:en'.

I wonder if the v2 API rules around what can be done in queries are coming into play here? I wouldn't be completely surprised if it's a bug on Twitter's side too.

https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query#availability

[edsu]

Testing the equivalent query in Python leads me to the conclusion that the problem resides here in twitter-v2 and not on the Twitter side. This is good news because that means it should be easily fixable :-)

[edsu]

I've added a unit test that demonstrates the problem, which is currently failing:

https://github.com/edsu/twitter-v2/blob/master/test/e2e/search.js#L72-L105

[edsu]

I might be wrong, but it appears that the space in the query parameter gets converted to a + when calling URL.toString() which is correct. But somewhere in the underlying call to oauth-1.0a/OAuth.authorize here any + in the URL are being converted to %20 prior to signing? This is causing the request authorization to be invalid.

[edsu]

More notes on this (sorry if they aren't helping). It looks like there's a mismatch in how oauth-1.0a and twitter-v2 are encoding spaces ' '.

The problem is that URL.toString() encodes a space ' ' in a query paramter as + whereas encodeURIComponent() encodes a space ' ' as %20. Since the URL being GET doesn't match the URL that was signed Twitter's API is rightly throwing a 401.

> encodeURIComponent(' ')
'%20'
> u = new URL('http://example.com')
URL {
  href: 'http://example.com/',
  origin: 'http://example.com',
  protocol: 'http:',
  username: '',
  password: '',
  host: 'example.com',
  hostname: 'example.com',
  port: '',
  pathname: '/',
  search: '',
  searchParams: URLSearchParams {},
  hash: ''
}
> u.searchParams.set('foo', 'bar baz')
undefined
> u.toString()
'http://example.com/?foo=bar+baz'

[edsu]

I have submitted a PR with a fix #93. I really depend on this library in one of my projects and have adjusted my package.json to point to edsu/twitter-v2 until some kind of fix is available here.

[rn4n]

@edsu In one of the projects that I contribute, I also switched to use your fork too. Thanks!