403 Errors on Chinese Internet

tysonkerridge commented 5 years ago

So I’ve got a user who seems to not be able to use the app due to receiving errors for, I’m assuming, all API endpoints. What’s most interesting is the endpoints are returning 403s, with the one I got screenshots/videos of being the Trending Movies endpoint. They also attempted to sign in but end up getting errors once signing in, which I’m assuming is the Get Token endpoint.

They’re apparently seeing these issues when connected to their internet in China, which I assume could be part of the issue, but they seem to be able to log in successfully to the Trakt website, and also they seem to be able to use a VPN to connect outside of China and the app works when connected to that.

From my own debugging, I don’t see these issues else I’d have looked into it further myself, but also don’t have a way to test on a Chinese connection either.

Any ideas on what additional information I’d need to get from the user in order to work out what could be the issue in order to find a solution? I’d most likely have to make some changes to provide them some way to send me some logs, but I’m not sure what to include in the logs.

Thanks!

_{Sent with GitHawk}

rectifyer commented 5 years ago

My guess is Cloudflare is detecting a threat and preventing it from loading. On the website, they would put up a captcha for the user to complete, but I'm not sure what they do on an API call like this. There might be additional headers send in the API response, but I'm not sure what we can do since we need to keep threat protection in place.

tysonkerridge commented 5 years ago

Ahh yes, the user sent me a screen recording in which they were directed to a captcha entry, but then nothing seemed to ask them for Trakt username or password. Perhaps they did that in a previous session before recording so it kept them logged in or something when they tried to access the authentication sign in link again.

Thanks, I’ll let them know and see if I can get some more details.

_{Sent with GitHawk}

ShaneQi commented 5 years ago

Original	Translated

My app's Chinese users complaint that the API stopped working recently and one of them found out that requests that are sent to both trakt.tv and api.trakt.tv are asked to enter a verification code.

I was hoping you could contact CloudFlare and find a solution. They might be able to provide help with a secure config for the API that can prevent the CAPTCHA.

tysonkerridge commented 5 years ago

That captcha screen in the screenshot is the same as the one in the screen recording I was talking about.

_{Sent with GitHawk}

ShaneQi commented 5 years ago

More info:

curl --include \
     --header "Content-Type: application/json" \
     --header "trakt-api-version: 2" \
     --header "trakt-api-key:<REDACTED>" \
  'https://api.trakt.tv/shows/trending'

I sent this cURL command (generated by https://trakt.docs.apiary.io) to one of my users in China.

And the response he got back is:

HTTP/2 403
date: Tue, 23 Jul 2019 03:13:45 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d89f7edb99d57dda0a1defdabe28980ba1563851625; expires=Wed, 22-Jul-20 03:13:45 GMT; path=/; domain=.trakt.tv; HttpOnly
cache-control: max-age=2
expires: Tue, 23 Jul 2019 03:13:47 GMT
x-frame-options: SAMEORIGIN
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 4faa7af30aa1d37e-LAX

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>安全检查! | 百度云加速</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="yjs_styles-css" href="/cdn-cgi/styles/baidu.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='yjs_styles-ie-css' href="/cdn-cgi/styles/baidu.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/baidu.common.js"></script><!--<![endif]-->

<link rel="stylesheet"  href="//idm-su.baidu.com/config.css?r=20171010" type="text/css"  />

</head>
<body>
  <div id="yjs-wrapper">
    <div class="yjs-alert yjs-alert-error yjs-cookie-error" id="cookie-alert" data-translate="enable_cookies">请打开cookies。</div>
    <div id="yjs-error-details" class="yjs-error-details-wrapper">
      <div class="yjs-wrapper yjs-header yjs-error-overview">
        <h1 data-translate="challenge_headline">只差一步</h1>
        <h2 class="yjs-subheadline"><span data-translate="complete_sec_check">输入验证码，可以浏览</span> api.trakt.tv</h2>
      </div><!-- /.header -->

      <div class="yjs-section yjs-highlight yjs-captcha-container">
        <div class="yjs-wrapper">
          <div class="yjs-columns two">
            <div class="yjs-column">
              <div class="yjs-highlight-inverse yjs-form-stacked">
                <form class="challenge-form" id="challenge-form" action="/cdn-cgi/l/chk_captcha" method="get">
  <input type="hidden" name="s" value="02319d6c6a8a320b8cfd35d1ce1d973c3505e8f9-1563851625-1800-ARrYG7mKJGLWIknUj8IT3ni6Y6W6Tn6YfHvDRgetnUwTNLBGFq2Hxk6LH6bttaE5W+nG1zwdMyko/Yvk2Pvc8BFotlDJYnQpJSgmRE6cmNgzgRGnYPEq28eRiF80GhOMYA2V/a5iXUv16truCB4FSus="></input>
  <script type="text/javascript" src="/cdn-cgi/scripts/baidu.challenge.js" data-type="normal"  data-ray="4faa7af30aa1d37e" async></script>
  <noscript id="yjs-captcha-bookmark" class="yjs-captcha-info">
    <p>访问本页面,您的浏览器需要支持JavaScript</p>
  </noscript>
</form>

              </div>
            </div>

            <div class="yjs-column">
              <div class="yjs-screenshot-container">

                <span class="yjs-no-screenshot"></span>

              </div>
            </div>
          </div><!-- /.columns -->
        </div>
      </div><!-- /.captcha-container -->

      <div class="yjs-section yjs-wrapper">
        <div class="yjs-columns two">
          <div class="yjs-column">
            <h2 data-translate="why_captcha_headline">为什么需要输入验证码?</h2>

            <p data-translate="why_captcha_detail">输入验证码证明您不是机器人，输入后可以暂时浏览网站。</p>
</div>

          <div class="yjs-column">
            <h2 data-translate="resolve_captcha_headline">如何避免？</h2>

            <p data-translate="resolve_captcha_antivirus">如果您使用私人电脑，可以下载杀毒软件，进行全盘扫描保证没有中毒。</p>

            <p data-translate="resolve_captcha_network">如果您使用公用电脑，可以请网络管理员修正配置选项或查找病毒来源。</p>
          </div>
        </div>
      </div><!-- /.section -->

      <div class="yjs-error-footer yjs-wrapper">
  <p>
    <span class="yjs-footer-item">云加速 Event ID: <strong>4faa7af30aa1d37e</strong></span>
    <span class="yjs-footer-separator">&bull;</span>
    <span class="yjs-footer-item"><span>你的IP</span>: 14.155.221.197</span>
    <span class="yjs-footer-separator">&bull;</span>
    <span class="yjs-footer-item"><a href="https://su.baidu.com/helps/index.html#/0/page/1" id="support_link" target="_blank">帮助中心</a></span>
    <span class="yjs-footer-separator">&bull;</span>
    <span class="yjs-footer-item"><a href="http://su.baidu.com" id="yunjiasu_link" target="_blank">百度云加速</a></span>
  </p>
</div><!-- /.error-footer -->

    </div><!-- /#yjs-error-details -->
  </div><!-- /#yjs-wrapper -->

  <script type="text/javascript">
  window._yjs_translation = {};

</script>

</body>
</html>

rectifyer commented 5 years ago

Ok, so yeah for a captcha challenge is being presented from Cloudflare due to a potential threat. I'll have to research if its possible for Cloudflare to handle it differently if the request is application/json.

rectifyer commented 5 years ago

Following up on this, there doesn't seem to be a way to separate the challenge captcha and do something differently for the API. This seems like a feature request for Cloudflare, but I'm not really sure what can be done differently. There isn't a way for us to ask the user for additional info via the API.

ShaneQi commented 5 years ago

@rectifyer Thanks for the update.

IMO, this is a flaw on the Cloudflare/Baidu side that API requests are challenged with captcha but there isn't an API to get the captcha for users and send users' captcha input back. I agree that nothing can be done by us differently.

Did you explained the exact situation, in which our affected users are, to Cloudflare to explicitly request a feature/fix? I'm asking because:

Our affected users are in a special situation (challenges requests are API requests and this only happens for users in areas that Cloudflare partners with Baidu), it's easy for them to get our problem wrong.
I searched this problem a little bit and many questioners asked similar question but walked away by changing firewall settings (we don't want to do that to compromise our protection). We need to tell Cloudflare that it's their system flaw, not our configuration problem.
You are at a better place to request Cloudflare for a feature/fix because you are the direct customer of Cloudflare (correct me if I'm wrong).
If we could have a Cloudflare issue/ticket, it's better to track the progress.

ShaneQi commented 5 years ago

@rectifyer

Recently I got multiple negative feedbacks on my TV show tracker app from Chinese users because of the captcha issue. They don't understand the issue under the hood, they only know my app isn't usable anymore. I feel sorry for them.

I'm considering building a server (located in US) to setup a reserve proxy for Chinese users. My server will call Trakt api in the behave of my users to retrieve the resources they need.

I wanted to have your input on this before starting doing it: is there any reason that you do not recommend me to use a reserve proxy server to solve the particular issue our Chinese users are experiencing?

rectifyer commented 5 years ago

That is actually a pretty cool idea and way to handle it. Are you just planning to spin up a cloud server (AWS, Google, etc) and run API calls through there if you detect the person is located in China? I guess the only question would be with authenticated API methods, but I don't see any issues with doing this.

ShaneQi commented 5 years ago

@rectifyer

Yes what you said is exactly what I've been thinking.

There might be issues like you mentioned authenticated endpoints, but since it's just a simple proxy, shouldn't take much time, I'll just go ahead and try it.

Thanks for you opinion, I'll give updates here.

Tback1 commented 4 years ago

The question is when Cloudflare&Baidu cooperate the api was blocked by them and the user mostly didn't understand why Trakt will use the baidu , further they worried about it(they worried trakt will share the data with baidu) on the private and morality, the Baidu is notoriously on both things.

rectifyer commented 4 years ago

I was able to make some adjustments that should help with accessing the API in China, hopefully that helps this issue. Closing for now since I believe this will solve it.

trakt / api-help

403 Errors on Chinese Internet #92