TikTokScraper.user return always empty collector

[ambigos1]

Hi. Using latest version 1.4.22 using GeoSurf residential proxy.

and no matter how many attempts I am trying with different public profiles. I am always getting empty collector when invoking TikTokScraper.user(...) function.

Any help?

[Michaelmala]

Hi @ambigos1, I think you have to call the method like this:

TikTokScraper.user(username, { number: count, sessionList: sessionList, });

sessionList value can be got like described here: https://github.com/drawrowfly/tiktok-scraper/issues/437#issuecomment-817126996

[ambigos1]

I am working with this function for couple of month. I know how to use it. It worked for me until now.

Something need to be update here

[kwameHam]

@Michaelmala hey i am facing the same issue andi am using it with the sessionList (tried"sessionid_ss=...." and "sid_tt=...")

[boris-borisov]

Yes, we have the same problem as well. :-/

[Neacel]

I am having the same issue as well.

[asportnoy]

same issue here, @drawrowfly please take a look when you are able

[lu3do]

https://github.com/drawrowfly/tiktok-scraper/issues/520

[asportnoy]

@lu3do The user method doesn't throw when it fails. It just returns an empty collector. I've requested this to be changed on the discord.

[kgaspard]

+1

[lu3do]

@asportnoy yes, my report was mainly about the empty collector issue happening more frequently, no so much about the fact that it doesn't fail when it's empty.

[asportnoy]

@lu3do Sorry, I probably misunderstood your issue. I thought you were wondering why the collection was empty with no errors thrown.

[asportnoy]

PSA (30m ago)

[asportnoy]

Any updates on this @drawrowfly?

[DanielZhu03]

+1

[asportnoy]

The method just randomly started working again for me (One of my notifications went thru). Is it working again for anyone else?

Update: It's no longer working. Someone on the discord said it's been going off and on for them the whole day.

[fengyingdian]

tiktok has changed https://t.tiktok.com/api/post/item_list api

[fengyingdian]

they added "_signature" in params to enforce security

[ambigos1]

I saw that the _signatire field were removed lately. Did someone here checked if we will add the signature it will work again?

[Michaelmala]

Hi guys, I tried to call the api with the "_signature" value and it works, otherwise the response will ask for the captcha verification.

I noticed that this library already has the signer function but is not implemented to the api call.

Maybe I will do a pull requests to implement it.

[youngvo]

@Michaelmala could you please post here the code you did include the _signature to the API requests?

[Michaelmala]

@youngvo I used puppeteer to get the signature method from the browser window but is not ideal to do for many reasons, now I'm editing this library to implement the signature, I will submit a pull request when it will work as should.

[bluestar21]

I found that tiktok has updated his signature algorithm. The api can be accessed with the new signature, but the new signature algorithm has not been completely cracked. You can use puppeteer to build a signature server to temporarily solve it.

[fengyingdian]

I found that tiktok has updated his signature algorithm. The api can be accessed with the new signature, but the new signature algorithm has not been completely cracked. You can use puppeteer to build a signature server to temporarily solve it.

Some times we met the tiktok-verify-page, is there any good solutions?

[bluestar21]

I found that tiktok has updated his signature algorithm. The api can be accessed with the new signature, but the new signature algorithm has not been completely cracked. You can use puppeteer to build a signature server to temporarily solve it.

Some times we met the tiktok-verify-page, is there any good solutions?

If you use puppeteer, you can verify once after starting the browser once, and then keep the current page and browser, the signature function will be registered in the memory, and you can directly execute the js code to obtain the signature. ps:tiktok uses slider verification, you can use opencv to calculate the sliding distance, and then use puppeteer to control it for verification.

[fengyingdian]

I found that tiktok has updated his signature algorithm. The api can be accessed with the new signature, but the new signature algorithm has not been completely cracked. You can use puppeteer to build a signature server to temporarily solve it.

Some times we met the tiktok-verify-page, is there any good solutions?

If you use puppeteer, you can verify once after starting the browser once, and then keep the current page and browser, the signature function will be registered in the memory, and you can directly execute the js code to obtain the signature. ps:tiktok uses slider verification, you can use opencv to calculate the sliding distance, and then use puppeteer to control it for verification.

Yeah, exactly. I am doing this right now, and I use page.on('response') to fetch the item_list of video. I am trying make the system works like this: firstly, I open a brower and verify, then the task get into a loop, in each circle I fetch one tiktok creator's videos.

[sunnydev3301]

I found that tiktok has updated his signature algorithm. The api can be accessed with the new signature, but the new signature algorithm has not been completely cracked. You can use puppeteer to build a signature server to temporarily solve it.

so this is possible if I use modules and not CLI, right? @bluestar21

[asportnoy]

I was able to get it working with puppeteer. Here's what I did (copied from Discord):

1) Open the TikTok homepage with puppeteer 2) Generate the request URL without the signature (I used https://m.tiktok.com/api/post/item_list/?aid=1988&secUid=SEC_UID&count=VIDEO_COUNT&cursor=0&verifyFp=VERIFY_FP) 3) Use this code with puppeteer to generate the signature (page should be the page that has TikTok open):

async function sign(url) {
  const signature = await page.evaluate((url) => {
    return byted_acrawler.sign({
      url: url
    });
  }, url);

  // ...
}

4) Use the value in the variable as your _signature query string and make the request

Note that the output is formatted differently than in TikTokScraper. If you want to be able to download the videos, you need to supply a tt_webid_v2 cookie as shown in the readme

Hope this helps :)

[fredpe]

Also got this working using puppeteer, then using the byted_crawler.sign method. Anyone already found a way to not use puppeteer at all using a standalone function implementing the now updated signing logic? Tried it myself for 1-2h, but it's not an easy one. Basically requires writing a standalone function matching the logic in https://www.tiktok.com/acrawler/acrawler.js, without requiring a browser's window object.

[bluestar21]

so this is possible if I use modules and not CLI, right? @bluestar21

yes,I do like this ,you can follow @asportnoy's method.

[DevWedeloper]

Is this still not fixed yet? Also can someone link me the discord server.

[fengyingdian]

@asportnoy I generated _signature by using your method, but it doesn't work when I add "_signature=MY_SIGNATRUE" after this "https://m.tiktok.com/api/post/item_list/?aid=1988&secUid=SEC_UID&count=VIDEO_COUNT&cursor=0&verifyFp=VERIFY_FP", do you know what's the problem? Thank you so much!

[Michaelmala]

Hi guys, after some calls to the user posts api with puppeteer I get the response "illegal request", does this happen to anyone else?Does anyone know how to solve the problem? I was thinking about proxies but i want to avoid this solution.

[quangtuyennguyen]

Hi guys! Can I config Windscribe VPN like a gateway from my server to Tiktok resouce?

[ambigos1]

Well... Is there any estimation for new version that will solve this issue ?

[drawrowfly]

Well... Is there any estimation for new version that will solve this issue ?

Well....... Maybe

[ambigos1]

@drawrowfly Thanks you very much for all of your hard work. Very appreciated :)

[ThaDeveloper]

Has a fix for this been added yet?

[fengyingdian]

@asportnoy I generated _signature by using your method, but it doesn't work when I add "_signature=MY_SIGNATRUE" after this "https://m.tiktok.com/api/post/item_list/?aid=1988&secUid=SEC_UID&count=VIDEO_COUNT&cursor=0&verifyFp=VERIFY_FP", do you know what's the problem? Thank you so much! It worked! I forgot the tt_webid_v2 cookie in the previous version! Thank you!

[sunnydev3301]

COULD SOMEONE EXPLAIN THIS WAY STEP BY STEP BECAUSE I'M USED TO THE TERMINAL

I was able to get it working with puppeteer. Here's what I did (copied from Discord):

1. Open the TikTok homepage with puppeteer
2. Generate the request URL without the signature (I used https://m.tiktok.com/api/post/item_list/?aid=1988&secUid=SEC_UID&count=VIDEO_COUNT&cursor=0&verifyFp=VERIFY_FP)
3. Use this code with puppeteer to generate the signature (page should be the page that has TikTok open):

async function sign(url) {
  const signature = await page.evaluate((url) => {
    return byted_acrawler.sign({
      url: url
    });
  }, url);

  // ...
}

1. Use the value in the variable as your _signature query string and make the request

Note that the output is formatted differently than in TikTokScraper. If you want to be able to download the videos, you need to supply a tt_webid_v2 cookie as shown in the readme

Hope this helps :)

[asportnoy]

COULD SOMEONE EXPLAIN THIS WAY

STEP BY STEP

BECAUSE I'M USED TO THE TERMINAL

I was able to get it working with puppeteer. Here's what I did (copied from Discord):

1. Open the TikTok homepage with puppeteer

2. Generate the request URL without the signature (I used https://m.tiktok.com/api/post/item_list/?aid=1988&secUid=SEC_UID&count=VIDEO_COUNT&cursor=0&verifyFp=VERIFY_FP)

3. Use this code with puppeteer to generate the signature (page should be the page that has TikTok open):

async function sign(url) {

  const signature = await page.evaluate((url) => {

    return byted_acrawler.sign({

      url: url

    });

  }, url);

  // ...

}

1. Use the value in the variable as your _signature query string and make the request

Note that the output is formatted differently than in TikTokScraper. If you want to be able to download the videos, you need to supply a tt_webid_v2 cookie as shown in the readme

Hope this helps :)

That's literally step by step right there.

[VasylKhoroshcho]

Can anyone confirm that approach with puppeteer still works? I'm getting a _signature, make the request and getting an empty response with status 200

[asportnoy]

Can anyone confirm that approach with puppeteer still works? I'm getting a _signature, make the request and getting an empty response with status 200

I'm getting the same thing. It's possible TikTok changed the query string requirements or something of the sort.

[nbertagnolli]

@VasylKhoroshcho The puppeteer method does not seem to be working anymore :(. It looks like they changed something in the last day or two.

[johnjilayi]

Has anyone else figured out how to scrap videos with hashtags using any other method or API's

[0xE232FE]

Does someone even try to use Chrome DevTools Protocol instead of using Puperteers or Headless Browsers?

[johndoughmaker22]

I haven't been using the module but I have been using the CLI and the functionality to scrape a public user's profile is broken as well. I suspect it's related. Anyone figure out a workaround yet?

[reinstallgentoo]

I don't quite know how to say this, but I think we may be in need of a cheeky update.

[donmark8]

I did "npm i -g tiktok-scraper" just now and still saw the same problem, got nothing for "tiktok-scraper user [id]", anyone else is seeing this problem or I miss something?

[asportnoy]

I did "npm i -g tiktok-scraper" just now and still saw the same problem, got nothing for "tiktok-scraper user [id]", anyone else is seeing this problem or I miss something?

The package hasn't been updated on NPM yet. @drawrowfly will likely tell us when it is fixed (and close this issue), in addition to pinging on the Discord server.

[asportnoy]

If you want to try out the new changes, you can install it manually thru GitHub with these steps. Keep in mind that there's no guarantee of this working and it might be a hassle. So you might just want to wait until it's officially released:

1. If you haven't already, install yarn

npm i -g yarn

2. Install the package from GitHub

# Node Module
yarn add asportnoy/tiktok-scraper

# CLI
yarn global add asportnoy/tiktok-scraper

I made additional changes on my fork that are not on this repository yet. If you want to use this repository, use drawrowfly/tiktok-scraper instead of asportnoy/tiktok-scraper.

3. Navigate to the package folder

# Node Module
cd ./node_modules/tiktok-scraper

# CLI (macOS / linux non-root)
cd ~/.config/yarn/global/tiktok-scraper

# CLI (linux root)
cd /usr/local/share/.config/yarn/global/tiktok-scraper

# CLI (Windows)
cd %LOCALAPPDATA%\Yarn\config\global/tiktok-scraper

Note that location and command may vary depending on your OS and setup.

4. Install and build

yarn install
yarn build

---

Again, this is probably way more trouble than it's worth and I highly recommend just waiting. Keep in mind that if you do use this, you will have to reinstall from NPM later in order to get updates.