0 posts in collector

[AyQWERTY]

Describe the bug In my project, I collect basic profile information (subscribers, likes) and in each video views, comments and shares. My project is used for myself, so the number of requests is small, about 15 profileInfo, and 15 user(about 35 posts on account) requests per hour.

After half a day or a day, the scraper begins to scraping 0 posts. I tried running the script on two VPS (different IPs), and both after that time stopped collecting posts(collector.length = 0). I tried putting a breakpoint in the place where tiktok-scraper receives response, to see what it receives and if it receives anything at all. And yes it does, but it's a response with no body, just head, scripts and stuff(If you need, I can send you response).

To Reproduce Just try to get user info throughout the day.

Info (VPS#1 | VPS#2)

• Used OS: Linux 4.4.180+ | Linux 4.9.0-16-amd64
• NodeJs Version: v14.16.0 | v14.17.5
• Scraper Version: 1.4.36 both
• CLI or Module: Module
• Used method: user Promise
• Used options: number: 1000, sessionList: [x1 sid_tt]
• Do you run the scraper through Proxy, VPN, VPS?: proxy/vpn - no, but I tried it through a proxy and it had no effect, still 0 posts
• If scraping from a profile. Did you checked if profile is Public?: yes, its public

[JeremiahMorenoArk]

I wish I had some popcorn when I read this thread. lmaooooo

[BK-Go-Python]

I used the login cookie to request information and got “{'status_code': 2484, 'status_msg': 'Too many attempts. Try again in 1 hour.', 'log_pb': {'impr_id': '202109180828510102451470991C014BD9'}}” 🤦‍♀️🤦‍♀️🤦‍♀️

[akaoppa]

Im having this issue "try to use proxy at TikTokScraper.getUserProfileInfo" once a while it's working do you think it's a proxy issue ?

[isaackogan]

Im having this issue "try to use proxy at TikTokScraper.getUserProfileInfo" once a while it's working do you think it's a proxy issue ?

Ready this thread & come back if you still have the same question. But given the entire thread is dedicated to the fact that this specific method isn't working... I'm going to say you won't.

[wiliam-toney]

Tiktok require msToken and X-Bogus query params while get user's posts.

Anybody resolved how do they generate these params?

[asportnoy]

We should pin this issue

[Xodarap]

https://github.com/davidteather/TikTok-Api/releases/tag/V4.0.2 also has been updated to create these parameters

[lamualfa]

few days ago, scraper work good ,but today it always fail, i think it made by tiktok upgrade,especially tiktok api domain change...just my guess

This open source project make the TikTok Engineer easier to find out how people scraping their content and they will update their system to prevent us from doing scraping.

[dencorg]

Hello, are there any updates on this issue? I don't want to rush anything, just checking on the progress. Thank you very much for your effort!

[vjleblanc]

Out of curiosity, would it make sense to create a signer function as an optional configuration parameter? The thinking being that it could optionally add signature headers given the raw request URL. I'm thinking about integrations w/ signing logic like this. That way each implementer could decide their comfort w/ adding headless browsers or not.

[zvictor]

I have been using the tiktok-signature in my fork for the last 9 days and that has solved this issue!

You can give it a try at https://github.com/zvictor/tiktok-scraper and let me know if any function fails for you.

[sampsn]

I have been using the tiktok-signature in my fork for the last 9 days and that has solved this issue!

You can give it a try at https://github.com/zvictor/tiktok-scraper and let me know if any function fails for you.

newbie here. How do i install your version?

[evgenycherkasov]

@zvictor Hello I am testing your fork, but getMusicFeed by music id method doesn't work now:( Its return "No more posts" exception message

[zvictor]

newbie here. How do i install your version?

To install locally:

git clone git@github.com:zvictor/tiktok-scraper.git
cd tiktok-scraper
npm i
npm run build

Then, for CLI access:

./bin/cli.js --help --verbose

For programmatic use, just move the new folder to where you used to have node_modules/tiktok-scraper and you are good to go.

[zvictor]

I am testing your fork, but getMusicFeed by music id method doesn't work now:( Its return "No more posts" exception message

Please run in verbose mode (adding --verbose or DEBUG=tiktok-scraper:*) and open an issue in my repo.

[sampsn]

newbie here. How do i install your version?

To install locally:

git clone git@github.com:zvictor/tiktok-scraper.git
cd tiktok-scraper
npm i
npm run build

Then, for CLI access:

./bin/cli.js --help

For programmatic use, just move the new folder to where you used to have node_modules/tiktok-scraper and you are good to go.

i know this is probably the last place to ask these questions, but other websites arent helping. When installing the original tiktok scraper, it runs but fails. When using yours, i get "zsh: permission denied: tiktok-scraper" when trying to run it. Is this a simple fix? am i dumb or am i doing something wrong?

[zvictor]

When using yours, i get "zsh: permission denied: tiktok-scraper" when trying to run it. Is this a simple fix? am i dumb or am i doing something wrong?

Could you send the cmd you run and the error message, in full?

[lamualfa]

I have been using the tiktok-signature in my fork for the last 9 days and that has solved this issue!

You can give it a try at https://github.com/zvictor/tiktok-scraper and let me know if any function fails for you.

Woah, it's awesome fork. Thanks @zvictor.

[vnnegt]

When using yours, i get "zsh: permission denied: tiktok-scraper" when trying to run it. Is this a simple fix? am i dumb or am i doing something wrong?

Could you send the cmd you run and the error message, in full?

hmm, i followed all the directions, but when tried to scrape a user i just get a perpetual TikTok Scraper Started message

Imgur

[roman-hrybinchuk]

@zvictor. I have cloned your branch but user and music do not return any posts for me. I am getting. Error: No more posts.

Doing like that

(async () => { try { const posts = await TikTokScraper.user('justinbieber', { number: 50 }); console.log(posts); } catch (error) { console.log(error); } })();

[sasoriza]

I have been using the tiktok-signature in my fork for the last 9 days and that has solved this issue!

You can give it a try at https://github.com/zvictor/tiktok-scraper and let me know if any function fails for you.

I am getting TikTok Scraper StartedNo more posts and then it's just stuck at TikTok Scraper Started without scraping anything. I tried both from-file and user mode

[roman-hrybinchuk]

Hey, @zvictor any comment about 2 last comments ?

[angelfire76]

So, It's discontinued? Is there any alternative around?

Edit: Found this.

[kepamuk]

problem solved? I can't figure out how to get its video from a specific channel,

try {
    const posts = await TikTokScraper.user('justinbieber', {
      number: 100
    });
    console.log(posts);
  } catch (error) {
    console.log(error);
  }

this code doesn't even work together with sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']

[UnixCro]

same

[UnixCro]

So, It's discontinued? Is there any alternative around?

Edit: Found this.

The best Tiktok downloader after tiktok-scraper is youtube-dl in my opinion, but it has the disadvantage that it is slower and is not able to download all videos at once. Unfortunately, tikup caused problems for me. Nevertheless, I hope that someone from the Github community will create a repository that has the ability to download the videos from Snaptik.com in order to both enable better video quality & finally be able to provide the videos without watermarks. That would help us temporarily until drawrowfly has fixed the problem.

Greeting UnixCro

[ColtraneNadler]

newbie here. How do i install your version?

To install locally:

git clone git@github.com:zvictor/tiktok-scraper.git
cd tiktok-scraper
npm i
npm run build

Then, for CLI access:

./bin/cli.js --help --verbose

For programmatic use, just move the new folder to where you used to have node_modules/tiktok-scraper and you are good to go.

I keep getting an error when trying to build

20 error code ELIFECYCLE
21 error errno 1
22 error tiktok-scraper@1.4.35 build: `rimraf build && tsc`
22 error Exit status 1
23 error Failed at the tiktok-scraper@1.4.35 build script.

any ideas?

[Sav0ia]

newbie here. How do i install your version?

To install locally:

git clone git@github.com:zvictor/tiktok-scraper.git
cd tiktok-scraper
npm i
npm run build

Then, for CLI access:

./bin/cli.js --help --verbose

For programmatic use, just move the new folder to where you used to have node_modules/tiktok-scraper and you are good to go.

Hi victor, I've installed your fork via YARN but get the below cli error. But the cli.js file is in that path. I'm on a mac.

code: 'MODULE_NOT_FOUND', requireStack: [ '/Users/ME/.config/yarn/global/node_modules/tiktok-scraper/bin/cli.js'

Thanks!

[Felixxx99xx]

Any updates on how to download videos currently? If someone found a way, please notify me. Thanks

[tuphamdev96]

Any update on it :( please notice me. Thanks so much

[kristianlaskarov]

Version 1.4.22 works with methods - .getVideoMeta() and .getUserProfileInfo(). Method .user() returns empty array. @drawrowfly , I hope the info will be useful for a little fix..

[ColtraneNadler]

does getUserProfileInfo return a list of media?

[kristianlaskarov]

No, only the author info.

[akaoppa]

No news about it ? it's been more than 6 months. It's surprising that nobody found a workaround for it yet

[vnnegt]

No news about it ? it's been more than 6 months. It's surprising that nobody found a workaround for it yet

workarounds have been found, but ppl are keeping it to themselves because the TikTok will immediately patch it

[BetterLifeR]

I've come from the closed issue #720. I haven't been able to read through the entirety of this issue since there are too many messages, but so far I haven't seen a common fix, and I'm not sure if what I'm about to mention has already been said. But I'm just going to drop in some info that I've found while trying to fix the issue myself.

after going through the src code in core/TikTok.ts I've traced the route of the main problem that everyone seems to be having (Can't extract user metadata from the html page. Make sure that user does exist and try to use proxy), to the async getUserProfileInfo method, specifically in the try-catch statement when splitting the const request object and assigning the value to const breakResponse.

It seems that the script tag (specifically the one with an id of __NEXT_DATA__) breakResponse is looking for no longer exists in the tiktok api? The only part of the response html that I found to contain any data relating to the user profile and video urls was in a script tag with an id of sigi-persisted-data when assigning window['SIGI_STATE'] a dictionary of meta data. 2 of the key values of interest to me were ItemList and ItemModule

TLDR in core/TikTok.ts the async method getUserProfileInfo tries to find a no longer existent script tag from the request's html, which seems to be the route of all our problems

[vanities]

@BetterLifeR this PR https://github.com/drawrowfly/tiktok-scraper/pull/721 seems to address your concerns? Can we merge that?

[BetterLifeR]

PR #721 seems to be a good fork to merge. I'm also working on a fork now which adds those implementations to the getUserProfileInfo method and reformats the response JSON to match the structure of the UserMetadata interface. This may take a while though as I've never worked in TypeScript before.

[akaoppa]

Thanks! if anyone have an workaround for : getUserProfileInfo it would be nice

[beshoo]

I have a problem with get user profile info, it returns rounded number for following users, and likes if the number is greater than 15000.

Is there any way around this?

[BetterLifeR]

Just as an update, I've got a fix for the 'SIGI_STATE' problem and I'm no longer experiencing a Can't extract user metadata from the html page. Make sure that user does exist and try to use proxy error. However it now seems that I'm having problems with the request-promise package (checking the package with npm shows that it's now deprecated), and I'm constantly getting ESOCKETTIMEDOUT

[phithu]

1+

[bleshik]

same, getting ESOCKETTIMEDOUT

[bleshik]

It seems like they've changed cookies format. I guess nothing works now...

[github-account1111]

@UnixCro

not able to download all videos at once.

What do you mean by this? This can download entire channels concurrently? How does it avoid getting rate limited or banned?

[akaoppa]

Anyone manage to make it works ? starting to have most of time : ESOCKETTIMEDOUT now It was working fine before, not sure if it's the proxies or not

[bleshik]

Yeah, it just doesn't work anymore, we had to switch to our own parser for out tasks.