missing records from getting scraped when input is given for 1K + records.

[itpothitech]

Describe the bug Missing records from getting scraped when input is given for 1K + records. There are 20% - 30% of records not getting scraped and it's random in nature.

I want to scrape for some posts having 1M+ posts. e.g. : Song: https://www.tiktok.com/music/Dance-Monkey-6717552289336314629

If fields below will be empty then issue will be Ignored and Closed

To Reproduce Steps to reproduce the behavior:

• Used OS: Ubuntu Desktop (Version: 18)

• NodeJs Version: 12.16

• Scraper Version: 1.1.1

• CLI or Module: Module

• Used method: music and hashtag

• Used options: const TikTokScraper = require("tiktok-scraper"); TikTokScraper.music(musicId, { number: records }); TikTokScraper.hashtag(hashTag, { number: records }); number: 1000 (but had received 800 - 900 records) When increased to 10K, recived around 4 K records.

• If scraping from a profile. Did you checked if profile is Public? Scraper won't work with Private profiles! N/A

• Preferably provide URL to the profile,hashtag,video and etc page Song: https://www.tiktok.com/music/Dance-Monkey-6717552289336314629 Hashtag: https://www.tiktok.com/tag/houseoftiktok

Screenshots If applicable, add screenshots to help explain your problem. I have not captured one.

[drawrowfly]

Its mostly because of the errors from the TikTok side

Anyway i will take a look at it closely

[itpothitech]

Thanks for your response.

[itpothitech]

Sorry, I am adding one more requirement here. Please let me know if I need to create another issue.

I also need to know if I can get a link to the post. Now I am able to get the link for the video. But having the link to the post will be good. e.g. I got details about the below post. But not the link to it. I am looking for this: https://www.tiktok.com/@davidnelmes/video/6741778973954542853

Even if I will get the video id# 6741778973954542853, I can construct the link the post, I believe

[drawrowfly]

in latest version webVideoUrl will contain the original video url

[itpothitech]

Thanks a lot for the update. I will use it.

[itpothitech]

Sorry, I couldnt find the field "webVideoUrl" in version 1.1.5 e.g. Music ID# 6717552289336314629

[drawrowfly]

Oh right. Fixed it in the 1.1.6

[itpothitech]

Thanks a lot. I could see it now. !! :) Not sure if you got some time to check about the initial issue i.e. records missing while scraping more than 1K. I even tried using "asyncScraping" but no luck. Do you suggest anything?

[drawrowfly]

You can now set timeout to slow down requests and set proxies as array:

[
"user:password@127.0.0.1:8080",
"user:password@127.0.0.1:8081",
"127.0.0.1:8082",
"socks5://127.0.0.1:8083",
"socks4://127.0.0.1:8084"
]

This will allow you to download more videos

[itpothitech]

Thanks !! I will try that.