drawrowfly / tiktok-scraper

TikTok Scraper. Download video posts, collect user/trend/hashtag/music feed metadata, sign URL and etc.
4.39k stars 799 forks source link

Watermark-less and HD downloading broken #266

Closed Fukitsu closed 4 years ago

Fukitsu commented 4 years ago

Describe the bug A clear and concise description of what the bug is. Downloading videos without the watermark and in HD doesn't work anymore, but I've noticed that it only seems to happen with videos that were uploaded today 2020-July-27.

If fields below will be empty then issue will be Ignored and Closed

To Reproduce Steps to reproduce the behavior:

drawrowfly commented 4 years ago

Well i have checked videos that were uploaded recently and video id value is missing, and that id is required to download video without the watermark

drawrowfly commented 4 years ago

Old videos "dataisomiso2avc1mp41<4datavid:v09044b00000bs7o7tgp8rs6d937drk0%dataLavf58.20.10free" New videos "dataisomiso2avc1mp41%dataLavf58.20.10free"

areltiyan commented 4 years ago

v09044b60000bsfiu7frscu75cjatncg

id for that video just checked by sniff TikTok App but idk how to get new id with old method

kamronbek29 commented 4 years ago

Fukitsu I have the same problem. Can not download videos without watermark, that were uploaded recently

drawrowfly commented 4 years ago

"Solution" that you have used requires implementing signature method, so it is not a solution X-Gorgon: X-Khronos:

areltiyan commented 4 years ago

"Solution" that you have used requires implementing signature method, so it is not a solution X-Gorgon: X-Khronos:

agree

kamronbek29 commented 4 years ago

@drawrowfly Yes, but there is no script to generate these signatures (X-Gorgon and X-Khronos)

drawrowfly commented 4 years ago

@kamronbek29 Off-course there won't be any public "script" that's why i have said "so it is not a solution"

Also there are ways to generate signature, the easiest way is through Frida, but this will require bunch of work

kamronbek29 commented 4 years ago

@drawrowfly Yes, agree with u. Waiting for updates. Thanks

harimambura commented 4 years ago

@areltiyan

v09044b60000bsfiu7frscu75cjatncg

id for that video just checked by sniff TikTok App but idk how to get new id with old method

But how you got id for new video?

vnnegt commented 4 years ago

i've noticed that it appears to be only videos that have the "Save" option disabled that exhibit this behavior

can anyone else confirm?

Fukitsu commented 4 years ago

@vnnegt Can confirm though I got "Error: options.uri is a required argument" with one of the three I tried with but it worked with the other two

lucasintel commented 4 years ago

i've noticed that it appears to be only videos that have the "Save" option disabled that exhibit this behavior

can anyone else confirm?

It looks like video id is missing from all new uploads regardless of privacy settings.

dliakin commented 4 years ago

Hello everyone! Have the same problem. Sorry for question, but I don't understand is it possible to make scrapper download nowatermark new videos? And will be scrapper updated soon?

drawrowfly commented 4 years ago

I was about to release new update with patching some active issues and now this ....

I will try to find a way or maybe someone else will be able

drdexter1989 commented 4 years ago

v09044b60000bsfiu7frscu75cjatncg

id for that video just checked by sniff TikTok App but idk how to get new id with old method

Same question here . How did you get that id?

areltiyan commented 4 years ago

v09044b60000bsfiu7frscu75cjatncg id for that video just checked by sniff TikTok App but idk how to get new id with old method

Same question here . How did you get that id?

just sniff with app tiktok, like a fiddler btw to get this vidid need to x-gorgon and X-Khronos (timestamp)

harimambura commented 4 years ago

I found that this https://api-m.tiktok.com/aweme/v1/aweme/detail/ api endpoint seems to be exist, but don't know how to get something with it..

kamronbek29 commented 4 years ago

Is there anyone who found a way to download tiktok videos without watermark?

drdexter1989 commented 4 years ago

Maybe I would help in that too?

areltiyan commented 4 years ago

wow thanks btw where you get that api? tiktok app?

drawrowfly commented 4 years ago

Everything is possible, but i don't see point in it , i knew about this way but it is unstable and requires extra hassle

drawrowfly commented 4 years ago

@drdexter1989 ok and ?)

Marky commented 4 years ago

what about using some API from https://rapidapi.com/search/tiktok as option?

ssovit commented 4 years ago

what about using some API from https://rapidapi.com/search/tiktok as option?

https://rapidapi.com/ssovit/api/tiktok-no-watermark1 may not always give result you want.. but you may find it useful

drawrowfly commented 4 years ago

This is already getting ridiculous please post here messages only about solutions related to the WebAPI!

  1. If you disclose something related to the mobile API in public, TikTok will take note of it, i already mentioned before few times that i spotted traffic from bytedance(tiktok owner) to this repo
  2. Decompiling an android app is an easy peasy task but getting something from complied and obfuscated C/C++ library that is responsible for generating different ID's/Tokens and etc is another thing
  3. Such things are never disclosed in public repo
drawrowfly commented 4 years ago

lol)

@Zipdox few decompilation tips for beginners

  1. Apktool won't always decompile the app(i'm not going in too why this is happening)
  2. You can always change file extension from apk to zip, and extract content
  3. There you will find lots of DEX files that can be "decompiled" with different tools, for example https://github.com/pxb1988/dex2jar
  4. After you can review each decompiled file with JD-GUI
liamengland1 commented 4 years ago

This is off-topic and I'm sure the repo owner doesn't appreciate it. Go coordinate this on your own repo

drawrowfly commented 4 years ago

sorry guys , but i don't appreciate off-topic . When i will have a working solution that does not require mobile api i will update the repo.