The download is stoping

Mxgnit0 commented 2 years ago

Appreciation

First , i want to say great work you have done here , i understand little bit of python and i can tell how much effort you put in it , so thank you i really appreciate what you did , because i spent a lot of time searching about new scripts that works nicely , and i was so happy when is found this , and "2 days ago" sign made my day .

Problem

note : I'm using no watermark option . so the script works fine but it stopped at certain amount of videos like 33 videos or less and i get this error message ;

File "/home/user/Documents/tiktok-bulk-downloader/tiktok_dl.py", line 387, in <module>
    main(args)
  File "/home/user/Documents/tiktok-bulk-downloader/tiktok_dl.py", line 381, in main
    download_posts(json_data["posts"], args.no_watermark)
  File "/home/user/Documents/tiktok-bulk-downloader/tiktok_dl.py", line 355, in download_posts
    nw_url(driver, base_url, post["id"]) if no_watermark else post["video"],
  File "/home/user/Documents/tiktok-bulk-downloader/tiktok_dl.py", line 316, in nw_url
    downloads = WebDriverWait(driver, 10).until(
  File "/home/user/.local/lib/python3.9/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Screenshot: available via screen

So i did little research and i found that maybe its the loading time of page which seems logic since it happens using regular browser , so i increased the waiting time 30 sec then 60 , then 100 WebDriverWait(driver, 100) and same error appears again but after downloading like 74 video out of 290 or 408 .

is my connection slow ? so that i get this problem ( the download is fine like : 1.2 / 1.5 MB/sec ) .
is it worked for you like 100% download out of like 290 videos ? if it so how much connection speed you test it out ?
is there a js anti bot script that blocking the process on the tiktok ?

Proposal

since it stopped at 74 videos can you make the script more dynamic like i can run it again and start from video number 76 , like which line of code the loop is for edit the i is ( which you don't use i think ) . i see that maybe a temporary solution , but it can be the same error since the phantomjs is gonna scroll the hole page again . So i hope the solution is simple , thank you .

0aub commented 2 years ago

Good morning Mxgnit0, I’m glad for what you said in “appreciation” section. It made my day.

About the problem, i extract all the videos ids from an apis that I fetched manually from tiktok website. After that, i pass each id to external website that have hidden api that I couldn’t extract, that is why i used phantomjs. Phantomjs will be used to pass the video url to the website and wait until the video fully loaded in the page. Here when the error occurred. I don’t believe the problem from you. I will test it on huge sample to find out why that happened. I have many thoughts about it but i have to test it over and over again to be sure.

your proposal is amazing.. to be honest, I didn’t thought of it and I’m now exited to add it to the code.

I will let you know the I finish all of that. Thank you for your wonderful comments.

Mxgnit02 commented 2 years ago

Hello , this is my github new account , let me know when you finish it all .

After you explained the algorithm , i'm convinced that your work is amazing . please do your best can't wait for the good news .

By the way i'm returning to python can you suggest me good tutorials , courses , blogs or forums that make me master it like you did . i'm open to any opportunity .

0aub commented 2 years ago

Good morning, dear Mxgnit

updates

well, I enhanced the code and handled all its exceptions (I hope I handled them all). I don't want to bother you with the details even if I dying to. I'm testing it for the last time on account with 965 video without any interrupt of suddenly stops. And I'm sorry if I took too long time to finish. I will upload it directly after finishing the last test.

learning

To be honest, I'm really bad with courses because I get board of watching. But courses and tutorials are very important just in the beginning. After you learn the basics and the logic, I advise you to put yourself on a project that you think it is bigger than you and start your real learning path from the project. Boosting your desire, I know I may seem silly by saying that but emotions have a huge part on the desire. It is okay to code just to impress or stalk someone or maybe to get revenge. I don't say that it is okay to hack or code a malware, I just want you to understand how to boost your own desire. Example: what you said up there is what made me put a log of effort in this code. It boosted my desire to do a great job.

courses

I recommend these two courses from udemy to learn python for different uses. You can start any interesting idea you have directly after you know python packages and how to use them. Remember, these courses just to prepare you to start coding any idea you think about, It doesn't mean that it will teach you everything, you always have you experience new mistakes and errors, and practice handling them.

100 Days of Code: The Complete Python Pro Bootcamp for 2022 you can find this course for free here

2022 Complete Python Bootcamp From Zero to Hero in Python you can find this course for free here

I'm here if you needed anything

0aub / tiktok-bulk-downloader