search

sudoguy / tiktok_bot

New tiktok bot on https://github.com/sudoguy/tiktokpy
https://github.com/sudoguy/tiktokpy
MIT License
134 stars 39 forks source link

Downloading X number of videos based on a hashtag/trend #14

Open savvas17 opened 4 years ago

savvas17 commented 4 years ago

I have seen good work done on downloading user videos. What I am struggling to figure out is how to scrape videos based on a certain hashtag or trend. I am struggling to find a way to do this because the video links don't appear in the source of the page. These pages only respond to GET requests and not POST.

Thank you :)

socialmediaguy commented 4 years ago

@savvas17 Did you manage to figure this out?

savvas17 commented 4 years ago

@socialmediaguy I didn't manage to do it in python. But what I did manage to do is write some javascript which you inject into the browser and pulls video URLs from the page and scrolls down etc. (it is not that complicated). Those URLs then can be downloaded in a batch. Unfortunately it is a manual process for now as you have to physically navigate to the trend page in a browser. But it's something I guess.

socialmediaguy commented 4 years ago

@savvas17 Thanks for your reply! Does that pull them out based on most popular or certain number of views or likes? I.e the most likes videos or viewed videos...?

savvas17 commented 4 years ago

@socialmediaguy hmm as far as I can see TikTok seems to show you the trending ones first based on your location. Here is an example page https://www.tiktok.com/tag/catchthesplash The videos you see here are likely different to the ones I see which are local to my country. All the code does is go through these videos in a page sequentially and get the links for them (it just keeps scrolling down). Then those links can be downloaded. I'm sure there would be a nicer way to automate this. In fact, im sure that an extension can be written where you just click it on a page and it produces the video links.

socialmediaguy commented 4 years ago

@savvas17 I noticed that too, but it doesn't show them in order or likes or views. Maybe based on trending in x time. I want something that gives me even just links of videos in order of most viewed to least in a hashtag.

socialmediaguy commented 4 years ago

I can't seem to edit my last comment but @savvas17 I guess you are halfway there. Now if you can read the likes/views per video then just order the links in ascending order

savvas17 commented 4 years ago

@socialmediaguy It is definitely possible to order them. The issue of my implementation is that it requires a bit of manual work by visiting the page in the browser. But thereafter it should be simple. Depending on your use case (if I may ask?) this might not be appropriate to manually visit links like this. But it can work for some situations.

socialmediaguy commented 4 years ago

@savvas17 I'm curious to see what kind of videos in a particular hashtag get the most likes or views so for my use case i won't need to 100s, just the top ones and checking on adhoc basis so I don't mind if it is manual (of course automated is better) Speaking to a local friend who is pretty good with javascript to see if it can quickly knock something together, otherwise I will spend some time on it over the weekend.

savvas17 commented 4 years ago

@socialmediaguy Yeah your javascript friend will accomplish the task easily. My only concern is that we don't really know what order tiktok displays those videos... but maybe grabbing the first 100 or 1000 is a safe bet to find the highly liked ones