When I crawle diffrent users, I only get 13 photos in many users wherever my '-n' is set to 200 and these user in fact have more than 13 photos.
So I think loadmore.click() in line 168 may not work well. Dose anyone suffer the same problem
some info:
posts: 148, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/linyaudavy
Quitting driver...
headless mode on
dir_prefix: ./data/, query: hecs_510, crawl_type: photos, number: 200, caption: False, authentica
tion: auth.json
posts: 273, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/hecs_510
Quitting driver...
headless mode on
dir_prefix: ./data/, query: da1sun, crawl_type: photos, number: 200, caption: False, authenticati
on: auth.json
posts: 2140, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/da1sun
Quitting driver...
When I crawle diffrent users, I only get 13 photos in many users wherever my '-n' is set to 200 and these user in fact have more than 13 photos. So I think loadmore.click() in line 168 may not work well. Dose anyone suffer the same problem
some info: posts: 148, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/linyaudavy
Quitting driver...
headless mode on
dir_prefix: ./data/, query: hecs_510, crawl_type: photos, number: 200, caption: False, authentica tion: auth.json
posts: 273, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/hecs_510
Quitting driver...
headless mode on
dir_prefix: ./data/, query: da1sun, crawl_type: photos, number: 200, caption: False, authenticati on: auth.json
posts: 2140, number: 200
Scraping photo links...
Number of photo_links: 13
Saving...
Downloading 12 images to ta/da1sun
Quitting driver...