search

shaikhsajid1111 / twitter-scraper-selenium

Python's package to scrap Twitter's front-end easily
https://pypi.org/project/twitter-scraper-selenium
MIT License
305 stars 47 forks source link

can't set firefox profile path #34

Closed rachmadaniHaryono closed 1 year ago

rachmadaniHaryono commented 1 year ago 
from twitter_scraper_selenium.topic import scrap_topic
scrap_topic(
    filename='linux',
    url='https://twitter.com/i/topics/848959431836487680',
    headless=False,
    browser_profile='/home/r3r/Documents/selenium_profile'
)

output

> python steamdeck.py
INFO:root:Loading Profile from /home/r3r/Documents/selenium_profile
[WDM] - Driver [/home/r3r/.wdm/drivers/geckodriver/linux64/v0.31.0/geckodriver] found in cache
INFO:WDM:Driver [/home/r3r/.wdm/drivers/geckodriver/linux64/v0.31.0/geckodriver] found in cache
INFO:seleniumwire.storage:Using default request storage
INFO:seleniumwire.backend:Created proxy listening on 127.0.0.1:44371

this will freeze after those text output

looking at source code for selenium setting profile is deprecated https://github.com/SeleniumHQ/selenium/blob/a4995e2c096239b42c373f26498a6c9bb4f2b3e7/py/selenium/webdriver/firefox/options.py#L101-L105

i have better success by using profile argument just like chrome https://github.com/rachmadaniHaryono/twitter-scraper-selenium/tree/bugfix/profile

but i only test it with linux

shaikhsajid1111 commented 1 year ago

I've tested your code on Windows Machine. It is working, you can create PR. Thanks!