search

jbms / finance-dl

Tools for automatically downloading/scraping personal financial data.
GNU General Public License v2.0
288 stars 37 forks source link

CHROMEDRIVER_CHROME_BINARY TypeError: Binary Location Must be a String #93

Open KamarajuKusumanchi opened 9 months ago

KamarajuKusumanchi commented 9 months ago

I am getting

TypeError: Binary Location Must be a String

when trying to run finance_dl.cli with schwab configuration. It is looking for os.getenv("CHROMEDRIVER_CHROME_BINARY") but could not find it. Could you please tell me how to fix the issue?

Steps to reproduce: Create an environment file

% cat env_test_finance-dl.yml
name: test_finance-dl
channels:
  - defaults
dependencies:
  - python=3.12
  - pip
  - pip:
    - git+https://github.com/jbms/finance-dl

Create the environment

% conda env create -f env_test_finance-dl.yml

Activate the environment

% conda activate test_finance-dl

Create a configuration file

% cat finance_dl_config.py
import os
profile_dir = os.path.join(os.getenv('HOME'), '.cache', 'finance_dl')
data_dir = '/home/rajulocal/x/x5'
def CONFIG_schwab():
    return dict(
        module='finance_dl.schwab',
        credentials={
            'username': 'XXXXXX',
            'password': 'XXXXXX',
        },
        output_directory=os.path.join(data_dir, 'schwab'),
        profile_dir=profile_dir,
        headless=False,
        min_start_date='2023-11-01'
    )

Run

% python -m finance_dl.cli --config-module finance_dl_config --config schwab
Traceback (most recent call last):
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 411, in retry
    return func()
           ^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 430, in fetch
    with temp_scraper(scraper_class, **kwargs) as scraper:
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 393, in temp_scraper
    scraper = scraper_type(*args, download_dir=download_dir,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/schwab.py", line 87, in __init__
    super().__init__(use_seleniumrequests=True, **kwargs)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 146, in __init__
    chrome_options.binary_location = os.getenv("CHROMEDRIVER_CHROME_BINARY")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/selenium/webdriver/chromium/options.py", line 52, in binary_location
    raise TypeError(self.BINARY_LOCATION_ERROR)
TypeError: Binary Location Must be a String
Waiting 0 seconds before retrying
Traceback (most recent call last):
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 411, in retry
    return func()
           ^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 430, in fetch
    with temp_scraper(scraper_class, **kwargs) as scraper:
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 393, in temp_scraper
    scraper = scraper_type(*args, download_dir=download_dir,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/schwab.py", line 87, in __init__
    super().__init__(use_seleniumrequests=True, **kwargs)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 146, in __init__
    chrome_options.binary_location = os.getenv("CHROMEDRIVER_CHROME_BINARY")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/selenium/webdriver/chromium/options.py", line 52, in binary_location
    raise TypeError(self.BINARY_LOCATION_ERROR)
TypeError: Binary Location Must be a String
Waiting 0 seconds before retrying
Traceback (most recent call last):
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 411, in retry
    return func()
           ^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 430, in fetch
    with temp_scraper(scraper_class, **kwargs) as scraper:
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 393, in temp_scraper
    scraper = scraper_type(*args, download_dir=download_dir,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/schwab.py", line 87, in __init__
    super().__init__(use_seleniumrequests=True, **kwargs)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 146, in __init__
    chrome_options.binary_location = os.getenv("CHROMEDRIVER_CHROME_BINARY")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/selenium/webdriver/chromium/options.py", line 52, in binary_location
    raise TypeError(self.BINARY_LOCATION_ERROR)
TypeError: Binary Location Must be a String
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/cli.py", line 94, in <module>
    main()
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/cli.py", line 90, in main
    module.run(**spec)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/schwab.py", line 374, in run
    scrape_lib.run_with_scraper(SchwabScraper, **kwargs)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 433, in run_with_scraper
    retry(fetch)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 411, in retry
    return func()
           ^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 430, in fetch
    with temp_scraper(scraper_class, **kwargs) as scraper:
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 393, in temp_scraper
    scraper = scraper_type(*args, download_dir=download_dir,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/schwab.py", line 87, in __init__
    super().__init__(use_seleniumrequests=True, **kwargs)
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/finance_dl/scrape_lib.py", line 146, in __init__
    chrome_options.binary_location = os.getenv("CHROMEDRIVER_CHROME_BINARY")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/selenium/webdriver/chromium/options.py", line 52, in binary_location
    raise TypeError(self.BINARY_LOCATION_ERROR)
TypeError: Binary Location Must be a String

There is no CHROMEDRIVER_CHROME_BINARY environment variable set on my machine.

% ipython
Python 3.12.0 | packaged by Anaconda, Inc. | (main, Oct  2 2023, 17:29:18) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: 
import os

In [2]: 
os.getenv("CHROMEDRIVER_CHROME_BINARY") is None
Out[2]: 
True

FWIW chromedriver-path shows

% chromedriver-path 
/opt/rajulocal/miniconda3/envs/test_finance-dl/lib/python3.12/site-packages/chromedriver_binary
paulirish commented 9 months ago

I got passed this with pip install -Iv selenium==4.2.0

KamarajuKusumanchi commented 9 months ago

Thanks @paulirish . I am able to get past this error with selenium 4.2.0 . But it looks like the schwab scraper is broken. It opens the chrome window, enters the password but it is not able to login. It does not work even if I manually enter the password. Since it is a different issue, I will open another issue for it.

The original environment file

 % cat env_test_finance-dl.yml
name: test_finance-dl
channels:
  - defaults
dependencies:
  - python=3.12
  - pip
  - pip:
    - git+https://github.com/jbms/finance-dl

is pulling

 % conda list "^python$|selenium|chromedriver-binary" -n test_finance-dl   
# packages in environment at /opt/rajulocal/miniconda3/envs/test_finance-dl:
#
# Name                    Version                   Build  Channel
chromedriver-binary       122.0.6211.0.0           pypi_0    pypi
python                    3.12.0               h996f2a0_0  
selenium                  4.16.0                   pypi_0    pypi
selenium-requests         2.0.3                    pypi_0    pypi

The error does not show up with

 % cat env_test_finance-dl_v2.yml
name: test_finance-dl_v2
channels:
  - defaults
dependencies:
  - python=3.12
  - pip
  - pip:
    - selenium==4.2.0
    - chromedriver-binary==120.*
    - git+https://github.com/jbms/finance-dl

which is pulling

 % conda list "^python$|selenium|chromedriver-binary" -n test_finance-dl_v2
# packages in environment at /opt/rajulocal/miniconda3/envs/test_finance-dl_v2:
#
# Name                    Version                   Build  Channel
chromedriver-binary       120.0.6099.109.0          pypi_0    pypi
python                    3.12.0               h996f2a0_0  
selenium                  4.2.0                    pypi_0    pypi
selenium-requests         1.4.1                    pypi_0    pypi

I am specifying the chromedriver-binary version in the environment file since it has to match with my chrome's version.

 % dpkg -l google-chrome-stable                                            
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version          Architecture Description
+++-====================-================-============-=================================
ii  google-chrome-stable 120.0.6099.199-1 amd64        The web browser from Google

The latest version of selenium is 4.16.0 (released on 2023-12-06 as per https://pypi.org/project/selenium/#history). It would be nice if finance-dl can support the latest version. If that is not possible, the correct selenium version should be specified in the dependencies.

moritzj29 commented 4 months ago

this is probably related to the fact that previously an empty CHROMEDRIVER_CHROME_BINARY was allowed but this is no longer the case (https://github.com/SeleniumHQ/selenium/pull/12328)

in my case I use a remote driver anyway (selenium docker container) so I decided to skip the setting if a remote is specified: https://github.com/moritzj29/finance-dl/commit/ae145aa9fb6708609549cf8f1eda8c025a07f9ed

but since people here do not mention remote drivers, I guess there needs to be some further tweaking... maybe just skip the setting if the env does not exist?