Search and data download not working

[danilogalisteu]

MVE:

import investpy as ip
res = ip.search_quotes('gold')

results in:

Traceback (most recent call last):

  File "<ipython-input-23-994b566a65ab>", line 1, in <module>
    res = ip.search_quotes('gold')

  File "D:\WPy64-3771\python-3.7.7.amd64\lib\site-packages\investpy\search.py", line 127, in search_quotes
    data = req.json()

  File "D:\WPy64-3771\python-3.7.7.amd64\lib\site-packages\requests\models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)

  File "D:\WPy64-3771\python-3.7.7.amd64\lib\site-packages\simplejson\__init__.py", line 525, in loads
    return _default_decoder.decode(s)

  File "D:\WPy64-3771\python-3.7.7.amd64\lib\site-packages\simplejson\decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)

  File "D:\WPy64-3771\python-3.7.7.amd64\lib\site-packages\simplejson\decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())

JSONDecodeError: Expecting value

I traced the problem (similar to this) to the fact that Investing.com is returning content compressed with Brotli, which is accepted according to the search headers (search.py line 107):

    head = {
        "User-Agent": random_user_agent(),
        "X-Requested-With": "XMLHttpRequest",
        "Accept": "text/html",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }

but can't be decompressed by the requests package (search.py line 127):

        data = req.json()

Additionally, the headers used to retrieve recent or historical data with a SearchObj also contain the br accepted encoding (search._obj.py lines 258 and 279), resulting on an error raised on search._obj.py line 155:

ERR#0004: data retrieval error while scraping.

So, the solution is to either add brotli as a package requirement and register the decoder with requests, or remove the br accepted encoding in search.py line 111, search._obj.py lines 258 and 279, and others as required.

[alvarobartt]

Thanks a lot to everyone, especially to @nhlsm and to @danilogalisteu, as the solution they proposed works like a charm. The tests are already running, but once I check everything is OK (besides the Investing.com issues that have nothing to do with investpy), I'll upload the new release mentioning everyone involved in solving it! :fire: Thanks for this awesome support and sorry for my delayed response.

[alvarobartt]

You can install the new-fixed version as it follows:

pip install investpy==1.0.5

Sorry for the inconvenience, seems that Investing.com is applying a lot of changes and under maintenance... Everyone thank both @danilogalisteu and @nhlsm for their awesome support! 👍🏻

[alvarobartt]

Feel free to drop some comments here https://github.com/alvarobartt/investpy/discussions/334 and more information if applicable, such as the IDs of the solved issues and so on, so that I can include all the information into the Release Notes! :hugs: Thank you!