druzsan
commented
3 months ago Dear @andreaiacoponi, could you check this again? At my side, everything is working as expected. It could be a temporary issue at JustETF side.
Closed andreaiacoponi closed 1 month ago
druzsan
commented
3 months ago Dear @andreaiacoponi, could you check this again? At my side, everything is working as expected. It could be a temporary issue at JustETF side.
andreaiacoponi
commented
3 months ago Hey @druzsan thanks for getting back to me. Unfortunately I am still seeing the same error as before. There are any test or debug I can do to collect more information about the error? Thanks.
druzsan
commented
2 months ago Hey @andreaiacoponi, I updated the code so that in case of the former assertion, it prints the exact error code and saves the response itself to a local file. Could you update the package, run it again and post your output and error file here? I hope this can bring us forward to the solution.
BTW, you can also try to run df = justetf_scraping.load_chart("IE00B0M62Q58") just to check whether the whole domain doesn't work for you or only the overview endpoint.
andreaiacoponi
commented
2 months ago Hi, I just did as you requested.
I uninstalled and reinstalled the package, using the command:
pip install justetf-scraping[all]@git+https://github.com/druzsan/justetf-scraping.git
Strangely after the installation, a lot of new packages were also installed:
Successfully installed anyio-4.4.0 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 arrow-1.3.0 asttokens-2.4.1 async-lru-2.0.4 attrs-24.2.0 babel-2.16.0 bleach-6.1.0 comm-0.2.2 debugpy-1.8.5 executing-2.0.1 fastjsonschema-2.20.0 fqdn-1.5.1 httpcore-1.0.5 httpx-0.27.0 ipykernel-6.29.5 ipython-8.26.0 isoduration-20.11.0 jedi-0.19.1 json5-0.9.25 jsonpointer-3.0.0 jsonschema-4.23.0 jsonschema-specifications-2023.12.1 jupyter-client-8.6.2 jupyter-core-5.7.2 jupyter-events-0.10.0 jupyter-lsp-2.2.5 jupyter-server-2.14.2 jupyter-server-terminals-0.5.3 jupyterlab-4.2.4 jupyterlab-pygments-0.3.0 jupyterlab-server-2.27.3 justetf-scraping-0.1.0 matplotlib-inline-0.1.7 mistune-3.0.2 nbclient-0.10.0 nbconvert-7.16.4 nbformat-5.10.4 nest-asyncio-1.6.0 notebook-7.2.1 notebook-shim-0.2.4 overrides-7.7.0 pandocfilters-1.5.1 parso-0.8.4 platformdirs-4.2.2 prometheus-client-0.20.0 prompt-toolkit-3.0.47 pure-eval-0.2.3 python-json-logger-2.0.7 pyzmq-26.1.0 referencing-0.35.1 requests-2.32.3 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.20.0 seaborn-0.13.2 send2trash-1.8.3 stack-data-0.6.3 terminado-0.18.1 tomli-2.0.1 tornado-6.4.1 traitlets-5.14.3 types-python-dateutil-2.9.0.20240316 uri-template-1.3.0 wcwidth-0.2.13 webcolors-24.8.0 websocket-client-1.8.0
However, the overview is working with warnings, and the chart seems to be working without warnings. Here is the output:
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import justetf_scraping
>>> df = justetf_scraping.load_overview()
/home/mint21/.local/lib/python3.10/site-packages/justetf_scraping/overview.py:404: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
df[column] = df[column].replace({"Yes": True, "No": False}).astype("bool")
/home/mint21/.local/lib/python3.10/site-packages/justetf_scraping/overview.py:404: FutureWarning: Downcasting behavior in `replace` is deprecated and will be removed in a future version. To retain the old behavior, explicitly call `result.infer_objects(copy=False)`. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
df[column] = df[column].replace({"Yes": True, "No": False}).astype("bool")
>>> df
wkn ticker valor ... last_year_max_drawdown last_three_years_max_drawdown last_five_years_max_drawdown
isin ...
IE00B5BMR087 A0YEDG SXR8 10737041 ... -9.09 -17.11 -33.71
IE00B4L5Y983 A0RPWH EUNL 10608388 ... -8.75 -16.88 -33.91
IE00B3XXRP09 A1JX53 VUSA 18575508 ... -9.09 -17.28 -33.70
IE00B3YCGJ38 A1CYW7 P500 11358996 ... -9.09 -17.20 -33.69
IE00BKM4GZ66 A111X9 IS3N 24209517 ... -10.36 -22.30 -32.62
... ... ... ... ... ... ... ...
XS2427363036 A3GWVS 2FLY <NA> ... -37.34 NaN NaN
XS2427474023 A3GWVN O2IG <NA> ... -28.76 NaN NaN
CH1210548884 A3G04F USLG <NA> ... -70.98 NaN NaN
IE00B8GKPP93 A1VBKG 3DES <NA> ... -51.16 NaN NaN
IE00B8JF9153 A1VBKH PCFC <NA> ... -51.61 NaN NaN
[3311 rows x 42 columns]
>>> df = justetf_scraping.load_chart("IE00B0M62Q58")
>>> df
quote relative ... quote_with_reinvested_dividends relative_with_reinvested_dividends
date ...
2005-10-28 20.60 0.000000 ... 20.600000 0.000000
2005-10-29 20.60 0.000000 ... 20.600000 0.000000
2005-10-30 20.60 0.000000 ... 20.600000 0.000000
2005-10-31 20.99 1.893204 ... 20.990000 1.893204
2005-11-01 21.03 2.087379 ... 21.030000 2.087379
... ... ... ... ... ...
2024-08-05 64.90 215.048544 ... 85.543909 315.261692
2024-08-06 65.95 220.145631 ... 86.927901 321.980101
2024-08-07 65.74 219.126214 ... 86.651102 320.636419
2024-08-08 66.78 224.174757 ... 88.021914 327.290844
2024-08-09 67.26 226.504854 ... 88.654596 330.362117
[6861 rows x 9 columns]
>>>
Hi @andreaiacoponi, I'm glad it works for you now.
Regarding many installed packages, I assume you either got update in many packages, or you previously installed justeft-scraping without all extra (installs Jupyter Notebook and Seaborn with all their dependencies above).
Anyway, if the problem comes again, the debugging is already on, so it's going to be much easier to understand the problem.
Warnings are already addressed in #12, so I will post a solution there.
I am facing this error. It's is likely that something has changed on the other end, preventing scraping to execute.