search

druzsan / justetf-scraping

Scraping the justETF
MIT License
36 stars 13 forks source link

Issue with POST request #10

Closed Cashalote closed 6 months ago

Cashalote commented 6 months ago

POST request seems to have stopped working. I outputted the response to a file and did not look like a data table of correct json format. Wanted to know if something changed with API since I cannot find any news online.

druzsan commented 6 months ago

Hi @Cashalote,

you're right. Overview API doesn't work for me ATM. Chart API works as expected though. I'm starting an investigation...

pauleonline commented 6 months ago

Hidden-URL: https://www.justetf.com/servlet/etfs-table seems to have been moved... 404 response...

xaver291 commented 6 months ago

Have you had the chance to take a look at this and maybe found another way around that?

druzsan commented 6 months ago

Hi everyone,

the overview API was indeed changed. I can see another POST request instead but unfortunately I can't replicate it from Python as straight forward as it was before. I will post here more info soon.

Any help from your side is also highly appreciated!

druzsan commented 6 months ago

@Cashalote @pauleonline @xaver291 good news!

The current hidden URL for POST requests looks like https://www.justetf.com/en/search.html?2-1.0-container-tabsContentContainer-tabsContentRepeater-1-container-content-etfsTablePanel=&search=ETFS&_wicket=1 where 2 in 2-1.0 could be apparently replaced with any other number dynamically. So one has to first request the https://www.justetf.com/en/search.html page, then parse it for the currently allowed hidden URL and then make a POST request.

I could replicate it in Python, so I have to test it a little bit and then I will build it into the project.

druzsan commented 6 months ago

This issue should be fixed by now.

xaver291 commented 6 months ago

Thank you very much.

pauleonline commented 6 months ago

Hi Alex, thank you very much for the quick fix , ... you rock ! Christoph