GB Scrape Issue

[WhyAydan]

Seems that EPEX Web Scrape isn't working again with the following error code

403, message='Forbidden', url=URL('https://www.epexspot.com/en/market-data?market_area=GB&trading_date=2024-06-28&delivery_date=2024-06-29&auction=30-call-GB&modality=Auction&sub_modality=DayAhead&product=30&data_mode=table&ajax_form=1')

[janhenk102]

I am getting the same error (403 forbidden) for NL market area.

[TomF79]

Same for Austria

[lieven121]

Same for BE

When going to the url i get this

https://www.epexspot.com/en/market-data?market_area=BE&trading_date=2024-06-28&delivery_date=2024-06-29&auction=MRC&modality=Auction&sub_modality=DayAhead&product=60&data_mode=table&ajax_form=1

[{"command":"insert","method":"prepend","selector":null,"data":"\u003Cdiv data-drupal-messages\u003E\n  \u003Cdiv class=\u0022messages__wrapper\u0022\u003E\n              \u003Cdiv class=\u0022alert alert-danger alert-dismissible\u0022 role=\u0022alert\u0022 aria-label=\u0022Error message\u0022\u003E\n      \u003Cbutton type=\u0022button\u0022 role=\u0022button\u0022 class=\u0022close\u0022 data-dismiss=\u0022alert\u0022 aria-label=\u0022Close\u0022\u003E\u003Cspan aria-hidden=\u0022true\u0022\u003E\u0026times;\u003C\/span\u003E\u003C\/button\u003E\n              \u003Ch2 class=\u0022sr-only\u0022\u003EError message\u003C\/h2\u003E\n                    \u003Cp\u003EAn unrecoverable error occurred. The uploaded file likely exceeded the maximum file size (1 GB) that this server supports.\u003C\/p\u003E\n          \u003C\/div\u003E\n    \u003C\/div\u003E\n\u003C\/div\u003E\n","settings":null}]

So my guess would be a problem with epex spot itself

I i remove the ajax_form it gives me an actual table tough

[mathieudz]

It seems to be blocked based on the user agent header. When I remove the user agent header from my browser request, I also get 403 Forbidden.

[mathieudz]

It works again after applying this patch: https://github.com/mathieudz/ha_epex_spot/commit/1af3b4ab18c73b64e6d527ae718ff0962ec5ecd9 I don't think it's good to hardcode a random user agent though, therefor I didn't submit it as pull request.

[lieven121]

Can confirm it works Until this is fixed in the repo i used the vscode addon to change the lines. noted by @mathieudz

Thank you @mathieudz

[AkshayRao27]

It works again after applying this patch: mathieudz@1af3b4a I don't think it's good to hardcode a random user agent though, therefor I didn't submit it as pull request.

Worked for me as well. I hope there's a way to fix this issue without hardcoding a user agent, but in the mean time, I'll take it xD

[TomF79]

It works again after applying this patch: mathieudz@1af3b4a I don't think it's good to hardcode a random user agent though, therefor I didn't submit it as pull request.

Worked for me as well. I hope there's a way to fix this issue without hardcoding a user agent, but in the mean time, I'll take it xD

Please can anyone tell me what I have exactly to do?

[PetziAt]

Please can anyone tell me what I have exactly to do?

Open the file editor. Load file custom_components/epex_spot/EPEXSpot/EPEXEXSpotWeb/init.py

and add the lines from the patch posted by mathiudz.

If you don't familiar with coding please wait for the update. ;)

[ullibak]

@TomF79 : Using a text file editor, open the file custom_components/epex_spot/EPEXSpot/EPEXSpotWeb/init.py Then look at mathieudz@1af3b4a

Add the three lines starting with "headers = ..." in line 196-198 (make sure that the indentations are exactly the same) and then change the line starting with "self.URL..." marked in red to the version marked in green. Save the file and restart home assistant.

[WimDeceuninck]

It works again after applying this patch: https://github.com/mathieudz/ha_epex_spot/commit/1af3b4ab18c73b64e6d527ae718ff0962ec5ecd9

But I hope these interuptions can be avoided in the future?

[lieven121]

@WimDeceuninck sadly, i dont think this is in our/ the maintainers control, this uses epex spot website if they change things it will break the integration again

idealy we use the epex spot api but i think this is payed

what i am wondering if we can setup one gateway that for example polls every day (when the new prices are available) and caches this, thus resulting in a small fee since there will be only one call a day

[PetziAt]

It works again after applying this patch: mathieudz@1af3b4a

But I hope these interuptions can be avoided in the future?

@WimDeceuninck No. Most of API providers such EPEX do not announce changes publicy. Only paying customers are getting forefront informations. You can order an commercial API access for € 18.33 / MONTH per user.

See https://webshop.eex-group.com/epex-spot-public-market-data

[PetziAt]

what i am wondering if we can setup one gateway that for example polls every day (when the new prices are available) and caches this, thus resulting in a small fee since there will be only one call a day

It doesn't apply with the EPEX usage policy. ;)

[WimDeceuninck]

@WimDeceuninck sadly, i dont think this is in our/ the maintainers control, this uses epex spot website if they change things it will break the integration again

idealy we use the epex spot api but i think this is payed

what i am wondering if we can setup one gateway that for example polls every day (when the new prices are available) and caches this, thus resulting in a small fee since there will be only one call a day

Hi Lieven, thank you for replying to my comment and of course for your work as contributer. You're right, the integration remains a scraper. I'm only a slow adapter/learner, and a happy user. I am greatfull for those who developped the integration, and those contributing to maintain it.

[mampfes]

Release 2.3.8: https://github.com/mampfes/ha_epex_spot/releases/tag/2.3.8

[lieven121]

@WimDeceuninck sadly, i dont think this is in our/ the maintainers control, this uses epex spot website if they change things it will break the integration again

idealy we use the epex spot api but i think this is payed

what i am wondering if we can setup one gateway that for example polls every day (when the new prices are available) and caches this, thus resulting in a small fee since there will be only one call a day

Hi Lieven, thank you for replying to my comment and of course for your work as contributer. You're right, the integration remains a scraper. I'm only a slow adapter/learner, and a happy user. I am greatfull for those who developped the integration, and those contributing to maintain it.

Hi

Event tough I'm more than happy to take credit for replying to you, i do want send your thanks to @mampfes Since i am no contributor to this project but only a enthousiastic user and try to be helpfull.

Happy to welcome you in the community and hope you have a successful future.

And again thank you to @mampfes ❤️ for the integration