search

Skons / hassio-addons

7 stars 4 forks source link

Access forbidden #63

Open HA-Peet opened 7 months ago

HA-Peet commented 7 months ago

Since several days no more updates on prices, probably after update: No changes to HA except all updates being done without troubles

[2024-02-01 13:12:13] INFO : Using username 'addons' to connect to 'core-mosquitto' [2024-02-01 13:12:13] INFO : Connected with result code '0' [2024-02-01 13:12:13] INFO : Received payload '{ "fuel_type": "euro95", "radius": 5, "latitude": 52.xxxxxxxx, "longitude": 4.xxxxxxx, "to_publish": 3 }' on topic 'dgp/gas_stations' [2024-02-01 13:12:14] ERROR : _write_allstationdata: statuscode '403' [2024-02-01 13:12:14] ERROR : _write_allstationdata: Used header '{'Accept': '/', 'Connection': 'keep-alive', 'User-Agent': 'Mozilla/5.0 (X11; Linux i686 on x86_64; rv:60.2.2) Gecko/20100101 Firefox/60.2.2'}' [2024-02-01 13:12:14] ERROR : _write_allstationdata: Used IP 'xxx.xxx.xxx.xxx' [2024-02-01 13:12:14] ERROR : _write_allstationdata: Response text 'Apache Tomcat/7.0.76 - Error report

HTTP Status 403 - Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen

type Status report

message Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen

description Access to the specified resource has been forbidden.

Apache Tomcat/7.0.76

' [2024-02-01 13:12:14] INFO : There are '0' stations within range for fuel type 'euro95'

Skons commented 7 months ago

How often do you request data? Can you access this website? https://directlease.nl/tankservice/

Have you restarted the addon?

Joostie155 commented 7 months ago

Same issue and log error as HA-Peet, requesting data every 3 hours via automation. https://directlease.nl/tankservice/ does not not show any station on any fuel type in my whole area.

I tested the website on another network and it seems to work, I see all kinds of stations. Seems to be limited or something.

BebeMischa commented 7 months ago

After long months succesfuly running DGP it suddenly stopped here too. Data request once in an hour...

[2024-02-01 09:05:00] INFO    : Received payload '{"fuel_type":"euro98","radius":5,"latitude":52.3344586,"longitude":4.8597941,"friendly_name_template":"[brand] ([station_street])", "to_publish":10}' on topic 'dgp/gas_stations'
[2024-02-01 09:05:00] ERROR   : _write_allstationdata: statuscode '403'
[2024-02-01 09:05:00] ERROR   : _write_allstationdata: Used header '{'Accept': '*/*', 'Connection': 'keep-alive', 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2; rv:55.0.3) Gecko/20100101 Firefox/55.0.3'}'
[2024-02-01 09:05:01] ERROR   : _write_allstationdata: Used IP '95.98.212.210'
[2024-02-01 09:05:01] ERROR   : _write_allstationdata: Response text '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 403 - Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'
[2024-02-01 09:05:01] INFO    : There are '0' stations within range for fuel type 'euro98'
[2024-02-01 09:05:01] INFO    : publishing stations to mqtt
[2024-02-01 09:05:01] INFO    : publishing stations done
[2024-02-01 09:05:01] INFO    : publishing status to mqtt
[2024-02-01 09:05:02] INFO    : publishing status done
Skons commented 7 months ago

Same issue and log error as HA-Peet, requesting data every 3 hours via automation. https://directlease.nl/tankservice/ does not not show any station on any fuel type in my whole area.

It does not seem to be an issue with the addon, but the website is blocking your requests. I do not know why, the only thing i can think of is "excessive" usage of the api. But a few requests every 3 hours does not strike me as excessive.

BebeMischa commented 7 months ago

Worked here since you started this addon (and even before the addon) fine with hourly access. Did they change the limits recently then?

Yasp0 commented 7 months ago

I also have the same issues at my home network, the website https://directlease.nl/tankservice/ shows no stations whatever location I try.

When I try with mobile data on my phone the website returns stations.

Maybe it is blocking IP addresses?

BebeMischa commented 7 months ago

Just woke up today and find it working again. So looks like some hick-up on the website side ;-)

Quevinsta commented 7 months ago

Just woke up today and find it working again. So looks like some hick-up on the website side ;-)

Oke, that's weird. Here it is the same status as before. Still no connection with the Directlease site, also no stations are showed when I visit the physical website.

HA-Peet commented 7 months ago

Standard once an hour, no stations showed on Directlease

Roy75 commented 7 months ago

Same as HA-Peet, same errors, no stations showed on Directlease.

windos2k commented 7 months ago

I have the same issues:

HTTP Status 403 - Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen

[2024-02-04 12:05:20] INFO : There are '0' stations within range for fuel type 'euro95' [2024-02-04 12:05:20] INFO : publishing stations to mqtt [2024-02-04 12:05:20] INFO : publishing stations done [2024-02-04 12:05:20] INFO : publishing status to mqtt [2024-02-04 12:05:21] INFO : publishing status done

misterwesley commented 7 months ago

I e-mailed the address and got this response:

"Ik kan me niet herinneren dat we je ooit toestemming hebben gegeven om onze data zo maar te gebruiken. Lijkt me terecht dat je wordt geblokkeerd.

Als je vindt dat dat anders is horen we graag waar we de factuur heen kunnen sturen voor het gebruik van de data."

So it looks like we need to find another way to get fuel prices.

Yasp0 commented 7 months ago

@misterwesley That's to bad :(

Skons commented 7 months ago

I e-mailed the address and got this response:

"Ik kan me niet herinneren dat we je ooit toestemming hebben gegeven om onze data zo maar te gebruiken. Lijkt me terecht dat je wordt geblokkeerd.

Als je vindt dat dat anders is horen we graag waar we de factuur heen kunnen sturen voor het gebruik van de data."

So it looks like we need to find another way to get fuel prices.

That was a reaction i would have expected from them. So, there are a few things that can be done:

poudenes commented 7 months ago

Hi All,

I have same issue:

This is a part of the log:

<h1>HTTP Status 403 - Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen</h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u>Toegang geweigerd; mail naar tankservice-block@app-it-up.com voor vragen</u></p><p><b>description</b> <u>Access to the specified resource has been forbidden.</u></p>

I saw a other post on HA Forum that the person did mail them and got a reply that the block of the IP was correct. The way he get the gas info was illegal and when the user say that it was correct, the user can send them address for the invoice of using the data of directlease.

So i guess that they block all IP numbers that using the system on different way then access the website.

Skons commented 7 months ago

Does everybody that has this issue, automated the retrieval of the price? Do you request the station regularly? It seems that you can be unblocked as it happened with @BebeMischa. If you have an automation, disable it, wait a while and see if the addon replies (or the website).

Also, if you are mailing to directlease, maybe omit the fact that you are using this addon, but try to show the website https://directlease.nl/tankservice/ with the error. The error can (i think) be retreived by pressing f12, click on the network tab, refresh the page (click the load stations button if needed, and look at the response image

BebeMischa commented 7 months ago

It stopped working here again day later. I will try now longer time between refresh in the automation...

RoelandPer commented 7 months ago

I stopped my automations as well and asked Google if he (she?) knows another source. Came across this one https://www.tankplanner.nl/api/v1/ Don’t know if this could be an alternative provider. They are at least open for contact.

poudenes commented 7 months ago

(A post in dutch)

Dit heb ik naar het email adres gestuurd die in de log voorkomt:

Beste lezer,

Hebben jullie de mogelijkheid om gebruik te maken van een API dienst. 
Hiermee eigen toepassingen te maken om bijvoorbeeld voor privé gebruik 
goedkope tankstations in de omgeving weer te geven in een webportaal 
op een tabled aan de muur.

Benieuwd wat voor een antwoord ik ga krijgen.

fbloemhof commented 7 months ago

I think my ban is permanent. Waited 48 hours but still the same message.

poudenes commented 7 months ago

It's a strange thing. you can download the "DirectLease" application to check gas stations. You can check as many as you want. So basically there is no difference... Even more ... We as HA only do a update every X minutes or hours... So we are less bandwidth users then the application users....

Its the way how we get the information what they bother...

Skons commented 7 months ago

That was running in my mind also, it differs even more i think in the way it retreives the data (it does it relatively efficient). So i was wondering if i could change the way DGP worked, but now @poudenes pointed me to the app, i think things can be simplified. I will take a look into that app to see how and what it is doing. Don't expect anything soon.

My advice for now is: stop your automations and only use only when needed. It has not stopped working for me for instance. I do not request stations every day...

Skons commented 7 months ago

It's a strange thing. you can download the "DirectLease" application to check gas stations. You can check as many as you want. So basically there is no difference... Even more ... We as HA only do a update every X minutes or hours... So we are less bandwidth users then the application users....

Its the way how we get the information what they bother...

I was wondering, can you use the app while the website isn't working? Or is the app also not working?

Quevinsta commented 7 months ago

It's a strange thing. you can download the "DirectLease" application to check gas stations. You can check as many as you want. So basically there is no difference... Even more ... We as HA only do a update every X minutes or hours... So we are less bandwidth users then the application users.... Its the way how we get the information what they bother...

I was wondering, can you use the app while the website isn't working? Or is the app also not working?

Also the app doesn’t work anymore.

poudenes commented 7 months ago

working?

App isn't working also. When I'm.on wifi (same external ip)

robbinonline commented 6 months ago

Unfortunately I am also banned. The app and website no longer work on WiFi. It remains a mysterious industry that this is so difficult to get some fuel prices.

MWeesenaar commented 6 months ago

I stopped my automations as well and asked Google if he (she?) knows another source. Came across this one https://www.tankplanner.nl/api/v1/ Don’t know if this could be an alternative provider. They are at least open for contact.

I quickly made a proof of concept for this, on my local machine. It seems to work pretty well when using the geopy.distance module. I guess I will test it out in my fork and create a PR when it seems to be stable enough.

Did anyone already reach out to the owner(s) of tankplanner? If not; I am happy to do so.

poudenes commented 6 months ago

@MWeesenaar I did not contact company behind Directlease info. I guess they don't wanna unblock us because of some reasons:

1) The iOS application you have to refresh manual. So no intervals. You never exceed the max then 2) There is no other way to get information other than we all did with the integration. That's an automated interval.

I guess they will say something like: The iOS/Android app can't make that many connections, the website also. You did something outside our system and that's not allowed. ( the last part is what I think they gonna say)

fbloemhof commented 6 months ago

I stopped my automations as well and asked Google if he (she?) knows another source. Came across this one https://www.tankplanner.nl/api/v1/ Don’t know if this could be an alternative provider. They are at least open for contact.

I quickly made a proof of concept for this, on my local machine. It seems to work pretty well when using the geopy.distance module. I guess I will test it out in my fork and create a PR when it seems to be stable enough.

Did anyone already reach out to the owner(s) of tankplanner? If not; I am happy to do so.

Yes, I contacted the owner of Tankplanner and he is ok with it to use the API. I started developing an HA integration around it but it would be my first and I'm a little low on spare time currently.

The data behind it is updated every two hours.

I already did some testing with a rest entity for a specific gas station:

rest:
  - resource: https://www.tankplanner.nl/api/v1/price/euro95/
    scan_interval: 7200
    sensor:
      - name: "Tankstation Fieten Olie Peize"
        unit_of_measurement: "€"
        icon: mdi:fuel
        unique_id: gas_station_euro95_peize
        value_template: >-
          {{ value_json | selectattr('organization', 'eq', 'Fieten Olie') | selectattr('address', 'eq', 'Havenstraat 12') | map(attribute='price') | list | first }}
        json_attributes_path: "$.[?(@.address=='Havenstraat 12')]"
        json_attributes:
          - organization
          - address
          - town
          - gps
MWeesenaar commented 6 months ago

@MWeesenaar I did not contact company behind Directlease info. I guess they don't wanna unblock us because of some reasons:

  1. The iOS application you have to refresh manual. So no intervals. You never exceed the max then
  2. There is no other way to get information other than we all did with the integration. That's an automated interval.

I guess they will say something like: The iOS/Android app can't make that many connections, the website also. You did something outside our system and that's not allowed. ( the last part is what I think they gonna say)

I meant the alternative that was provided: tankplanner.nl/api/v1.

-- Looks great @fbloemhof ! I was implementing along the way how the current addon is working: provide GPS coordinates and a maximum distance and check the cheapest locations. Whenever I got something working, I will post an update in this repo.

fbloemhof commented 6 months ago

@MWeesenaar I did not contact company behind Directlease info. I guess they don't wanna unblock us because of some reasons:

  1. The iOS application you have to refresh manual. So no intervals. You never exceed the max then
  2. There is no other way to get information other than we all did with the integration. That's an automated interval.

I guess they will say something like: The iOS/Android app can't make that many connections, the website also. You did something outside our system and that's not allowed. ( the last part is what I think they gonna say)

I meant the alternative that was provided: tankplanner.nl/api/v1.

-- Looks great @fbloemhof ! I was implementing along the way how the current addon is working: provide GPS coordinates and a maximum distance and check the cheapest locations. Whenever I got something working, I will post an update in this repo.

I already have some Python ready with indeed similar behavior as the DGP addon, but still need to convert this to an integration.

The owner told that it is not a problem to make this call every two hours, filtering the needed data locally. As long as you mention the source of it in any documentation.

sga-noud commented 6 months ago

I already did some testing with a rest entity for a specific gas station:

rest:
  - resource: https://www.tankplanner.nl/api/v1/price/euro95/
    scan_interval: 7200
    sensor:
      - name: "Tankstation Fieten Olie Peize"
        unit_of_measurement: "€"
        icon: mdi:fuel
        unique_id: gas_station_euro95_peize
        value_template: >-
          {{ value_json | selectattr('organization', 'eq', 'Fieten Olie') | selectattr('address', 'eq', 'Havenstraat 12') | map(attribute='price') | list | first }}
        json_attributes_path: "$.[?(@.address=='Havenstraat 12')]"
        json_attributes:
          - organization
          - address
          - town
          - gps

Thanks a lot @RoelandPer for finding this other resource, and thanks @fbloemhof for the REST sensor example. This is a neat alternative, especially since it doesn’t require as much resources as the image/text recognition from this add-on. @fbloemhof if you have a similar REST example for getting the cheapest stations nearby (or, along a route given that that’s the only way this tankplanner API seems to work), that would be awesome.

crispy78 commented 6 months ago

I e-mailed the address and got this response:

"Ik kan me niet herinneren dat we je ooit toestemming hebben gegeven om onze data zo maar te gebruiken. Lijkt me terecht dat je wordt geblokkeerd.

Als je vindt dat dat anders is horen we graag waar we de factuur heen kunnen sturen voor het gebruik van de data."

So it looks like we need to find another way to get fuel prices.

Their website states: "Al vanaf de introductie van de eerste smartphones maken wij apps voor iOS (Apple) en Android (Google)." ("We have been making apps for iOS (Apple) and Android (Google) since the introduction of the first smartphones."). I guess the internet must haven been very poor or blocked for decades now in the Enschede region, because they don't seem to grasp the power of the internet and bad press. I'm not arguing their point of view, I'm only arguing their response.

Has someone already asked this company what the use of their database would cost?

bobskee commented 3 months ago

I stopped my automations as well and asked Google if he (she?) knows another source. Came across this one https://www.tankplanner.nl/api/v1/ Don’t know if this could be an alternative provider. They are at least open for contact.

I quickly made a proof of concept for this, on my local machine. It seems to work pretty well when using the geopy.distance module. I guess I will test it out in my fork and create a PR when it seems to be stable enough. Did anyone already reach out to the owner(s) of tankplanner? If not; I am happy to do so.

Yes, I contacted the owner of Tankplanner and he is ok with it to use the API. I started developing an HA integration around it but it would be my first and I'm a little low on spare time currently.

The data behind it is updated every two hours.

I already did some testing with a rest entity for a specific gas station:

rest:
  - resource: https://www.tankplanner.nl/api/v1/price/euro95/
    scan_interval: 7200
    sensor:
      - name: "Tankstation Fieten Olie Peize"
        unit_of_measurement: "€"
        icon: mdi:fuel
        unique_id: gas_station_euro95_peize
        value_template: >-
          {{ value_json | selectattr('organization', 'eq', 'Fieten Olie') | selectattr('address', 'eq', 'Havenstraat 12') | map(attribute='price') | list | first }}
        json_attributes_path: "$.[?(@.address=='Havenstraat 12')]"
        json_attributes:
          - organization
          - address
          - town
          - gps

It seems that either the domain tankplanner.nl or the server behind it has changed. Your solution worked perfectly before that. I would like to get in touch with its owner about this but just wanted to check if you are already in contact about this matter.

fbloemhof commented 3 months ago

I stopped my automations as well and asked Google if he (she?) knows another source. Came across this one https://www.tankplanner.nl/api/v1/ Don’t know if this could be an alternative provider. They are at least open for contact.

I quickly made a proof of concept for this, on my local machine. It seems to work pretty well when using the geopy.distance module. I guess I will test it out in my fork and create a PR when it seems to be stable enough. Did anyone already reach out to the owner(s) of tankplanner? If not; I am happy to do so.

Yes, I contacted the owner of Tankplanner and he is ok with it to use the API. I started developing an HA integration around it but it would be my first and I'm a little low on spare time currently. The data behind it is updated every two hours. I already did some testing with a rest entity for a specific gas station:

rest:
  - resource: https://www.tankplanner.nl/api/v1/price/euro95/
    scan_interval: 7200
    sensor:
      - name: "Tankstation Fieten Olie Peize"
        unit_of_measurement: "€"
        icon: mdi:fuel
        unique_id: gas_station_euro95_peize
        value_template: >-
          {{ value_json | selectattr('organization', 'eq', 'Fieten Olie') | selectattr('address', 'eq', 'Havenstraat 12') | map(attribute='price') | list | first }}
        json_attributes_path: "$.[?(@.address=='Havenstraat 12')]"
        json_attributes:
          - organization
          - address
          - town
          - gps

It seems that either the domain tankplanner.nl or the server behind it has changed. Your solution worked perfectly before that. I would like to get in touch with its owner about this but just wanted to check if you are already in contact about this matter.

Yes I contacted Sjoerd, he received a cease and desist from one of the (sub) data sources. He took down the service for the time being. Hopefully it will be back online soon.