[Bug]: abfall_io source for f_id_kommune 899 (AWB Göppingen) needs a time period in the request to get a valid ICS response

exbachi commented 6 months ago

I Have A Problem With:

A specific source

What's Your Problem

My waste scheduler source for f_id_kommune: 899 stopped working a few days ago. After some tests to verify my configuration, I realised that the request from generated from the awb-gp.de adds a timeperiod=20240101-20241231 parameter. Without this, the ICS has no values For the moment I added the generated URL as ICS source and it works now. But the Source abfall_io seems to be broken without this timeperion parameter (which cant be set in the configuration)

Source (if relevant)

No response

Logs

no relevant logs

Relevant Configuration

- name: abfall_io
      args:
        key: !secret abfall_io_key
        f_id_kommune: 899
        f_id_strasse: XXX
        f_abfallarten:
          - 20
          - 17
          - 59
          - 18
          - 19
          - 60
      calendar_title: "Müllkalender"
      customize:
        - type: Hausmüll
          icon: mdi:trash-can
        - type: Gelber Sack
          icon: mdi:recycle
        - type: Bioabfall
          icon: mdi:biohazard
        - type: Grüngutsammlung
          icon: mdi:leaf

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[X] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[X] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

gbyhtn commented 6 months ago

I can confirm the issue. f_id_kommune: 908 isn't working any longer, too. (also AWB Göppingen) Seems to do not fetch any data, due to missing parameter.

Wilkware commented 6 months ago

I can also confirm the issue, but the parameter timeperiod is completly different from the normal request. It seems to be a new qraphql based widget?!?

exbachi commented 6 months ago

I can also confirm the issue, but the parameter timeperiod is completly different from the normal request. It seems to be a new qraphql based widget?!?

For me the parameter was part of the GET parameters directly in the URL like the other parameters, too

allgoewer commented 6 months ago

f_id_kommune 911 is also not working. AWB Göppingen as well.

fabssor commented 6 months ago

Can confirm the same for f_id_kommune 913. @exbachi Tried to get it manually working but no luck. Can you post your current workaround? Thanks!

ET1981 commented 6 months ago

f_id_kommune: 902 is also not working. AWB Göppingen as well.

cfkugler commented 6 months ago

f_id_kommune: 898 is also not working. AWB Göppingen as well.

exbachi commented 6 months ago

Can confirm the same for f_id_kommune 913. @exbachi Tried to get it manually working but no luck. Can you post your current workaround? Thanks!

I used the developer tools in chrome to get the request to download the ICS: https://api.abfall.io/?key=XXXX&mode=export&idhousenumber=XXXX&wastetypes=20,17,59,18,19,60&timeperiod=20240101-20241231&showinactive=false&type=ics

You can create your own URL here and get it with the dev tools: https://awb-gp.de/abfallabholung/abfuhrtermine Then you can use the URL as an ICS source as waste scheduler source

Wilkware commented 6 months ago

The wohle AWG Göttingen ist out of order ...

894 = Adelberg 895 = Aichelberg 896 = Albershausen 897 = Bad Boll 898 = Bad Ditzenbach 899 = Bad Überkingen 900 = Birenbach 901 = Böhmenkirch 902 = Börtlingen 903 = Deggingen 904 = Donzdorf 905 = Drackenstein 906 = Dürnau 907 = Ebersbach 908 = Eislingen 909 = Eschenbach 910 = Gammelshausen 911 = Geislingen 912 = Gingen 913 = Göppingen 914 = Gruibingen 915 = Hattenhofen 916 = Heiningen 917 = Hohenstadt 918 = Kuchen 919 = Lauterstein 920 = Mühlhausen 921 = Ottenbach 922 = Rechberghausen 923 = Salach 924 = Schlat 925 = Schlierbach 926 = Süßen 927 = Uhingen 928 = Wangen 929 = Wäschenbeuren 930 = Wiesensteig 931 = Zell u. A.

I used the developer tools in chrome to get the request to download the ICS: https://api.abfall.io/?key=XXXX&mode=export&idhousenumber=XXXX&wastetypes=20,17,59,18,19,60&timeperiod=20240101-20241231&showinactive=false&type=ics

You can create your own URL here and get it with the dev tools: https://awb-gp.de/abfallabholung/abfuhrtermine Then you can use the URL as an ICS source as waste scheduler source

Xes, but it is not a solution here in this case!

exbachi commented 6 months ago

The wohle AWG Göttingen ist out of order ...

894 = Adelberg 895 = Aichelberg 896 = Albershausen 897 = Bad Boll 898 = Bad Ditzenbach 899 = Bad Überkingen 900 = Birenbach 901 = Böhmenkirch 902 = Börtlingen 903 = Deggingen 904 = Donzdorf 905 = Drackenstein 906 = Dürnau 907 = Ebersbach 908 = Eislingen 909 = Eschenbach 910 = Gammelshausen 911 = Geislingen 912 = Gingen 913 = Göppingen 914 = Gruibingen 915 = Hattenhofen 916 = Heiningen 917 = Hohenstadt 918 = Kuchen 919 = Lauterstein 920 = Mühlhausen 921 = Ottenbach 922 = Rechberghausen 923 = Salach 924 = Schlat 925 = Schlierbach 926 = Süßen 927 = Uhingen 928 = Wangen 929 = Wäschenbeuren 930 = Wiesensteig 931 = Zell u. A.

I used the developer tools in chrome to get the request to download the ICS: https://api.abfall.io/?key=XXXX&mode=export&idhousenumber=XXXX&wastetypes=20,17,59,18,19,60&timeperiod=20240101-20241231&showinactive=false&type=ics

You can create your own URL here and get it with the dev tools: https://awb-gp.de/abfallabholung/abfuhrtermine Then you can use the URL as an ICS source as waste scheduler source

Xes, but it is not a solution here in this case!

It is the workaround for fabssor to get the dates back working in HA, not a solution for the Problem

fabssor commented 6 months ago

I investigated a little bit more and got abfall.io working again for Göppingen. It seems to be that the API got changed at least for Göppingen. I tried my code for ALBA Berlin but this seems to be still using the old API. Also can be seen on the ALBA Berlin website

Wilkware commented 6 months ago

The funny thing is, the old API works still for Göppingen, but there are no EVENTS in the ICS file. This is always empty :(

gbyhtn commented 6 months ago

I investigated a little bit more and got abfall.io working again for Göppingen. It seems to be that the API got changed at least for Göppingen. I tried my code for ALBA Berlin but this seems to be still using the old API. Also can be seen on the ALBA Berlin website....

@fabssor @Wilkware : Does that mean you got 'waste-collection-schedule' back to working, or just being able to export working data to ICS? For me the problem wasn't getting data into the HA calendar, this could be done manually, but my cards with the integration aren't working anymore, so in my opinion the API isn't working for Goeppingen anymore.

fabssor commented 6 months ago

I got waste Collection Integration working again by migrating the abfall_io Python Script to the new API. I can later Share my Modifications.

Sadly the New API is Not supported by all communes. So my changes work for Göppingen but not for Berlin.

fabssor commented 6 months ago

this is my modified python script which seems to be working and seems not to break the old API:

import datetime
import logging
import re
from html.parser import HTMLParser

import requests
from waste_collection_schedule import Collection  # type: ignore[attr-defined]
from waste_collection_schedule.service.AbfallIO import SERVICE_MAP
from waste_collection_schedule.service.ICS import ICS

TITLE = "Abfall.IO / AbfallPlus"
DESCRIPTION = (
    "Source for AbfallPlus.de waste collection. Service is hosted on abfall.io."
)
URL = "https://www.abfallplus.de"
COUNTRY = "de"

def EXTRA_INFO():
    return [{"title": s["title"], "url": s["url"]} for s in SERVICE_MAP]

TEST_CASES = {
    "Waldenbuch": {
        "key": "8215c62763967916979e0e8566b6172e",
        "f_id_kommune": 2999,
        "f_id_strasse": 1087,
        # "f_abfallarten": [50, 53, 31, 299, 328, 325]
    },
    "Landshut": {
        "key": "bd0c2d0177a0849a905cded5cb734a6f",
        "f_id_kommune": 2655,
        "f_id_bezirk": 2655,
        "f_id_strasse": 763,
        # "f_abfallarten": [31, 17, 19, 218]
    },
    "Schoenmackers": {
        "key": "e5543a3e190cb8d91c645660ad60965f",
        "f_id_kommune": 3682,
        "f_id_strasse": "3682adenauerplatz",
        "f_id_strasse_hnr": "20417",
        # "f_abfallarten": [691,692,696,695,694,701,700,693,703,704,697,699],
    },
    "Freudenstadt": {
        "key": "595f903540a36fe8610ec39aa3a06f6a",
        "f_id_kommune": 3447,
        "f_id_bezirk": 22017,
        "f_id_strasse": 22155,
    },
    "Ludwigshafen am Rhein": {
        "key": "6efba91e69a5b454ac0ae3497978fe1d",
        "f_id_kommune": "5916",
        "f_id_strasse": "5916abteistrasse",
        "f_id_strasse_hnr": 33,
    },
    "Traunstein": {
        "key": "279cc5db4db838d1cfbf42f6f0176a90",
        "f_id_kommune": "2911",
        "f_id_strasse": "2374",
    },
    "AWB Limburg-Weilburg": {
        "key": "0ff491ffdf614d6f34870659c0c8d917",
        "f_id_kommune": 6031,
        "f_id_strasse": 621,
        "f_id_strasse_hnr": 872,
        "f_abfallarten": [27, 28, 17, 67],
    },
    "ALBA Berlin": {
        "key": "9583a2fa1df97ed95363382c73b41b1b",
        "f_id_kommune": 3227,
        "f_id_strasse": 3475,
        "f_id_strasse_hnr": 185575,
    },
    "Göppingen": {
        "key": "f35bd08b1d18d9c81fcdee75dbcce5d3",
        "idhousenumber": 7074,
        "wastetypes": [20,17,59,18,19,60]
    },
}
_LOGGER = logging.getLogger(__name__)

MODUS_KEY = "d6c5855a62cf32a4dadbc2831f0f295f"
HEADERS = {"user-agent": "Mozilla/5.0 (xxxx Windows NT 10.0; Win64; x64)"}

# Parser for HTML input (hidden) text
class HiddenInputParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self._args = {}

    @property
    def args(self):
        return self._args

    def handle_starttag(self, tag, attrs):
        if tag == "input":
            d = dict(attrs)
            if d["type"] == "hidden":
                self._args[d["name"]] = d["value"]

class Source:
    def __init__(
        self,
        key,
        f_id_kommune = None,
        f_id_strasse = None,
        f_id_bezirk=None,
        f_id_strasse_hnr=None,
        f_abfallarten=[],
        idhousenumber = None, # New API
        wastetypes = None, # New API
    ):
        self._key = key
        self._kommune = f_id_kommune
        self._bezirk = f_id_bezirk
        self._strasse = f_id_strasse
        self._strasse_hnr = f_id_strasse_hnr
        self._abfallarten = f_abfallarten  # list of integers
        self._idhousenumber = idhousenumber
        self._wastetypes = wastetypes
        self._ics = ICS()

    def fetch(self):
        if self._idhousenumber == None:
            # get token
            params = {"key": self._key, "modus": MODUS_KEY, "waction": "init"}

            r = requests.post("https://api.abfall.io", params=params, headers=HEADERS)

            # add all hidden input fields to form data
            # There is one hidden field which acts as a token:
            # It consists of a UUID key and a UUID value.
            p = HiddenInputParser()
            p.feed(r.text)
            args = p.args

            args["f_id_kommune"] = self._kommune
            args["f_id_strasse"] = self._strasse

            if self._bezirk is not None:
                args["f_id_bezirk"] = self._bezirk

            if self._strasse_hnr is not None:
                args["f_id_strasse_hnr"] = self._strasse_hnr

            for i in range(len(self._abfallarten)):
                args[f"f_id_abfalltyp_{i}"] = self._abfallarten[i]

            args["f_abfallarten_index_max"] = len(self._abfallarten)
            args["f_abfallarten"] = ",".join(map(lambda x: str(x), self._abfallarten))

            now = datetime.datetime.now()
            date2 = now.replace(year=now.year + 1)
            args["f_zeitraum"] = f"{now.strftime('%Y%m%d')}-{date2.strftime('%Y%m%d')}"

            params = {"key": self._key, "modus": MODUS_KEY, "waction": "export_ics"}

            # get csv file
            r = requests.post(
                "https://api.abfall.io", params=params, data=args, headers=HEADERS
            )
        else:
            params = {"key": self._key, "mode": "export", "type": "ics", "showinactive": "false"}

            now = datetime.datetime.now()
            params["timeperiod"] = f"{now.strftime('%Y0101')}-{now.strftime('%Y1231')}"
            params["idhousenumber"] = self._idhousenumber
            params["wastetypes"] = ','.join(str(wastetype) for wastetype in self._wastetypes)

            r = requests.get(
                "https://api.abfall.io", params=params, headers=HEADERS
            )

        # parse ics file
        r.encoding = "utf-8"  # requests doesn't guess the encoding correctly
        ics_file = r.text

        # Remove all lines starting with <b
        # This warning are caused for customers which use an extra radiobutton
        # list to add special waste types:
        # - AWB Limburg-Weilheim uses this list to select a "Sonderabfall <city>"
        #   waste type. The warning could be removed by adding the extra config
        #   option "f_abfallarten" with the following values [27, 28, 17, 67]
        html_warnings = re.findall(r"\<b.*", ics_file)
        if html_warnings:
            ics_file = re.sub(r"\<br.*|\<b.*", "\\r", ics_file)
            # _LOGGER.warning("Html tags removed from ics file: " + ', '.join(html_warnings))

        dates = self._ics.convert(ics_file)

        entries = []
        for d in dates:
            entries.append(Collection(d[0], d[1]))
        return entries

To make it work you need to supply the new parameters idhousenumber and wastetypes. idhousenumber is the same number as f_id_strasse. wastetypes should be self explanatory. An example configuration could look like this:

  sources:
    - name: abfall_io
      args:
        key: keykeykeykeykey
        idhousenumber: 1234456
        wastetypes: [20,17,59,18,19]

Now only the wizzard needs also to be adapted.

fabssor commented 6 months ago

Just explored a little bit more. As already mentioned there seems to be a graphql API (https://studio.apollographql.com/public/Abfallplus/variant/v1/home). The entry point of this should be the following query:

query GetCities($query: String) {
    cities(query: $query) {
        id
        name
        idHouseNumber
        districtsCount
        appointmentsSupported
    }
  }

This will return all cities in the commune with a unique ID. For Authentication the Header X-Abfallplus-Api-Key must be set. For Göppingen it is the value 7a4730955599a7e7c0ceb39c548b6abbae874dc7e6a0afc5. This key seems to be tied to the commune (here Göppingen). The API-Key can be get via GET https://api.abfall.io/?key=f35bd08b1d18d9c81fcdee75dbcce5d3&modus=d6c5855a62cf32a4dadbc2831f0f295f&waction=init, which will responde with a JSON object. The other communes will reponsed here with a form. So maybe they are migrating commune for commune to the new GraphQL based API?

mampfes / hacs_waste_collection_schedule