[Bug]: app_abfallplus_de returns names with numbers instead of the types

OnSive commented 1 week ago

I Have A Problem With:

A specific source

What's Your Problem

https://community.home-assistant.io/t/waste-collection-schedule-framework/186492/542

The attribute of my sensor do only contain names with numbers.

Source (if relevant)

app_abfallplus_de / ZAK Abfall App

Logs

N/A

Relevant Configuration

Sensor (you can use the following wizard: custom_components/waste_collection_schedule/waste_collection_schedule/wizard/app_abfallplus_de.py)

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappzak
        city: Fischen
        bezirk: Ortsteile

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[X] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[ ] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

specularius commented 6 days ago

Same here with these settings

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
       args:
          app_id: de.k4systems.lkemmendingen
          city: Biederbach
          strasse: Alle Straßen

5ila5 commented 6 days ago

That's quite a neat bug. Did some initial digging: The app does return the correct data, but the reverse engineered way returns this super weird data (I do not know who's names these are and why they are returned here).

Debugging this is pretty annoying, as it's often not quite clear what went wrong exactly. I already did some trial and error but without success. I will take a deeper look in the coming days.

OnSive commented 6 days ago

@5ila5 can you give me a hint where to start debugging? Couldn't find a starting point as first time user of ha and waste collection schedule.

Btw. I hope that those are not real people as there would be way to many less then 18 year old people in a waste collector database imo...

5ila5 commented 6 days ago

Generally debugging sources:

Cloning the reposetory
modifing the files (sources are placed in custom_components/waste_collection_schedule/waste_collection_schedule/source/ some sources have a corresponding Service script (app_abfallplus_de has one), which is called by the source script. it should be located at custom_components/waste_collection_schedule/waste_collection_schedule/service/
You can test the modified source using the test script documented here

app_abfallplus_de

To compare the calls made by the source with the request of the real app, you need: an android (vm?) or IOS (don't know hot this works) device where you can inspect the traffic made by the phone (I use Http Toolkit to inspect the traffic made by my android VM running inside Android Studio)

Some changes like the User-Angent and client ID were made with an IPhone (as far as I can tell). But I tried to change them (Andoird has 2 different User-Agents for different requests and client_id must be 5 separate sections instead of one long integer string) and it did not solve anything, so I think the error is probably somewhere else.

OnSive commented 6 days ago

@5ila5 thanks that's helpful, maybe i can find a hint to the bug. Got the dev env running 👍

5ila5 commented 6 days ago

I put together the requests that are made by the app. https://gist.github.com/5ila5/205672118616e30a2eaccbc89db42df4 Running this returns the XML with the names instead of the bin types, the app gets another XML (with the correct bin types), I do not find the difference between these requests and those made by the app.

HenryLoenwind commented 5 days ago

BTW, I suggest you complain to your local council (that is responsible for this public service) about this data being gated behind a private company that actively tries to prevent citizens from accessing it.

Tigra16v commented 5 days ago

The same Problem with these settings:

waste_collection_schedule: sources:

name: app_abfallplus_de args: app_id: de.k4systems.abfallinfocw city: Egenhausen strasse: Alle Straßen

two days ago everything was still going.

Fireblade900rr commented 5 days ago

Also the same for my city.

waste_collection_schedule: sources:

name: app_abfallplus_de args: app_id: de.k4systems.abfallinfoapp city: Weilerswist

coarsy commented 4 days ago

And here the same too. Waste collection schedulde is not running anymore:

waste_collection_schedule: sources:

name: app_abfallplus_de args: app_id: de.k4systems.zakb city: Heppenheim strasse: Schubertweg hnr: 6

bug_waste

MatzzeF commented 4 days ago

@HenryLoenwind pretty sure that this is a issue with the HA Plugin and not the data provider. The apps from the waste companies runs pretty fine here and shows the right data.

x4N70pHyLL commented 4 days ago

Same issue for Rastatt (Abfallplus). Getting Names with Numbers

mngsps commented 4 days ago

My sensors are all "unknown" since 29th of june...

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen

roninf commented 3 days ago

Same here for:

- name: app_abfallplus_de args: app_id: de.k4systems.abfallappbh

Looks to me like birthdays....

boun commented 3 days ago

@ReneNulschDE in my region de.k4systems.zawdw I did not get strange names, but calendar was empty (sometimes a strange date for a single event). I changed the code before the self.get_collections() call to set self._client to an id, that I sniffed from the app and sure enough, the calendar was filled up again. So there seems to be a problem in the set-up phase. I tested your changes but they did not fix the problem.

garionth commented 3 days ago

My sensors are all "unknown" since 29th of june...

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen

In your case i would suggest to check the log. I had the same (sensors going to unknown) a couple of days ago. In the logs i could see that my provider (also app_abfallplus_de) changed the required parameters for the config. The log said "Alle Straßen" is not a valid street name. And then gave a list of all the streetnames of my city.

btw thanks who ever is responsible for putting that return into the log. Very helpfull.

It was like in your case

city: real_city_name strasse: Alle Straßen (all streets)

and is now city: real_city_name strasse: real_street_name hsnr: real_number

garionth commented 3 days ago

I don't know if it helps, but from the looks of things on my end, abfallplus is still changing things atm.

2 days ago i did not have the name problem like the OP, but fantasy dates for the pickup. (between 60 and 200+ days in the future) on my sensors.

Now they are all unknown.

And as i said in the above post. 2-3 weeks ago the config for my city street changed. But this of course, could be a "normal" update on their side.

mngsps commented 3 days ago

My sensors are all "unknown" since 29th of june...
waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappbh
        city: Au
        strasse: Alle Straßen
In your case i would suggest to check the log. I had the same (sensors going to unknown) a couple of days ago. In the logs i could see that my provider (also app_abfallplus_de) changed the required parameters for the config. The log said "Alle Straßen" is not a valid street name. And then gave a list of all the streetnames of my city.

btw thanks who ever is responsible for putting that return into the log. Very helpfull.

It was like in your case

city: real_city_name strasse: Alle Straßen (all streets)

and is now city: real_city_name strasse: real_street_name hsnr: real_number

Thank you for the input! In their App its still "Alle Straßen" (all streets) in my case. Furthermore, nothing noticable in the logs:

2024-06-30 23:07:14.640 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration waste_collection_schedule which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-06-30 23:07:22.509 INFO (MainThread) [homeassistant.setup] Setting up waste_collection_schedule
2024-06-30 23:07:32.575 WARNING (MainThread) [homeassistant.setup] Setup of waste_collection_schedule is taking over 10 seconds.
2024-06-30 23:07:44.093 INFO (MainThread) [homeassistant.setup] Setup of domain waste_collection_schedule took 21.58 seconds
2024-06-30 23:07:44.095 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.116 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.119 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:44.131 INFO (MainThread) [homeassistant.components.sensor] Setting up waste_collection_schedule.sensor
2024-06-30 23:07:55.478 INFO (MainThread) [homeassistant.components.calendar] Setting up waste_collection_schedule.calendar

AKGitH commented 3 days ago

Yesterday I wrote an e-mail to support@abfallplus.de and explained the API problem getting person names back. grafik

But till now I got no feedback, but it looks that someone is playing arround with the data, today I get "unkown" back. Have a look to the data update date/time: grafik

Tigra16v commented 3 days ago

Same for me. After I tried it with street and house number and then undid it, I no longer have any names with numbers, just "unknown".

anderl78 commented 2 days ago

For me too.... I get only "unknown"

Fireblade900rr commented 2 days ago

Crazy, but after a restart for other reasons its working as expected, witout changes.

waste_collection_schedule: sources:

name: app_abfallplus_de args: app_id: de.k4systems.abfallinfoapp city: Weilerswist

Tigra16v commented 2 days ago

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

roninf commented 2 days ago

My calendar is empty meanwhile without any entry...

anderl78 commented 2 days ago

Crazy, but after a restart for other reasons its working as expected, witout changes.

waste_collection_schedule: sources:

name: app_abfallplus_de args: app_id: de.k4systems.abfallinfoapp city: Weilerswist

Doesn't work for me... after reboot still "unknow"

anderl78 commented 2 days ago

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

mngsps commented 2 days ago

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again. (In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

Same for me. No change after removing duplicates...

anderl78 commented 2 days ago

Oh, I just tested the abfallplus-app on my phone - it works without any problem... I even changed street and Number to see if it really gets new data...changed back, got my data back. Even checked there website - works well, all data is present there.

Tigra16v commented 2 days ago

I found the solution. At least in my case it works. I noticed that the sensor calendar.apps_by_abfall was present multiple times. calendar.apps_by_abfall_1 calendar.apps_by_abfall_2 and so on. After I deleted all but one and restarted home assistant everything works again.

(In Settings -> Devices and services -> Entities)

Thank you. I got the clones too. Deleted them too, followed by a reboot - but still the same problem : state of my sensors is still "unknown"

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

mngsps commented 2 days ago

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

I did exactly this. No change, still "unknown"

Tigra16v commented 2 days ago

The sensor that is left must not have a number at the end. Otherwise it must be renamed. It must be called calendar.apps_by_abfall.

I did exactly this. No change, still "unknown"

I'm sorry. It helped me. Then there are probably other reasons. I wish you continued luck

ReneNulschDE commented 2 days ago

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

Custom request header structure for the first request
Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

anderl78 commented 2 days ago

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

Custom request header structure for the first request
Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

Hello!

Thank you very much! Seems to fix the Problem for me!

Lackii commented 2 days ago

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Thanks a lot, this fixes the issue also for me. Before i have had the strange entries with names and since monday an empty schedule. Clearing the double calendar.apps_by_abfall entries did not changed anything.

AKGitH commented 2 days ago

After HA restart, I got this errors: grafik

Moving back to the original file, no errors showed, but still get "unkown".

mngsps commented 2 days ago

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

Custom request header structure for the first request

Not more than one request per second

Solved the problem in my case as well! Thank you!!

ReneNulschDE commented 1 day ago

@AKGitH : Your screenshot does not show the error message. Please share the log entry and your config.

AKGitH commented 1 day ago

Hi "ReneNulschDE", I have downloaded the py-file again and now it works. Great Job!

For your information: I use: -name: app_abfallplus_de -app_id: de.ucom.abfallavr xxx

Many thanks!

anderl78 commented 1 day ago

Btw, in case you need the information, Im using: name: app_abfallplus_de app_id: de.k4systems.zawdw

lorddagobert commented 1 day ago

@ReneNulschDE

Solved the problem also in my case as well! Thank you!!

name: app_abfallplus_de app_id: de.idcontor.abfalllu

Many thanks again

tobibachert commented 1 day ago

I spend some more hours on this topic. Here is a code that can be checked. (The code is not final but a starting point.)

Changes:

Custom request header structure for the first request
Not more than one request per second

Testing source app_abfallplus_de ...
  found 84 entries for de.k4systems.abfallappnf Ahrenviöl alle Straßen
  found 26 entries for de.albagroup.app Braunschweig Hauptstraße 7A  
  found 70 entries for de.k4systems.bonnorange Auf dem Hügel
  found 139 entries for de.ucom.abfallavr Brühl Habichtstr. 4A
  found 45 entries for de.k4systems.abfallappwug Bergen hauptstr. 1
  found 78 entries for de.k4systems.abfallappcux Wurster Nordseeküste Aakweg Alle Hausnummern
  found 40 entries for de.abfallwecker Mutzschen, Am Lindigt 1
  found 48 entries for de.k4systems.leipziglk Brandis Brandis
  found 46 entries for de.k4systems.leipziglk Machern Machern
  found 64 entries for de.k4systems.lkgoettingen, Abfallwirtschaft Altkreis Göttingen,  Adelebsen, Alle Straßen

manual testing (abfallappzak - Fischen, lkemmendingen - Biederbach, abfallinfoapp - Weilerswist)

THAT solved it for me. Thanks a lot @ReneNulschDE

Tigra16v commented 1 day ago

After the last core update I have the problem again. (sensor status unknown) Since I'm still a complete beginner and my English isn't great, I honestly don't understand what exactly solved the problem for you. Can someone please explain to me what exactly I need to do?

Or is there anything wrong with explaining it to me in German? As far as I can see, everyone here is German, right?

coarsy commented 1 day ago

@Tigra16v You have to replace the content of the file AppAbfallplusDe.py with this: https://github.com/mampfes/hacs_waste_collection_schedule/commit/6051222586494d6e31c1e1bf37542e91dc780815

Tigra16v commented 23 hours ago

ok. and how exactly do I do that? I can't copy the text in the link (in the github app).

thoduh commented 23 hours ago

Press the Link, then Press the tree Points:

Press "View File" Press "Copy raw file" Paste in Homeassistant...

thoduh commented 23 hours ago

@Tigra16v You have to replace the content of the file AppAbfallplusDe.py with this: 6051222

worked. thank you!

Tigra16v commented 23 hours ago

OK. I got it working. But via the browser and not in the app. But now nothing works. The sensors are no longer available and I get the following error message

Logger: homeassistant.setup Quelle: setup.py:485 Erstmals aufgetreten: 17:22:14 (5 Vorkommnisse) Zuletzt protokolliert: 17:22:14

Unable to prepare setup for platform 'waste_collection_schedule.sensor': Unable to set up component.

chma73 commented 22 hours ago

@Tigra16v I think your AppAbfallplusDe.py is not correct, are you sure that you did the copy and replace correct.

chma73 commented 22 hours ago

@ReneNulschDE thanks for your effort and support, your update works fine for me

Tigra16v commented 22 hours ago

At least I'm not aware of what I did wrong. I also get the following error message:Logger: homeassistant.setup Quelle: setup.py:404 Erstmals aufgetreten: 18:25:35 (1 Vorkommnisse) Zuletzt protokolliert: 18:25:35

Error during setup of component waste_collection_schedule Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 404, in _async_setup_component result = await task ^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/init.py", line 113, in async_setup await hass.async_add_executor_job( File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, *self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/init.py", line 201, in add_source_shell new_shell = SourceShell.create( ^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 204, in create source_module = importlib.import_module( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func return func(args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/app_abfallplus_de.py", line 1, in import waste_collection_schedule.service.AppAbfallplusDe as AppAbfallplusDe File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/AppAbfallplusDe.py", line 541 for a in soup.find_all( ^ SyntaxError: '(' was never closed

mampfes / hacs_waste_collection_schedule