[Bug]: de.k4systems.abfallappnf no entries in the calender

redskull1986 commented 4 months ago

I Have A Problem With:

A specific source

What's Your Problem

Hello,

I've the same problem as before 1.48.0. No entries in the calender.

It's like the Bug 2033

I've update and reinstall the integration.

HA Version 2024.1.6

Source (if relevant)

de.k4systems.abfallappnf

Logs

Dieser Fehler wurde von einer benutzerdefinierten Integration verursacht

Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
Integration: waste_collection_schedule (documentation)
First occurred: 20:46:54 (1 occurrences)
Last logged: 20:46:54

fetch failed for source Apps by Abfall+: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/app_abfallplus_de.py", line 178, in fetch for d in self._app.generate_calendar(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/AppAbfallplusDe.py", line 918, in generate_calendar return self.get_collections() ^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/AppAbfallplusDe.py", line 841, in get_collections r.raise_for_status() File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://app.abfallplus.de/struktur.xml.zip

Relevant Configuration

waste_collection_schedule:
    sources:
    - name: app_abfallplus_de
      args:
        app_id: de.k4systems.abfallappnf
        city: Leck
        strasse: alle Straßen

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[ ] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[ ] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

5ila5 commented 4 months ago

Looks like only de.k4systems.abfallappnf is affected (at least our test cases do return results)

@ReneNulschDE where did you find the user agents? By inspecting the traffic of an IOS App? When looking into the traffic of the de.k4systems.abfallappnf Android app, I see the user agent Mozilla/5.0 (Linux; Android 13; sdk_gphone_x86_64 Build/TE1A.220922.025; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/103.0.5060.71 Mobile Safari/537.36 for everything before the login request and Android / Abfall-AppNF 9.1.3 (1915091030) / DM=sdk_gphone_x86_64;DP=sdk_gphone_x86_64;DN=ranchu;DT=emu64xa;SN=google;SV=13 (33);MF=Google for the login request and everything after. But for some reason I get 401 UNAUTHORIZED for the first request (assistent/kommune/) for both of them

jwcappeln commented 3 months ago

Same problem here with the k4systems. Can i support any developer to fix it?

5ila5 commented 3 months ago

I found the error: The Sondermüll selection is the problem. The previous implementation would just add all possible waste types. de.k4systems.abfallappnf shows a lot of waste types below the Sondermüll headline. The API seems to not like that many collection types. I removed all collection types below a Sondermüll headline in #2120. Which fixes de.k4systems.abfallappnf but If other apps show a lot of waste types below another headline this fix would not apply, and we need to add these headlines to the ignore list. Our tests work (but they did before the fix as well) and there are just too many apps supported by this source to test all of them.

5ila5 commented 3 months ago

This will be part of release 1.49.0 or is available now as master (git) version (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)

mampfes / hacs_waste_collection_schedule