[Bug]: Germany: muellmax RSAG gets no ics file, missing last step, missing args in the request

IamNeo1 commented 12 months ago

I Have A Problem With:

The integration in general

What's Your Problem

The provider Müllmax.de with the service RASG "Rsa" has added another step to create an ICS calendar. After selecting your address, you now also have to select the types of waste collection. If you do not request the ICS file without selection. Triggering the submit without selection throws a validation error on the web pages. HA throws a parser error with the current config, because the parser expects an ICS and probably receives a web page or nothing at all. The entity shows status OFF and the calender is empty.

Müllmax RSAG: https://www.muellmax.de/abfallkalender/rsa/res/RsaStart.php

The multiple selection of garbage types must therefore be added to the wizard. All arguments with mm_frm_fra_ must also be included in the request in a further step.

When analyzing the request with all types selected, you see the following arguments in the payload:

mm_ses: XXX xxx: 1 mm_frm_type: termine mm_frm_fra_RM2T: RM2T mm_frm_fra_RM4T: RM4T mm_frm_fra_BIO1T: BIO1T mm_frm_fra_BIOT: BIOT mm_frm_fra_PAT: PAT mm_frm_fra_WET: WET mm_frm_fra_CHR: CHR mm_frm_fra_RM1B: RM1B mm_frm_fra_RM2B: RM2B mm_frm_fra_RM4B: RM4B mm_frm_fra_PA2B: PA2B mm_frm_fra_PA4B: PA4B mm_frm_fra_BIO1B: BIO1B mm_frm_fra_BIOB: BIOB mm_frm_fra_WEB: WEB mm_frm_fra_WE4B: WE4B mm_ica_gen: iCalendar-Datei laden

Error Log: HA is added

Source (if relevant)

No response

Logs

Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
Integration: waste_collection_schedule (documentation)
First occurred: 13:30:41 (1 occurrences)
Last logged: 13:30:41
fetch failed for source Müllmax: 
Traceback (most recent call last): 
File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 340, 
in parts validate_token(name) 
File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 128, 
in validate_token raise ValueError(name) 
ValueError: 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> 
During handling of the above exception, another exception occurred: 
Traceback (most recent call last): 
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, 
in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ 
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/muellmax_de.py", line 148, 
in fetch dates = self._ics.convert(r.text) ^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/ICS.py", line 48, 
in convert events: List[Any] = icalevents.events( ^^^^^^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/site-packages/icalevents/icalevents.py", line 57, 
in events found_events += parse_events( ^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/site-packages/icalevents/icalparser.py", line 288, 
in parse_events calendar = Calendar.from_ical(content) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/site-packages/icalendar/cal.py", line 330, 
in from_ical name, params, vals = line.parts() ^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 352, 
in parts raise ValueError( 
ValueError: Content line could not be parsed into parts: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">': <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

Logger: waste_collection_schedule.source_shell
Source: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:136
Integration: waste_collection_schedule ([documentation](https://github.com/mampfes/hacs_waste_collection_schedule#readme))
First occurred: 8. Dezember 2023 um 15:05:22 (3 occurrences)
Last logged: 01:21:26
fetch failed for source Müllmax: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 340, in parts validate_token(name) File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 128, in validate_token raise ValueError(name) ValueError: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 134, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/muellmax_de.py", line 148, in fetch dates = self._ics.convert(r.text) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/ICS.py", line 48, in convert events: List[Any] = icalevents.events( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/icalevents/icalevents.py", line 57, in events found_events += parse_events( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/icalevents/icalparser.py", line 288, in parse_events calendar = Calendar.from_ical(content) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/icalendar/cal.py", line 330, in from_ical name, params, vals = line.parts() ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/icalendar/parser.py", line 352, in parts raise ValueError( ValueError: Content line could not be parsed into parts: '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">': <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

Relevant Configuration

waste_collection_schedule:
  sources:
    - name: muellmax_de
      args:
        service: Rsa
        mm_frm_ort_sel: stadt
        mm_frm_str_sel: strasse

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[X] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[X] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

IamNeo1 commented 12 months ago

It was obviously a problem with the website, which occurs irregularly, the website responds very slowly and when starting HA nothing came back several times. After 4 restarts HA has received the ICS. With all kinds of garbage dates. A query of the wished types in the wizard and args in the config would be much better.

I will open a request for this.

IamNeo1 commented 11 months ago

The error occurs again and again. Each time the calendar is empty afterwards. So this extension cannot be used.

Isengo1989 commented 10 months ago

Any news on that one?

GitPat2 commented 4 months ago

Does not work here either :-(

BergCyrill commented 3 months ago

Since I did have the same issue: The PR #2471 fixed the issue above for me. Currently not released but I have copied the content from muellmax_de.py

Update: the changes from above PR are included in Release 2.1.0 now

GitPat2 commented 3 months ago

2.1.0 does not work for me: "Die Quelle hat eine ungültige Antwort zurückgegeben: "Got invalid response from the server, please recheck your arguments". Bitte überprüfe die Argumente und versuche es erneut."

aschmitz-GH commented 3 months ago

same for me, also the muellmax_de.py does not work (see other issue)

lxffm commented 2 months ago

Same for me, I can't actually configure the integration at all.

Die Quelle hat eine ungültige Antwort zurückgegeben: "Got invalid response from the server, please recheck your arguments". Bitte überprüfe die Argumente und versuche es erneut.

Where do I actually get the value for service: from? I picked Germany / Maintal from the setup GUI, and it put "Mai" in the service field. I don't have a clue, whether this is the issue or whether the muellmax_de Service is not responding correctly.

hendrik001973 commented 1 month ago

Same here. I picked RSAG from Muellmax_de.

aschmitz-GH commented 1 month ago

Don’t know if it already is updated in HA Integration. Short after I posted the issue, it was fixed in GIT, but not updated in the HA Integration. So I downloaded the new Version of muellmax_de.py from GIT and exchanged it in HA, it worked fine.

Just go through the files in GIT (same structure as in HA) and download the muellmax_de.py. Then open the file Editor in HA, navigate to the file and replace it with the new one. Restart and try again.

hendrik001973 commented 1 month ago

After I downloaded the new Version of muellmax_de.py from GIT and exchanged it in HA, it worked fine.

Where do I find it? Thanks in advance!

aschmitz-GH commented 1 month ago

After I downloaded the new Version of muellmax_de.py from GIT and exchanged it in HA, it worked fine.

Where do I find it? Thanks in advance!

i updated my comment. You should find it here: https://github.com/mampfes/hacs_waste_collection_schedule/tree/master/custom_components/waste_collection_schedule/waste_collection_schedule/wizard

hendrik001973 commented 1 month ago

I already have the new version of this file, but it seems, that it only works on a small number of tries

mampfes / hacs_waste_collection_schedule