[Bug]: City of Oklahoma City okc_gov fetch failed

joshnewman17 commented 3 weeks ago

I Have A Problem With:

A specific source

What's Your Problem

Was excited to see my city is supported but unfortunately it looks like it's currently not functioning for me. Attempting initial setup and receiving the error in the attached logs.

Source (if relevant)

okc_gov

Logs

fetch failed for source City of Oklahoma City: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/okc_gov.py", line 43, in fetch json_data = json.loads(r.text) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/__init__.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py", line 145, in fetch entries = self._source.fetch() ^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/okc_gov.py", line 45, in fetch raise Exception("Invalid response returned from data.okc.gov") from e Exception: Invalid response returned from data.okc.gov

Relevant Configuration

waste_collection_schedule:
    sources:
    - name: okc_gov
      args:
        objectID: "1111111"

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[X] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[ ] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

joshnewman17 commented 3 weeks ago

Update - I confirmed searching my objectID on the site is functional, but it looks like they're blocking downloading the data, so if this is how it gets captured it seems like this is likely a change on their side that broke it.

5ila5 commented 3 weeks ago

We get the data from https://data.okc.gov/services/portal/api/data/records/Address%20Trash%20Services

but it looks like they now use Incapsula to prevent scraping, so I'm not sure if we can fix this

sanctas commented 2 weeks ago

We get the data from https://data.okc.gov/services/portal/api/data/records/Address%20Trash%20Services

but it looks like they now use Incapsula to prevent scraping, so I'm not sure if we can fix this

I can host a puppeteer server to create a drop in replacement URL that bypasses Incapsula.

If I do this, can it be added to the repo?

5ila5 commented 2 weeks ago

Yes this would probably be fine, But it does work again for me, and I do not get an Incapsula message anymore

sanctas commented 2 weeks ago

Yes this would probably be fine, But it does work again for me, and I do not get an Incapsula message anymore

Mine hasn't worked for months. I just assumed it was a "me" thing and didn't troubleshoot further. I was updating my dashboard this week and saw this issue on GitHub and realized it wasn't just me.

Anyway, I made a drop-in replacement for the URL: https://okc.schizo.dev/trash

Just add that line into the code.

I've never made a pull request so I'm not really sure how to do it, to be honest.

5ila5 commented 2 weeks ago

Thanks for that. The old one seems to work again, so I would keep this and replace the url if this happens again

sanctas commented 2 weeks ago

Thanks for that.

The old one seems to work again, so I would keep this and replace the url if this happens again

Official API still doesn't work for me, FYI.

May be a regional thing or something, not sure. You could maybe add it as a fallback if the main URL fails.

5ila5 commented 2 weeks ago

You're right now I have the problem again, not sure when this triggers but

curl "https://okc.schizo.dev/trash?objectID=1935340"
# returns

{"status":"error","code":502,"message":"Application failed to respond","request_id":"W7Dgxx7_QeqtNh5Uwp2dFg_3500563535"}%

sanctas commented 2 weeks ago

Yeah, I slapped this code together in like 10 minutes. I need to add some error handling. I'll restart the server.

But according to the okc_gov.py the parameter should be recordID, not objectID.

5ila5 commented 2 weeks ago

This source now uses the unofficial server by @sanctas

This will be part of the next release or is available now as master (git) version

install master/git: Developer tools -> actions -> select update: Install update -> click Coose entity and select WCS -> select version and type master in the input field -> run the action -> restart HA

install master with HACS version < 2.0: (3-dot menu -> redownload -> change version to master -> fully restart HA (this will increase update cycle by a lot)

mampfes / hacs_waste_collection_schedule