Gotify integration broken since release v2.3-ls139

pjbroad commented 2 years ago

Expected Behavior

I've been using the Gotify integration for a while. You can test and integration using the integration tab of a project. This succeeds with versions prior to v2.3-ls139,

Current Behavior

With version v2.3-ls139 and v2.3-ls140, the Gotify integration always fails. For my test project, I get the message "Could not send a test notification. Connection failed." There is nothing of note in the logs, not even with Debug enabled.

Steps to Reproduce

Setup a Gotify integration for a project.
Use the "test" button on the integration tab.

Environment

OS: Ubuntu 22.04 LTS CPU architecture: x86_64 How docker service was installed: OS repo via apt

Command used to create docker container (run/create/compose/screenshot)

----> section from docker-compose.yml image: lscr.io/linuxserver/healthchecks container_name: healthchecks environment:

PUID=${STNDUSERID}
PGID=${STNDGROUPID}
SITE_ROOT=https://healthchecks.${LOCALDOMAIN}:${LOCALPORT}
SITE_NAME=Home Healthchecks
DEFAULT_FROM_EMAIL=hc@${HOMEDOMAIN}
SUPERUSER_EMAIL=${HEALTHCHECKSSUEMAIL}
SUPERUSER_PASSWORD=${HEALTHCHECKSSUPASS}
REGENERATE_SETTINGS=True
DEBUG=True volumes:
./healthchecks/config:/config restart: unless-stopped

Docker logs

[pid: 175|app: 0|req: 2/45] 172.22.0.8 () {58 vars in 1239 bytes} [Sat Oct 22 15:55:24 2022] POST /integrations/0e2f3764-d07b-4b69-bddf-a94d6549cedd/test/ => generated 0 bytes in 497 msecs (HTTP/1.0 302) 10 headers in 554 bytes (1 switches on core 0) [pid: 175|app: 0|req: 3/46] 172.22.0.8 () {52 vars in 1289 bytes} [Sat Oct 22 15:55:25 2022] GET /projects/8a287f56-2ca5-4f30-a8bb-e53a9bfdab60/integrations/ => generated 24915 bytes in 53 msecs (HTTP/1.0 200) 10 headers in 499 bytes (1 switches on core 0)

Other Comments

I'd be very happy to debug if you have something I could do to help.

github-actions[bot] commented 2 years ago

Thanks for opening your first issue here! Be sure to follow the bug or feature issue templates!

pjbroad commented 2 years ago

I should add that sending a message to my Gotify instance from within the healthcheck container, works: curl -X POST "https://<my gotify domain>/message?token=<token>" -F "title=my title" -F "message=my message"

NiNiyas commented 2 years ago

Also running into the same issue when using webhooks. Downgrading to v2.3-ls138 works.

thespad commented 2 years ago

Could be the change from requests to pycurl in 2.4.0, I'll do some tests

thespad commented 2 years ago

I can reliably replicate the problem, but the logging is rubbish and using pycurl directly inside the container to interface with my local gotify server seems to work fine, so I've not really got any idea how to go about troubleshooting this.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

pjbroad commented 1 year ago

I can confirm that the latest release, v2.4.1-ls145, still has this problem.

nigelm commented 1 year ago

Same issue with pushover notification and ntfy (only available in very recent versions). Still showing in v2.5-ls146

No useful log data :-(

thespad commented 1 year ago

Can you try doing an apk add py3-curl inside the container and see if it makes any difference? I'm wondering if there's a dependency missing somewhere that pip isn't installing.

nigelm commented 1 year ago

Tried that - no difference to running system...

However that could be as the processes are running (everything already loaded), and its tricky to restart them in the container without the container closing down...

Looking in the container - fresh start - the path /usr/lib/python3.10/site-packages/curl/__init__.py which should be the pycurl code, does exist and looks like the content on github

thespad commented 1 year ago

Can you try this PR image: lspipepr/healthchecks:v2.5-pkg-1eaf64f2-pr-101 and see if it helps.

nigelm commented 1 year ago

Can you try this PR image: lspipepr/healthchecks:v2.5-pkg-1eaf64f2-pr-101 and see if it helps.

That image behaves the same as mainline 2.5 image - an attempt to test the pushover integration gives a banner with

Could not send a test notification. Connection failed.

nigelm commented 1 year ago

Just as a check I have tested using a fresh test config (exactly as per the sample other than adding the Pushover config) using the official healthchecks/healthchecks image latest version.

This does successfully send a pushover notification.

nigelm commented 1 year ago

And I have a culprit...

As of version 2.4 healthchecks blocks all connections to private IP ranges including if you are using a http proxy in private IP address space unless INTEGRATIONS_ALLOW_PRIVATE_IPS is set true.

See

I added

INTEGRATIONS_ALLOW_PRIVATE_IPS = True

to config/local_settings.py (likely could have gone in docker-compose.yaml but hadn't checked how much this image picks up off the environment). With this change the current latest image works fine.

Our installation, like many corporate docker installs does not have open access to internet address space so setting up proxy settings is routine, and I had overlooked this as it used to just work.

thespad commented 1 year ago

That's such a weird random thing to start blocking by default without calling it out in the release notes anywhere.

I'll get the PR and readme docs updated to account for it.

thespad commented 1 year ago

@nigelm OK can you test with lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101, it should now support setting INTEGRATIONS_ALLOW_PRIVATE_IPS as an environment variable that should be honoured (I've also updated the readme to call it out, given the issues it can cause if not set).

Obviously it'll need to not be already set in the local_settings.py - the way we've set this container up has made a bit of a rod for our own backs but changing it at this point would probably break more people's installs than it would help.

nigelm commented 1 year ago

Can @pjbroad and @NiNiyas check whether INTEGRATIONS_ALLOW_PRIVATE_IPS = True fixes things for them using lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101. It seems unlikely that I have usurped their issue for something completely different but worth checking.

nigelm commented 1 year ago

@nigelm OK can you test with lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101, it should now support setting INTEGRATIONS_ALLOW_PRIVATE_IPS as an environment variable that should be honoured (I've also updated the readme to call it out, given the issues it can cause if not set).

Tested with modifications to local_settings.py commented out, and INTEGRATIONS_ALLOW_PRIVATE_IPS=True set in the docker-compose environment section.

Test notification via Pushover worked.

NiNiyas commented 1 year ago

Can @pjbroad and @NiNiyas check whether INTEGRATIONS_ALLOW_PRIVATE_IPS = True fixes things for them using lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101. It seems unlikely that I have usurped their issue for something completely different but worth checking.

I can confirm that this fixes webhooks. Tested on latest tag right now. I can check lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101 in a few hours.

Edit: Webhooks are working in this image lspipepr/healthchecks:v2.5-pkg-e78fe047-pr-101

pjbroad commented 1 year ago

I can also confirm that adding "INTEGRATIONS_ALLOW_PRIVATE_IPS = True" to config/local_settings.py, or to the docker environment, for the latest version fixes the reported issue. Great work @nigelm, thank you.

thespad commented 1 year ago

Thanks for testing guys. PR merged and should be built within the hour.

linuxserver / docker-healthchecks