stashapp / CommunityScripts

This is a public repository containing plugin and utility scripts created by the Stash Community.
https://docs.stashapp.cc/add-ons/
GNU Affero General Public License v3.0
180 stars 140 forks source link

[FileMonitor] dying on ssl errors? #380

Closed MrRandoMan closed 1 week ago

MrRandoMan commented 1 month ago

Hi there. I think FileMonitor is having a hard time connecting to stash's with ssl enabled.

time="2024-07-31 03:43:17" level=error msg="[Plugin / FileMonitor] StdInRead={\"server_connection\":{\"Scheme\":\"https\",\"Host\":\"0.0.0.0\",\"Port\":443,\"SessionCookie\":{\"Name\":\"session\",\"Value\":\"MTcyMjQxMTc5NnxEdi1CQkFFQ180SUFBUkFCRUFBQVpfLUNBQUlHYzNSeWFXNW5EQWdBQm5WelpYSkpSQVp6ZEhKcGJtY01BZ0FBQm5OMGNtbHVad3dWQUJOMmFYTnBkR1ZrVUd4MVoybHVjMGh2YjJ0ekcxdGRjMlZ6YzJsdmJpNVdhWE5wZEdWa1VHeDFaMmx1U0c5dmFfLUZBZ0VDXzRZQUFmLUVBQUE1XzRNREFRRVJWbWx6YVhSbFpGQnNkV2RwYmtodmIyc0JfNFFBQVFJQkNGQnNkV2RwYmtsRUFRd0FBUWhJYjI5clZIbHdaUUVNQUFBQUJmLUdBZ0FBfPPtY4XH9U-TYzX7DWwLWknm22tX-tGHsxnlHkvqtrpZ\",\"Path\":\"\",\"Domain\":\"\",\"Expires\":\"0001-01-01T00:00:00Z\",\"RawExpires\":\"\",\"MaxAge\":0,\"Secure\":false,\"HttpOnly\":false,\"SameSite\":0,\"Raw\":\"\",\"Unparsed\":null},\"Dir\":\"C:\\progs\\stash\",\"PluginDir\":\"plugins\\community\\filemonitor\"},\"args\":{\"mode\":\"start_library_monitor\"}} (len(sys.argv)=1)" time="2024-07-31 03:43:17" level=error msg="[Plugin / FileMonitor] Could not connect to Stash at https://127.0.0.1:443/graphql" time="2024-07-31 03:43:17" level=error msg="[Plugin / FileMonitor] HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)')))"

DogmaDragon commented 1 month ago

@David-Maisonave mentioning the author

feederbox826 commented 1 month ago

seems to be upstream in stashapi as requests is unused @stg-annon

David-Maisonave commented 1 month ago

seems to be upstream in stashapi as requests is unused @stg-annon

I'm working on an updated version where I've already removed requests and the following unused imports:

import time
import shutil
import fileinput
import hashlib
import stashapi.log as log # Importing stashapi.log as log for critical events ONLY

But are you saying having the unused imports is causing the issue, or not calling graphql_request is the issue?

David-Maisonave commented 1 month ago

Hi there. I think FileMonitor is having a hard time connecting to stash's with ssl enabled.

feederbox826 commented 1 month ago

But are you saying having the unused imports is causing the issue, or not calling graphql_request is the issue?

neither is the issue, your http interface is through stashapi which still implements https verification. you'll have to rewrite it without stashapi and only requests if you want to solve it on your end

David-Maisonave commented 1 month ago

neither is the issue, your http interface is through stashapi which still implements https verification. you'll have to rewrite it without stashapi and only requests if you want to solve it on your end

Are you saying stashapi doesn't work with ssl enabled?

MrRandoMan commented 1 month ago

Nope, a hostname. Port 443

  • Are you calling FileMonitor as a plugin, or calling it in command line mode?

Plugin task

  • If calling in the command line mode, and if you're not using default URL, did you populate the filemonitor_config.py with the correct URL details?

N/A?

  • Can you please enable [Debug Tracing] in the UI, and get logging details from stash\plugins\FileMonitor\filemonitor.log?

Sure

(Note this is from stash.log as plugins\community\filemonitor\filemonitor.log is empty.)

``` time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Attempting to read stdin. (len(sys.argv)=1)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] StdInRead={\"server_connection\":{\"Scheme\":\"https\",\"Host\":\"0.0.0.0\",\"Port\":443,\"SessionCookie\":{\"Name\":\"session\",\"Value\":\"MTcyMjQ4OTgxN3xEdi1CQkFFQ180SUFBUkFCRUFBQVpfLUNBQUlHYzNSeWFXNW5EQWdBQm5WelpYSkpSQVp6ZEhKcGJtY01BZ0FBQm5OMGNtbHVad3dWQUJOMmFYTnBkR1ZrVUd4MVoybHVjMGh2YjJ0ekcxdGRjMlZ6YzJsdmJpNVdhWE5wZEdWa1VHeDFaMmx1U0c5dmFfLUZBZ0VDXzRZQUFmLUVBQUE1XzRNREFRRVJWbWx6YVhSbFpGQnNkV2RwYmtodmIyc0JfNFFBQVFJQkNGQnNkV2RwYmtsRUFRd0FBUWhJYjI5clZIbHdaUUVNQUFBQUJmLUdBZ0FBfGelb6CvylFt152Cf1k7pPTf6EPdHkWW7dfLKYOpWQFg\",\"Path\":\"\",\"Domain\":\"\",\"Expires\":\"0001-01-01T00:00:00Z\",\"RawExpires\":\"\",\"MaxAge\":0,\"Secure\":false,\"HttpOnly\":false,\"SameSite\":0,\"Raw\":\"\",\"Unparsed\":null},\"Dir\":\"C:\\\\progs\\\\stash\",\"PluginDir\":\"plugins\\\\community\\\\filemonitor\"},\"args\":{\"mode\":\"start_library_monitor\"}} (len(sys.argv)=1)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Could not connect to Stash at https://127.0.0.1:443/graphql" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)')))" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Traceback (most recent call last):" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connectionpool.py\", line 466, in _make_request" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] self._validate_conn(conn)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connectionpool.py\", line 1095, in _validate_conn" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] conn.connect()" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connection.py\", line 652, in connect" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] sock_and_verified = _ssl_wrap_socket_and_match_hostname(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connection.py\", line 805, in _ssl_wrap_socket_and_match_hostname" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ssl_sock = ssl_wrap_socket(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\util\\ssl_.py\", line 465, in ssl_wrap_socket" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\util\\ssl_.py\", line 509, in _ssl_wrap_socket_impl" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] return ssl_context.wrap_socket(sock, server_hostname=server_hostname)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\ssl.py\", line 455, in wrap_socket" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] return self.sslsocket_class._create(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\ssl.py\", line 1042, in _create" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] self.do_handshake()" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\ssl.py\", line 1320, in do_handshake" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] self._sslobj.do_handshake()" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] During handling of the above exception, another exception occurred:" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Traceback (most recent call last):" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connectionpool.py\", line 789, in urlopen" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] response = self._make_request(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connectionpool.py\", line 490, in _make_request" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] raise new_e" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] The above exception was the direct cause of the following exception:" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Traceback (most recent call last):" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\requests\\adapters.py\", line 667, in send" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] resp = conn.urlopen(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\connectionpool.py\", line 843, in urlopen" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] retries = retries.increment(" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\urllib3\\util\\retry.py\", line 519, in increment" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)')))" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] During handling of the above exception, another exception occurred:" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] Traceback (most recent call last):" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\stash\\plugins\\community\\filemonitor\\filemonitor.py\", line 98, in " time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] stash = StashInterface(FRAGMENT_SERVER)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\stashapi\\stashapp.py\", line 49, in __init__" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] self.version = self.stash_version()" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\stashapi\\stashapp.py\", line 219, in stash_version" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] result = self.call_GQL(\"query StashVersion{ version { build_time hash version } }\")" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\stashapi\\stashapp.py\", line 216, in call_GQL" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] return self._GQL(query, variables)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\stashapi\\classes.py\", line 224, in _GQL" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] response = self.s.post(self.url, json=json_request)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\requests\\sessions.py\", line 637, in post" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] return self.request(\"POST\", url, data=data, json=json, **kwargs)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\requests\\sessions.py\", line 589, in request" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] resp = self.send(prep, **send_kwargs)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\requests\\sessions.py\", line 703, in send" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] r = adapter.send(request, **kwargs)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] File \"C:\\progs\\_python\\Lib\\site-packages\\requests\\adapters.py\", line 698, in send" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] raise SSLError(e, request=request)" time="2024-08-01 01:23:44" level=error msg="[Plugin / FileMonitor] requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /graphql (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1000)')))" time="2024-08-01 01:23:44" level=error msg="Plugin returned error: exit status 1" ```

If not smart, is it possible to disable FM's cert verification as a workaround?

stg-annon commented 1 month ago

seems to be upstream in stashapi as requests is unused @stg-annon

you would need to change the ssl verify flag in the session to ignore self signed certs, stash.s.verify = False

stg-annon commented 1 month ago

neither is the issue, your http interface is through stashapi which still implements https verification. you'll have to rewrite it without stashapi and only requests if you want to solve it on your end

Are you saying stashapi doesn't work with ssl enabled?

it does this individual just has a self signed cert for SSL so not a verified cert from a certificate authority think this warning image

you can disable it with a flag

MrRandoMan commented 1 month ago

For what its worth if I recall correctly I needed a host name cert to make upnp\dlna\android\vlc a lot less ornery. Considering stash's use case, I think most user ssl certs are going to be self signed anyhow, eh?

Where's the best spot to set stash.s.verify ?

stg-annon commented 1 month ago

For what its worth if I recall correctly I needed a host name cert to make upnp\dlna\android\vlc a lot less ornery. Considering stash's use case, I think most user ssl certs are going to be self signed anyhow, eh?

there are various ways to get a signed cert for things like this, I would say the majority of the users are just using http as the app is intended to be internal

Where's the best spot to set stash.s.verify ?

it would need to be early on, now that I think about it it may need to be in the beginning of the init as stashapi now does a couple of calls during the init process, I'll add a flag for it there

published with v0.2.48

MrRandoMan commented 1 month ago

I think I figured out how to pull down your update and patch FM.... And yeah I think it works now. All the https warnings, but that is "desirable".

Thanks for the workaround!

That said I found another plugin that I think has the same issue. (Tho I don't think they're using stashapi)

stg-annon commented 1 month ago

That said I found another plugin that I think has the same issue. (Tho I don't think they're using stashapi)

its quite possible requests is quite popular and the verify flag is by default True so unless they also specify it it will likely have the same problem

David-Maisonave commented 1 month ago

For what its worth if I recall correctly I needed a host name cert to make upnp\dlna\android\vlc a lot less ornery. Considering stash's use case, I think most user ssl certs are going to be self signed anyhow, eh?

IMHO, most users are not going to use self signed cert, because most users are not going to use ssl for a local setup.

If you don't mine me asking, what's the use-case for using ssl on Stash (with self signed cert) in a local setup?

If there's a good common use-case, I could look into adding an option in FileMonitor to handle this, but if this is just a one off, it's bettered handled by the end user.

feederbox826 commented 1 month ago

published with v0.2.48

already resolved upstream

David-Maisonave commented 1 month ago

(Note this is from stash.log as plugins\community\filemonitor\filemonitor.log is empty.) I just installed clean version of FileMonitor from CommunityScripts, and the logging works. The file gets created and populated. Even with [Debug Tracing] disabled, some minimal logging occurs.

Nothing gets log to stash.log from FileMonitor. That's to avoid cluttering stash.log, which could happen if the user dumps thousands of files in a monitored path. I did mean to log exceptions in stash.log, but I see I forgot to add that code. Will add it on the next version.

I'm really concerned that you're not getting any logging in filemonitor.log, and I like to see if we can figure out why.

This version of FileMonitor is a beta version, and I'm already working on some improvements. Because of this logging issue, I'm going to add a UI option that when enabled, will send info logging to stash.log.

David-Maisonave commented 1 month ago

already resolved upstream

If resolved in v0.2.48, should I include that as minimum version in the requirements.txt file?

stashapp-tools >= 0.2.48
pyYAML
watchdog
MrRandoMan commented 1 month ago

If you don't mine me asking, what's the use-case for using ssl on Stash (with self signed cert) in a local setup?

Well this network isn't "isolated" so I wanted to to use ssl. IIRC there was a bunch a weird android stuff (browsers and/or vlc) that was just hardcoded to ignore/block localhost certs if they weren't from 127.1. I would bet big that most users who wanted this, just gave up instead of generating a hostname cert.

I'm really concerned that you're not getting any logging in filemonitor.log, and I like to see if we can figure out why.

it's logging now, now that FM is running.

If I undo my "patch"

stash = StashInterface(FRAGMENT_SERVER, verify_ssl=False)

The exceptions are thrown back to stash's log. IMHO I think this is because of the type of fatal error this is. I thiiiik stash runs plugins in a separate process? So this is all StdError output I think, and FM is bailing too early to have ever written to its own debug log.

Maybe... shrugs

David-Maisonave commented 1 month ago

it's logging now, now that FM is running. and FM is bailing too early to have ever written to its own debug log.

Thanks for the update.

David-Maisonave commented 1 week ago

Hi MrRandoMan, If this issue has been addressed, can you please close it? 😃

Thanks