Closed LibertX closed 1 year ago
Does the script now work again? I assume that post.token
was not present anymore on szuru, so the similarity check couldn't be done.
Unfortunately no, I still get the same error on 0.8.0:
[CRITICAL] [08.05.2023, 09:03:51 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
File "/usr/local/bin/upload-media", line 6, in <module>
sys.exit(main())
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 301, in main
upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 237, in upload_post
post.exact_post, similar_posts = check_similarity(szuru, post.token)
TypeError: cannot unpack non-iterable NoneType object
Have you tried restarting the szurubooru-server before?
In src/szurubooru_toolkit/scripts/upload_media.py
you can add on line 96 response.json()
to check the response from the server.
I just restarted szurubooru-server, no change.
response.json() throwed a decode error:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/upload-media", line 6, in <module>
sys.exit(main())
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 302, in main
upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 238, in upload_post
post.exact_post, similar_posts = check_similarity(szuru, post.token)
File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 100, in check_similarity
logger.critical(response.json())
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
root@szurubooru-toolkit-869966577b-5bbnd:/szurubooru-toolkit# vim src/szurubooru_toolkit/scripts/upload_media.py
root@szurubooru-toolkit-869966577b-5bbnd:/szurubooru-toolkit# upload-media
I just printed "response", looks like the server returns error 500:
[CRITICAL] [09.05.2023, 09:50:39 UTC] [upload_media.check_similarity]: <Response [500]>
[CRITICAL] [09.05.2023, 09:50:39 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
The server throws the following error:
[2023-05-09 09:58:53] waitress Exception while serving /posts/reverse-search
Traceback (most recent call last):
File "/usr/lib/python3.8/site-packages/waitress/channel.py", line 350, in service
task.service()
File "/usr/lib/python3.8/site-packages/waitress/task.py", line 171, in service
self.execute()
File "/usr/lib/python3.8/site-packages/waitress/task.py", line 441, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/opt/app/szurubooru/rest/app.py", line 104, in application
response = handler(ctx, match.groupdict())
File "/opt/app/szurubooru/api/post_api.py", line 301, in get_posts_by_image
ctx, posts.search_by_image_exact(content)
File "/opt/app/szurubooru/func/posts.py", line 924, in search_by_image_exact
db.session.query(model.Post)
File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3467, in one_or_none
raise orm_exc.MultipleResultsFound(
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none()
It looks like a server issue. Probably: https://github.com/rr-/szurubooru/issues/539
That being said, as a workaround, would it be possible to catch the exception on upload_post and cancel upload with warning if check_similarity fails ?
Good to know. I pushed a quick fix to main. Can you test if it works now?
Sure, however I think it should abort upload entirely, instead of just skipping the similarity check, because it'll probably reupload the file each time it runs.
https://github.com/reluce/szurubooru-toolkit/commit/26bfa147fa956477d287b2db046ee317dd634744
Maybe return True, []
instead ?
I tested with return True, []
, it works fine now.
Fair point, but if you enabled that source files should get deleted, you'll esentially lose your data as it's neither local nor uploaded.
I just pushed a commit where it uses an error variable: If the similarity check couldn't be done, it will skip the upload and won't delete the file (if cleanup is set). If you still get the error from szuru it would be great if you could test it.
Indeed, nice catch.
The last patch works fine, I did not test the cleanup part tho because I don't want to remove my original pictures.
Hi,
Do you think you could make a release with those changes ? Patching the container is not ideal, especially in Kubernetes environments.
I just pushed 0.9.0. I was a bit hesistant as I did a lot of refactoring and Python 3.11 is now required. Let me know if everything still works.
Hi,
Did some tests, so far so good ! :) I've not encountered any crash.
Thanks
Hello,
I have a (big) library of pictures that I'd like to upload (around 11k items). It uploaded quite a lot, but now when it starts, it fails with the following error (at 483/11566):
Thanks!