Issue with upload_media feature

[LibertX]

Hello,

I have a (big) library of pictures that I'd like to upload (around 11k items). It uploaded quite a lot, but now when it starts, it fails with the following error (at 483/11566):

[CRITICAL] [27.04.2023, 13:03:41 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/bin/upload-media", line 6, in <module>
    sys.exit(main())
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 300, in main
    upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 237, in upload_post
    post.exact_post, similar_posts = check_similarity(szuru, post.token)
TypeError: cannot unpack non-iterable NoneType object

Thanks!

[reluce]

Does the script now work again? I assume that post.token was not present anymore on szuru, so the similarity check couldn't be done.

[LibertX]

Unfortunately no, I still get the same error on 0.8.0:

[CRITICAL] [08.05.2023, 09:03:51 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/bin/upload-media", line 6, in <module>
    sys.exit(main())
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 301, in main
    upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 237, in upload_post
    post.exact_post, similar_posts = check_similarity(szuru, post.token)
TypeError: cannot unpack non-iterable NoneType object

[reluce]

Have you tried restarting the szurubooru-server before?

In src/szurubooru_toolkit/scripts/upload_media.py you can add on line 96 response.json() to check the response from the server.

[LibertX]

I just restarted szurubooru-server, no change.

response.json() throwed a decode error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/upload-media", line 6, in <module>
    sys.exit(main())
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 302, in main
    upload_post(file, file_ext=Path(file_to_upload).suffix[1:], file_to_upload=file_to_upload)
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 238, in upload_post
    post.exact_post, similar_posts = check_similarity(szuru, post.token)
  File "/szurubooru-toolkit/src/szurubooru_toolkit/scripts/upload_media.py", line 100, in check_similarity
    logger.critical(response.json())
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
root@szurubooru-toolkit-869966577b-5bbnd:/szurubooru-toolkit# vim src/szurubooru_toolkit/scripts/upload_media.py
root@szurubooru-toolkit-869966577b-5bbnd:/szurubooru-toolkit# upload-media

I just printed "response", looks like the server returns error 500:

[CRITICAL] [09.05.2023, 09:50:39 UTC] [upload_media.check_similarity]: <Response [500]>
[CRITICAL] [09.05.2023, 09:50:39 UTC] [upload_media.check_similarity]: An error occured during the similarity check: Expecting value: line 1 column 1 (char 0)

[LibertX]

The server throws the following error:

[2023-05-09 09:58:53] waitress Exception while serving /posts/reverse-search
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/waitress/channel.py", line 350, in service
    task.service()
  File "/usr/lib/python3.8/site-packages/waitress/task.py", line 171, in service
    self.execute()
  File "/usr/lib/python3.8/site-packages/waitress/task.py", line 441, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/opt/app/szurubooru/rest/app.py", line 104, in application
    response = handler(ctx, match.groupdict())
  File "/opt/app/szurubooru/api/post_api.py", line 301, in get_posts_by_image
    ctx, posts.search_by_image_exact(content)
  File "/opt/app/szurubooru/func/posts.py", line 924, in search_by_image_exact
    db.session.query(model.Post)
  File "/usr/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 3467, in one_or_none
    raise orm_exc.MultipleResultsFound(
sqlalchemy.orm.exc.MultipleResultsFound: Multiple rows were found for one_or_none()

It looks like a server issue. Probably: https://github.com/rr-/szurubooru/issues/539

[LibertX]

That being said, as a workaround, would it be possible to catch the exception on upload_post and cancel upload with warning if check_similarity fails ?

[reluce]

Good to know. I pushed a quick fix to main. Can you test if it works now?

[LibertX]

Sure, however I think it should abort upload entirely, instead of just skipping the similarity check, because it'll probably reupload the file each time it runs.

https://github.com/reluce/szurubooru-toolkit/commit/26bfa147fa956477d287b2db046ee317dd634744 Maybe return True, [] instead ?

[LibertX]

I tested with return True, [], it works fine now.

[reluce]

Fair point, but if you enabled that source files should get deleted, you'll esentially lose your data as it's neither local nor uploaded.

I just pushed a commit where it uses an error variable: If the similarity check couldn't be done, it will skip the upload and won't delete the file (if cleanup is set). If you still get the error from szuru it would be great if you could test it.

[LibertX]

Indeed, nice catch.

The last patch works fine, I did not test the cleanup part tho because I don't want to remove my original pictures.

[LibertX]

Hi,

Do you think you could make a release with those changes ? Patching the container is not ideal, especially in Kubernetes environments.

[reluce]

I just pushed 0.9.0. I was a bit hesistant as I did a lot of refactoring and Python 3.11 is now required. Let me know if everything still works.

[LibertX]

Hi,

Did some tests, so far so good ! :) I've not encountered any crash.

Thanks