0013393: Missing Attachments Throws Exception

[nhalstead]

I get past the database sync, but when syncing attachments, it throws an exception and doesn't seem to sync any of the other files.

User Information:
===================

Using the Gramps Web Sync.

Error Details:
===================

36794434: WARNING: graphview.py: line 2519: Relationship loop detected
36865766: WARNING: graphview.py: line 2519: Relationship loop detected
37109541: WARNING: graphview.py: line 2519: Relationship loop detected
37323802: WARNING: graphview.py: line 2519: Relationship loop detected
37723364: WARNING: graphview.py: line 2519: Relationship loop detected
50039645: ERROR: grampsapp.py: line 188: Unhandled exception
Traceback (most recent call last):
  File "/Users/****/plugins/GrampsWebSync/grampswebsync.py", line 401, in _async_transfer_media
    self.handle_server_errors(self.download_files)
  File "/Users/****/plugins/GrampsWebSync/grampswebsync.py", line 409, in handle_server_errors
    return callback(*args)
           ^^^^^^^^^^^^^^^
  File "/Users/****/plugins/GrampsWebSync/grampswebsync.py", line 304, in download_files
    self.downloaded[gramps_id] = self._download_file(handle)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/****/plugins/GrampsWebSync/grampswebsync.py", line 325, in _download_file
    return self.api.download_media_file(handle=handle, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/****/plugins/GrampsWebSync/webapihandler.py", line 207, in download_media_file
    with open(path, "wb") as f:
         ^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/****/PXL_20201107_165755853.jpg'

System Information:
===================

Gramps version: 5.2.3
Python version: 3.11.9
BSDDB version: 6.2.9 (4, 8, 30)
sqlite version: 3.46.0 (2.6.0)
LANG: en_US.UTF-8
OS: Darwin
Distribution: 23.5.0

GTK version : 3.24.42
gobject version: 3.48.2
cairo version : (1, 26, 0)

[DavidMStraub]

Ok I suspect the problem is it tries to create the file but the subfolders don't exist. Can you please try if it syncs after you create the containing folder manually?

[nhalstead]

I should have clarified, my local db has references to files that are missing and I haven't fixed it. When it runs into the missing file that it was going to upload, it causes an exception. No such file or directory: '/Users/****/PXL_20201107_165755853.jpg'

Can this be updated to log a warning that the file was missing during upload, allowing the rest to continue?

[DavidMStraub]

No but the error is _download_media_file, so it tries to download a file from the server to your computer. And open(path, "wb") means it tries to write, not read, so it's not about a missing file but about a file that can't be created. Missing files don't raise exceptions.

[nhalstead]

Ok. I fixed the missing files and storage path.

Upon syncing the attachments with remote (upload all local attachments), it gives a 500.

Off the cuff without reading that code, does the desktop software really have to match the server paths? Can it just rewrite the file path and the plugin on the desktop rewrite the path to the local attachments directory?

Server Log Exception for the 500 response.

[2024-08-19 01:54:06 +0000] [16] [ERROR] Error handling request /api/media/e987d4fb86b1c2750cc0ac233ba/file?uploadmissing=1
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/local/lib/python3.11/dist-packages/gunicorn/workers/sync.py", line 178, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/auth.py", line 44, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 110, in view
    return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/views.py", line 191, in dispatch_request
    return current_app.ensure_sync(meth)(**kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/webargs/core.py", line 649, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/resources/file.py", line 107, in put
    media_handler.upload_file(f, checksum, mime, path=path)
  File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/media.py", line 126, in upload_file
    rel_path: FilenameOrPath = Path(path).relative_to(self.base_dir)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 731, in relative_to
    raise ValueError("{!r} is not in the subpath of {!r}"
ValueError: '/Users/nhalstead/Gramps/attachments/1604787625708-95408cd8-7442-4b20-a118-3632808135a2.jpg' is not in the subpath of '/app/media' OR one path is relative and the other is absolute.

gramps_webapi/api/resources/file.py

[nhalstead]

I was able to resolve the issue using the Media Manager and converting absolute paths to relative paths of the defined attachments folder in Gramps Preferences.