search

gramps-project / gramps-web-api

A RESTful web API for Gramps
GNU Affero General Public License v3.0
76 stars 40 forks source link

Deleting media object #527

Open gnufxela opened 1 month ago

gnufxela commented 1 month ago

Tried to delete a media object(through browser). It seems to be deleted when looking at it through front end. I am running grampsweb docker container. However docker logs shows this error. (I deleted 2 objects). 

Gramps 5.2.2
Gramps Web API 2.3.1
Gramps Web Frontend 24.7.0
Gramps QL 0.3.0
locale: en
multi-tree: false
task queue: true
grampsweb  | Exception possibly due to cache backend.
grampsweb  | Traceback (most recent call last):
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 536, in make_cache_key_thumbnails
grampsweb  |     obj = db_handle.get_media_from_handle(handle)
grampsweb  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps/gen/db/generic.py", line 1368, in get_media_from_handle
grampsweb  |     return self._get_from_handle(MEDIA_KEY, Media, handle)
grampsweb  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps/gen/db/generic.py", line 1341, in _get_from_handle
grampsweb  |     raise HandleError("Handle %s not found" % handle)
grampsweb  | gramps.gen.errors.HandleError: Handle fa86664c5dc374901c1be088e98 not found
grampsweb  | 
grampsweb  | During handling of the above exception, another exception occurred:
grampsweb  | 
grampsweb  | Traceback (most recent call last):
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/flask_caching/__init__.py", line 374, in decorated_function
grampsweb  |     cache_key = make_cache_key(*args, **kwargs)
grampsweb  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 538, in make_cache_key_thumbnails
grampsweb  |     abort_with_message(404, f"Handle {handle} not found")
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 655, in abort_with_message
grampsweb  |     raise exc
grampsweb  | werkzeug.exceptions.HTTPException: 404 Not Found: Handle fa86664c5dc374901c1be088e98 not found
grampsweb  | Exception possibly due to cache backend.
grampsweb  | Traceback (most recent call last):
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 536, in make_cache_key_thumbnails
grampsweb  |     obj = db_handle.get_media_from_handle(handle)
grampsweb  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps/gen/db/generic.py", line 1368, in get_media_from_handle
grampsweb  |     return self._get_from_handle(MEDIA_KEY, Media, handle)
grampsweb  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps/gen/db/generic.py", line 1341, in _get_from_handle
grampsweb  |     raise HandleError("Handle %s not found" % handle)
grampsweb  | gramps.gen.errors.HandleError: Handle fa871f92b6761f31478482573be not found
grampsweb  | 
grampsweb  | During handling of the above exception, another exception occurred:
grampsweb  | 
grampsweb  | Traceback (most recent call last):
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/flask_caching/__init__.py", line 374, in decorated_function
grampsweb  |     cache_key = make_cache_key(*args, **kwargs)
grampsweb  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 538, in make_cache_key_thumbnails
grampsweb  |     abort_with_message(404, f"Handle {handle} not found")
grampsweb  |   File "/usr/local/lib/python3.11/dist-packages/gramps_webapi/api/util.py", line 655, in abort_with_message
grampsweb  |     raise exc
grampsweb  | werkzeug.exceptions.HTTPException: 404 Not Found: Handle fa871f92b6761f31478482573be not found

Let me tack on 1 other issues I am facing that I might be potentially related(maybe not too):

  1. I also tried to set up S3 storage (on non AWS platform), I am not sure it is set up successfully. I was expecting that when I upload a jpeg, it would show up in my bucket. That did not happen.

I am not sure this error is related though. I did double check that AWS_SECRET_ACCESS_KEY , AWS_ACCESS_KEY_ID and AWS_ENDPOINT_URL are set in environment(inside the running container). Confirmed via python shell boto3 library that I can get access to the bucket. I didn't see other errors in the log. MEDIA_BASE_DIR="s3://<bucket-name>" is also set in config.cfg.

DavidMStraub commented 1 month ago

Hi,

the error you pasted shows that Web API looks for a media object (not media file) and doesn't find it in the database. Is that the one you deleted? If yes, the question is why it's still being looked for. Can you check the browser console which API call triggers the 404, and when opening which view?

Concerning S3: yes, that's how it's supposed to work - working for me. Btw AWS_ENDPOINT_URL can be left empty when you use AWS itself.

gnufxela commented 1 month ago 
0dec5bd1.js:324 

       GET https://<domain>/api/media/fa8a9a809291757e15aab1ae211?locale=en&profile=all&backlinks=true&extend=all 404 (Not Found)
Dn @ 0dec5bd1.js:324
_updateData @ 0dec5bd1.js:5811
(anonymous) @ 0dec5bd1.js:5811
jn @ 0dec5bd1.js:324
await in jn (async)
On @ 0dec5bd1.js:324
(anonymous) @ 0dec5bd1.js:11668
Promise.then (async)
_submit @ 0dec5bd1.js:11668
handleEvent @ 0dec5bd1.js:2
0dec5bd1.js:324 

       GET https://<domain>/api/media/fa8a9a809291757e15aab1ae211?locale=en&profile=all&backlinks=true&extend=all 404 (Not Found)
Dn @ 0dec5bd1.js:324
_updateData @ 0dec5bd1.js:5811
(anonymous) @ 0dec5bd1.js:5811
as @ 0dec5bd1.js:389
_deleteSelf @ 0dec5bd1.js:3208
await in _deleteSelf (async)
as @ 0dec5bd1.js:389
_deleteObject @ 0dec5bd1.js:936
_handleDialog @ 0dec5bd1.js:936
handleEvent @ 0dec5bd1.js:2
emitNotification @ 0dec5bd1.js:407
notifyClosed @ 0dec5bd1.js:407
(anonymous) @ 0dec5bd1.js:407
setTimeout (async)
e.close @ 0dec5bd1.js:407
e.handleClick @ 0dec5bd1.js:407
0dec5bd1.js:324 

       GET https://<domain>/api/media/fa8aa1357184544946351992579?locale=en&profile=all&backlinks=true&extend=all 404 (Not Found)
Dn @ 0dec5bd1.js:324
_updateData @ 0dec5bd1.js:5811
(anonymous) @ 0dec5bd1.js:5811
as @ 0dec5bd1.js:389
_deleteSelf @ 0dec5bd1.js:3208
await in _deleteSelf (async)
as @ 0dec5bd1.js:389
_deleteObject @ 0dec5bd1.js:936
_handleDialog @ 0dec5bd1.js:936
handleEvent @ 0dec5bd1.js:2
emitNotification @ 0dec5bd1.js:407
notifyClosed @ 0dec5bd1.js:407
(anonymous) @ 0dec5bd1.js:407
setTimeout (async)
e.close @ 0dec5bd1.js:407
e.handleClick @ 0dec5bd1.js:407
0dec5bd1.js:324 

       GET https://<domain>/api/media/fa8aa1357184544946351992579/face_detection 404 (Not Found)
Dn @ 0dec5bd1.js:324
_updateData @ 0dec5bd1.js:2725
as @ 0dec5bd1.js:389
_deleteSelf @ 0dec5bd1.js:3208
await in _deleteSelf (async)
as @ 0dec5bd1.js:389
_deleteObject @ 0dec5bd1.js:936
_handleDialog @ 0dec5bd1.js:936
handleEvent @ 0dec5bd1.js:2
emitNotification @ 0dec5bd1.js:407
notifyClosed @ 0dec5bd1.js:407
(anonymous) @ 0dec5bd1.js:407
setTimeout (async)
e.close @ 0dec5bd1.js:407
e.handleClick @ 0dec5bd1.js:407

Console log here. I noticed that backend message shows up when I deleted the media object. 

0dec5bd1.js:324 

       GET https://<domain>/api/metadata/ 401 (Unauthorized)
Dn @ 0dec5bd1.js:324
_loadDbInfo @ 0dec5bd1.js:12368
connectedCallback @ 0dec5bd1.js:12368
(anonymous) @ 0dec5bd1.js:12042

This also shows up the first time i refreshed the page after deleting(Doesn't seem to be consistent to me).

Sorry not sure what's the easiest way to verify Handle fa871f92b6761f31478482573be == the media object that is uploaded, then deleted?

gnufxela commented 1 month ago

Re: S3

Trying to best understand the source code. I think this is how the app gets started at create_app

https://github.com/gramps-project/gramps-web-api/blob/master/gramps_webapi/app.py#L94 Then, I tried stepping through it. 

>>> app.config["MEDIA_BASE_DIR"]
's3://STILLCORRECT'
>>> os.getenv("MEDIA_BASE_DIR")
>>> app.config.from_prefixed_env(prefix="GRAMPSWEB")
True
>>> app.config["MEDIA_BASE_DIR"]
'/app/media'

I didn't find an instance where create_app is called with dict in non-test cases.

Found this env set GRAMPSWEB_MEDIA_BASE_DIR=/app/media. Is this set somewhere else? I didn't find it in the docker-compose.yml.

Then, I also noticed that the SECRET_KEY doesn't match /app/secret/secret and gramps -l tree name does not match my config.cfg TREE value. tree name does match GRAMPSWEB_TREE env variable in docker file.

Thinking that my TREE and SECRET_KEY values from config are not used, even though the config is mounted. Somehow maybe I created a tree with the docker env variable and generated SECRET_KEY(through docker-entrypoint.sh) and it is persisting across multiple restart of docker compose up/down

But I still don't see how GRAMPSWEB_MEDIA_BASE_DIR gets persist though.

Edited later: For future people I did confirm it. It is due to GRAMPSWEB_MEDIA_BASE_DIR exist in the ghcr.io/gramps-project/grampsweb:v24.7.0 image. And at https://github.com/gramps-project/gramps-web-api/blob/master/gramps_webapi/app.py#L94, I found that the GRAMPSWEB_ environment variable overrides the MEDIA_BASE_DIR value set in the config file.

After specifying my own GRAMPSWEB_MEDIA_BASE_DIR in the docker-compose.yml to override the base image environment variable. I was able to get S3 storage working. I also got rid of config.cfg completely and just docker-compose.yml to specify configuration settings(https://www.grampsweb.org/Configuration/)

The SECRET_KEY and TREE discrepancy helped me figure this out. Also note that SECRET_KEY is not that important. It just logs everyone out.

DavidMStraub commented 1 month ago

Please open a discussion or forum post on your S3 setup questions, this is unrelated to the issue.

DavidMStraub commented 5 days ago

There is not enough information in this issue to understand if something is wrong with Gramps Web API. I suggest to run check & repair on the database and see if the error persists. If it does, it is crucial to check which API call actually fails. You can use the network tab in the browser dev console to find out.