search

UUDigitalHumanitieslab / EDPOP

A virtual research environment (VRE) that lets you collect, align and annotate bibliographical and biographical records from several online catalogs.
BSD 3-Clause "New" or "Revised" License
1 stars 0 forks source link

Feature/collection api #181

Open lukavdplas opened 1 month ago

lukavdplas commented 1 month ago

This adds an API for collections which are saved in the triplestore. close #159, close #160

It adds list, retrieve, create, update, and delete endpoints for collections. (Which replace the existing API for collections.)

This is implemented as a "regular" JSON API, but collections are identified by an URI instead of an integer.

This update only implements the API on the backend, so it will break the frontend (which expects a slightly different API for collections). I expect adjusting the frontend will go smoothly, but I did not want to bloat this PR.

In the meantime, you can test the browsable API at /api/collections/.

This update also does not include a data migration or adjust the existing data migration script; I'll add that in a later PR.

Other changes:

lukavdplas commented 1 month ago

Current status: this PR is ready for review; I've made it a draft because it should only be merged in combination with a frontend update.

Request for review: I recommend exploring the browsable API and/or looking at api_test.py to evaluate the actual endpoints. I'm particularly interested if you foresee any difficulties in the translation to frontend functions, or edge cases that can lead to issues.

The technical implementation is still a bit experimental, but quite well-tested, so it should work well enough for a first version. Suggestions/comments are welcome but addressing them may be better left until later.

jgonggrijp commented 1 month ago

@lukavdplas I merged develop into a local copy of your branch so I could use Docker to try the browsable API. There were a few minor merge conflicts that were easy to fix and I had to update backend/requirements.{in,txt} to replace the feature/json-ld branch by develop for the restframework_rdf package, because the former branch was already merged.

Unfortunately, I cannot get past the migrations of the projects app. It gets stuck on number 2, log below. Does this look familiar to you and if so, do you know how to address it? The Project class does have a graph method.

2024-07-17 17:14:53 System check identified no issues (0 silenced).
2024-07-17 17:14:54 Operations to perform:
2024-07-17 17:14:54   Apply all migrations: admin, auth, authtoken, contenttypes, projects, sessions, vre
2024-07-17 17:14:54 Running migrations:
2024-07-17 17:14:54 Traceback (most recent call last):
2024-07-17 17:14:54   File "/usr/src/app/backend/manage.py", line 22, in <module>
2024-07-17 17:14:54     execute_from_command_line(sys.argv)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
2024-07-17 17:14:54     utility.execute()
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
2024-07-17 17:14:54     self.fetch_command(subcommand).run_from_argv(self.argv)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
2024-07-17 17:14:54     self.execute(*args, **cmd_options)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
2024-07-17 17:14:54     output = self.handle(*args, **options)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
2024-07-17 17:14:54     res = handle_func(*args, **kwargs)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
2024-07-17 17:14:54     post_migrate_state = executor.migrate(
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
2024-07-17 17:14:54     state = self._migrate_all_forwards(
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
2024-07-17 17:14:54     state = self.apply_migration(
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
2024-07-17 17:14:54     state = migration.apply(state, schema_editor)
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
2024-07-17 17:14:54     operation.database_forwards(
2024-07-17 17:14:54   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
2024-07-17 17:14:54     self.code(from_state.apps, schema_editor)
2024-07-17 17:14:54   File "/usr/src/app/backend/projects/migrations/0002_researchgroups_to_projects.py", line 22, in research_groups_to_projects
2024-07-17 17:14:54     store_project_graph(Project, project, True)
2024-07-17 17:14:54   File "/usr/src/app/backend/projects/signals.py", line 21, in store_project_graph
2024-07-17 17:14:54     g = instance.graph()
2024-07-17 17:14:54 AttributeError: 'Project' object has no attribute 'graph'
2024-07-17 17:14:54   Applying projects.0002_researchgroups_to_projects...
lukavdplas commented 1 month ago

It does. This happens because migrations use a reconstruction of the model based on the migration history, rather than the actual class. So this migration tried to access the custom method graph(), but that isn't available during the migration.

This is caused by a mistake in the source code, I'll update it and let you know.

lukavdplas commented 1 month ago

Okay, I've updated it and it looks like the migrations run fine now. I'm not very happy with the solution (importing the actual model class during the migration) but it works.

jgonggrijp commented 1 month ago

Getting one for Collection as well:

2024-07-18 14:48:00 System check identified no issues (0 silenced).
2024-07-18 14:48:00 Operations to perform:
2024-07-18 14:48:00   Apply all migrations: admin, auth, authtoken, contenttypes, projects, sessions, vre
2024-07-18 14:48:00 Running migrations:
2024-07-18 14:48:01   Applying projects.0002_researchgroups_to_projects... OK
2024-07-18 14:48:01   Applying projects.0003_project_uri... OK
2024-07-18 14:48:01   Applying projects.0004_fill_project_uri... OK
2024-07-18 14:48:01   Applying projects.0005_alter_project_uri... OK
2024-07-18 14:48:01 Traceback (most recent call last):
2024-07-18 14:48:01   File "/usr/src/app/backend/manage.py", line 22, in <module>
2024-07-18 14:48:01     execute_from_command_line(sys.argv)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
2024-07-18 14:48:01     utility.execute()
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
2024-07-18 14:48:01     self.fetch_command(subcommand).run_from_argv(self.argv)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv
2024-07-18 14:48:01     self.execute(*args, **cmd_options)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 458, in execute
2024-07-18 14:48:01     output = self.handle(*args, **options)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
2024-07-18 14:48:01     res = handle_func(*args, **kwargs)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
2024-07-18 14:48:01     post_migrate_state = executor.migrate(
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
2024-07-18 14:48:01     state = self._migrate_all_forwards(
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
2024-07-18 14:48:01     state = self.apply_migration(
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
2024-07-18 14:48:01     state = migration.apply(state, schema_editor)
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
2024-07-18 14:48:01     operation.database_forwards(
2024-07-18 14:48:01   File "/usr/local/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
2024-07-18 14:48:01     self.code(from_state.apps, schema_editor)
2024-07-18 14:48:01   File "/usr/src/app/backend/vre/migrations/0006_annotation_context_collection_context.py", line 46, in save_collection_managing_group_as_context
2024-07-18 14:48:01     matching_name = obj.managing_group.filter(name=obj.name)
2024-07-18 14:48:01 AttributeError: 'Collection' object has no attribute 'name'
2024-07-18 14:48:01   Applying vre.0006_annotation_context_collection_context...
lukavdplas commented 1 month ago

Ah, that should be fixed now! Thanks for trying all this out btw 😅