Explore whether to embed files in file set or store file ids in file set

[elrayle]

Descriptive summary

In Valkyrie, there are two options for making an association between two resources:

1. Store the id of other resource. In this case, all resources have their own rows in database
2. Embed (nest) one resource in another. In this case, the parent resource has a row in the database and the children (nested) resources are stored in the same row with the parent.

This Issue explores the impact of these choices in Wings adapter and in Hyrax in general.

Impact on Wings

Abbreviations:

• AF - ActiveFedora
• VR - Valkyrie::Resource

query_service.find_by(id: file_set)

• find AF file_set using AF
• convert AF file_set to VR
  • NOW: file_ids attribute is populated by iterating over AF file_set files and getting the id from each AF file with file.id. This all happens using the general relationships conversion process. It is not special or unique to files.
  • EMBEDDED: files (or file_metadata) attribute is populated by iterating over AF file_set files and calling Wings::FileConverterService(af_file:) for each AF file. This will require special processing in model_transformer to know that the relationship is a file and should be converted using the FileConverterService.

NOTE: This assumes that VR FileSet with embedded FileMetadata expects an eager load of nested resources and that the nested resource is accessible in-memory from the VR FileSet (i.e. vr_file_set.files.each -- all files are already in-memory and ready to be processed)

persister.save(resource: file_set)

• convert VR to AF using ActiveFedoraConverter
  • NOW: ActiveFedoraConvert loops through file_ids; calls Hydra::PCDM::File.new(file_id) for each file and sets the association
  • EMBEDDED: ActiveFedoraConvert will loop through VR file_set files (or file_metadata), get the id from each FileMetadata instance and perform the same steps that it does now with file_ids

This is not substantially different performance wise.

custom_queries

find_file_metadata_id(id:) find_file_metadata_by_alternate_identifier(alternate_identifier:) find_many_file_metadata_by_ids(ids:) find_many_file_metadata_by_use(resource:, use:) where resource is the VR file_set

• NOW: Custom queries are used to find file_metadata by id. These are used to find file_metadata for a VR file_set using the file_ids attribute. Even the by_use query does this as it calls find_many_file_metadata_by_ids to get all file_metadata and then selects the ones with the desired use.
• EMBEDDED: Embedded resources do not have an id, so all custom queries using ids will no longer be valid.

Needs more analysis.

[tpendragon]

One significant difference will be handling changes in persisting changes to the FileMetadata. You won't be able to just query for it, change it, and save it. With this strategy changing one FileMetadata node will probably save all the other ones in the Fileset (at least for now)

[no-reply]

One significant difference will be handling changes in persisting changes to the FileMetadata. You won't be able to just query for it, change it, and save it. With this strategy changing one FileMetadata node will probably save all the other ones in the Fileset (at least for now)

as discussed in slack, this is particularly an issue for Wings, which would need to query each FileMetadata independently in order to access the FileSet; i.e. it would create an N+1 query problem for FileSet access, over the number of FileMetadata objects.

there's a second question about whether the one-FileSet per File restriction is acceptable (discussion yesterday concluded loosely "yes").

and still a third question about whether it's acceptable for FileMetadata saves to necessitate FileSet saves. i'm less certain about this last one, leaning toward "it's not ideal, but may be worth it for the other benefits of the nesting model".

i'm stuck on the first issue though, and don't think we can seriously consider nesting without well considered benchmarks showing the N+1 issue to be a non-problem up to a "reasonable" number of FileSets. how many is "reasonable"? without telemetry data, i think our best bet would be to ask in slack, via email, and in Samvera Tech.