Technical Metadata Extraction for Files

[ppanopticon]

Task Description

For retrievables that represent a file, we need the ability extract, store and retrieve file information (i.e., file name, absolute path, created, modified)

• [x] Fundamentally, this type of metadata should be implemented as a descriptor of type struct.
• [x] Implement and integrate metadata extraction and retrieval for any type of file.

This must probably be implemented at an enumerator level. However, there are some architectural considerations that need clarification:

• How do we specify whether or not a retrievable should be created per file?
• How do we propagate that retrievable (if created) to downstream operators?
• Is the generation of this 'metadata' feature a functionality of the enumerator itself or do we need an additional type of operator?

Dependencies

None

Boundary Conditions

None

[lucaro]

I'd argue that once a segmenter starts work on a new input, it should always generate a retrievable and send it downstream, independently of it having any associated content. The downstream operators are free to ignore it if it lacks the information they need. Since the segmenter stage is the first one that emits retirevables, it would be easiest to create these document-level instances there rather than in the enumerator. That does, however, mean that all the relevant information also needs to be propagated there. That should be doable via the source construct.

[ppanopticon]

This has been done with FileSourceMetadata. If additional fields are needed, open new ticket.