benhylau
commented
4 months ago Attributes
https://docs.google.com/spreadsheets/d/1Cyi7Poo0kfIbBh-j4c3723FxrTusrexnqXlPRXmUaAU/edit#gid=0
Project attributes
project_id and project_path
Attribute: project_id
Convention: <organization_id>-<collection_id>
Sync folder directories use project_id (by convention, not requirement):
./taiwan
./taiwan2
./reuters-123For example: /browsertrix/crawl_completed?project_id=taiwan
| Database: | project_id |
project_path |
|---|---|---|
| taiwan | /path/to/taiwan | |
| taiwan2 | /mnt/other/taiwan2 | |
| reuters-123 | /home/user/test/hello |
File attributes
media_type and other source-derived metadata
We no longer need to pass format in during setup, as we will detect in code (based on magic bytes + extension) and put the file type as an attribute named media_type.
These are generated at run-time and not through admin interface either:
file_namefile_sizelast_modifiedmd5sha256
We will get rid of the concept of default name and description, and add these later manually.
Author attributes
We can pre-configure these per collection, and leave null to skip these attributes:
author:type: The author type as specified in https://schema.org/author which can be either "Organization" or "Person".author:name: The author name as specified in https://schema.org/name which is human readable text. The name can contain spaces and special characters.author:identifier: The author identifier as specified in https://schema.org/identifier which is usually a URL. The identifier can be a cryptographic identity or other compliant identifiers.
It would be a useful if we know ahead of time who is writing to the folder.
Configurations
authsign:signer (deprecated)
Since we are signing authenticated attributes per entry with a Starling key, we no longer need to use authsign (domain SSL cert) to sign the archives. Deprecate this configuration and the authsign service.
c2pa:signer (deprecated)
We no longer automate C2PA manifest creation, this is manually done at a later stage with a selected subset of attributes at time of publishing. Deprecate this configuration.
storage (deprecated)
Assets can be public, private, or highly sensitive. This determines how assets are handled, such as encryption and storage policies. We should require this configuration when a project is set up, and determine their default handling.
We can get rid of this at the admin interface level as we can decide registration and storage policies down the road.
asset_id:key and asset_id:value -> asset_origin
These were provided by client per asset (via spreadsheet), to include a custom key-value pair (e.g. "photo_ref_no": "IMG-0123") to keep track of the ingested item. This is included in blockchain registration metadata so the client can link the registration back to the asset profile in their internal systems.
In the new system, any registration would write the Tx as an authenticated attribute back to the AuthAttr service, so propagating the identifier to blockchain registration is not strictly necessary. However, the client would probably query the AuthAttr service with this internal ID, so we need a way to add it as an attribute, and search-indexing on this key also seem important.
I think we should automatically store asset_origin that is structured and guaranteed unique, the CLI can go through a assetOrigin-to-internalId map to insert these as new attributes on each asset.
For most files this can be the local file path in the sync folder. For ProofMode bundles, these can be the path + the unzipped file names, so it's expanded to a set of files.
asset_signature -> asset_origin_signature
This can be any signature for the asset during the capture or when client took custody of the asset. This can be added as a root-of-trust signature in the metadata of the archive. This can be done after the fact, but for ingested assets containing signatures like ProofMode and WACZ web archives, we should extract their signatures and insert them automatically.
@YurkoWasHere to link structure to use. We'll need to indicate what type of signature this is.
date_created -> time_created
A RFC 3339 timestamp that indicates when the asset was created, such as 2024-05-21T14:50:05Z
We can use the file timestamp, and if it needs to be updated the client can update with CLI against AuthAttr service.
Custom attributes
These are key value pairs provided per asset. Like most other per asset attributes, these will be done after via CLI.
Previously, extras:key_1 and extras:value_1 indicate a key-value pair that can be published (as blockchain registration metadata, C2PA metadata, etc.). Meanwhile private:key_1 and private:value_1 indicate ones that stay in an encrypted archive and is never shared publicly. The latter will be handled as "private attributes" in AuthAttr, which depending on configuration will either be written encrypted to the same database, or will be written to a private database altogether.
williamchong
makew0rld
See boxes in pink.
We need an admin web frontend that allows a logged in user to define attributes to be added to AuthAttr service. This will replace the ingestion spreadsheets in v1, and an authorized user can use this interface to set up a collection (without restarting the service).
Additionally, the "recorder" (aka. preprocessor) also injects metadata.
See: https://github.com/starlinglab/integrity-schema/tree/main/integrity-backend/input-dropbox-examples