Make comprehensive tables for mentions

[willbeason]

Here's the normalized form of the software mentions files. Note that bounding boxes for references must be in their own table since their unique key additionally requires the id of the reference within the paper.

I'll need to get clarity on what some of these fields mean.

Bolded entries form a primary key, possibly composite.

PaperSchema

• uuid: the UUID of the paper in the SoftCite dataset
• application: the application which produced the record
• date: the date the record was processed
• file: the file that was processed
• softcite_file_name: the file name within the SoftCite dataset
• id: unknown
• md5: an md5 hash, unknown source
• metadata.id: unknown
• original_file_path: the file path when processed
• runtime: the time to process the entry, units unknown
• version: the version of the application used to produce the record

MentionsSchema

• uuid: the UUID of the paper this mention is in
• index: the index of the mention
• confidence: unknown
• context: a string including the mention
• documentContextAttributes.created.score
• documentContextAttributes.created.value: whether the paper is claiming to have created the mentioned item
• documentContextAttributes.shared.score
• documentContextAttributes.shared.value: whether the paper is sharing the mentioned item
• documentContextAttributes.used.score
• documentContextAttributes.used.value: whether the paper is claiming to use the mentioned item
• inDataAvailabilitySection: unknown
• lang: the human language of the paper
• language.normalizedForm: the normalized form of the machine language mentioned
• language.offsetEnd: unknown
• language.offsetStart: unknown
• language.rawForm: the form of the machine language as it appeared in the paper
• language.wikidataId: the Q-prefixed ID of the Wikidata entry for the language
• mentionContextAttributes.created.score
• mentionContextAttributes.created.value: unknown
• mentionContextAttributes.shared.score
• mentionContextAttributes.shared.value: unknown
• mentionContextAttributes.used.score
• mentionContextAttributes.used.value: unknown
• publisher.confidence
• publisher.lang
• publisher.normalizedForm
• publisher.offsetEnd
• publisher.offsetStart
• publisher.rawForm
• publisher.wikidataId
• publisher.wikipediaExternalRef
• software-name.confidence
• software-name.lang
• software-name.normalizedForm
• software-name.offsetEnd
• software-name.offsetStart
• software-name.rawForm
• software-name.wikidataId
• software-name.wikipediaExternalRef
• software-type
• type
• url.normalizedForm
• url.offsetEnd
• url.offsetStart
• url.rawForm
• version.normalizedForm
• version.offsetEnd
• version.offsetStart
• version.rawForm
• wikidataId
• wikipediaExternalRef

PagesSchema

• uuid: the UUID of the paper this mention is in
• index: the index of the page
• page_height
• page_width

ReferencesSchema

• uuid: the UUID of the paper this mention is in
• index: the index of the reference
• refKey
• tei

MentionsBoundingBoxes

• uuid: the UUID of the paper this mention is in
• index: the index of the page
• box_type: the data the box encloses ("language", "publisher", "software-name", "url", "version")
• h
• p: the page number on which the bounding box appears
• w
• x
• y

ReferencesBoundingBoxes

• uuid: the UUID of the paper this mention is in
• index: the index of the reference in the paper
• box_index: the specific box enclosing the reference (see ReferencesSchema)
• h
• p: the page number on which the bounding box appears
• w
• x
• y

[willbeason]

Just realized I forgot - there should be a "source" enum as well that specifies the actual object processing was run on (LaTeX/tei/etc.). That's an additional primary key on all tables.

[jameshowison]

Yeah, the UUID is the article (aka "work") but a single article can have multiple sources, so we'll need both.

[jameshowison]

Are you able to map everything to this documentation: https://github.com/softcite/software-mentions/blob/master/doc/annotation_schema.md