NZSL-53: Export from Signbank

[joshmcarthur]

This pull request satisfies the requirements of https://ackama.atlassian.net/browse/NZSL-53 by adding a new script to export data from Signbank, instead of Freelex. The output file formats are sufficiently similar for the native apps (currently the only primary users of this data) to function without modification.

These data files will be used to provide dictionary data to both the nzsl-share, and nzsl-online applications. As such, there is new data available rather than what we get from Freelex:

• location_identifier - The full location identifier, including a grouping number
• age_groups -
• contains_numbers
• hint
• id
  • inflection_manner_and_degree
  • inflection_plural
  • inflection_temporal
  • is_directional
  • is_fingerspelling
  • is_locatable
  • one_or_two_handed
  • related_to - this is the sign language the gloss is related to, not another gloss
  • usage
  • usage_notes
  • word_classes
  • gloss_normalized - main gloss with accented characters stripped for search
  • minor_normalized - minor gloss with accented characters stripped for search
  • maori_normalized - Māori gloss with accented characters stripped for search

There is also a second table in the output SQLite databasevideos. This contains any assets related to a row in the words table, such as examples, drawings and main gloss videos. The assets table contains sufficient data to identify the asset, and a public URL to access the asset. Only published videos are accessible.

The release process downloads and processes primary gloss drawings from Signbank for the native apps to use. These assets are also included in the videos table, so can be linked to directly (e.g. in the web apps), or use the processed image (e.g. native apps)

The Freelex script remains in place, but I have removed it from running in Actions automatically. Per https://ackama.atlassian.net/browse/NZSL-52, the update frequency has been modified to be monthly.

Example release: https://github.com/ODNZSL/nzsl-dictionary-scripts/releases/tag/signbank-2022-11-23

[lgtm-com[bot]]

This pull request introduces 5 alerts and fixes 1 when merging 2594d637b56b6c08e30ad2d92d1793e7f94b67d3 into a6c9866a1ab602a70a469da59d741261940fee39 - view on LGTM.com

new alerts:

• 4 for Unused import
• 1 for Unused local variable

fixed alerts:

• 1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com[bot]]

This pull request introduces 5 alerts and fixes 1 when merging 4b631992890ec0a592ea9b69055308d2638bacbd into a6c9866a1ab602a70a469da59d741261940fee39 - view on LGTM.com

new alerts:

• 4 for Unused import
• 1 for Unused local variable

fixed alerts:

• 1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com[bot]]

This pull request introduces 5 alerts and fixes 1 when merging decf08ac4b1e7bf577a89b9ef354a3de528b3074 into a6c9866a1ab602a70a469da59d741261940fee39 - view on LGTM.com

new alerts:

• 4 for Unused import
• 1 for Unused local variable

fixed alerts:

• 1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.

[lgtm-com[bot]]

This pull request introduces 5 alerts and fixes 1 when merging d6431d2d4798caed9f3d24b8423b67333f0f37cd into a6c9866a1ab602a70a469da59d741261940fee39 - view on LGTM.com

new alerts:

• 4 for Unused import
• 1 for Unused local variable

fixed alerts:

• 1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine :gear: that powers LGTM.com. For more information, please check out our post on the GitHub blog.