Include option to additionally retrieve external IDs for data

wkyoshida commented 8 months ago

Terms

[X] I have searched open and closed data issues
[X] I agree to follow Scribe-Data's Code of Conduct

Languages

ALL

Description

This issue is to discuss an option (i.e. a flag perhaps) to also retrieve external IDs for data when running the data process (this is optional, as I'm thinking this should probably be something to opt-in, i.e. not the default behavior). On the Scribe-Server side, this information could be later useful for tracking when specific data points are new or have been updated in the external sources Scribe references, e.g. Wikidata. For those interested, it could also potentially be useful to see the IDs.

For nouns, verbs, and prepositions, this is likely the Wikidata lexemes.
For translations, autosuggestions, and emoji keywords - sources for these data points are from elsewhere - e.g. Wikipedia, Unicode CLDR, translation models. I believe these wouldn't really have IDs tied to them.. Considerations for Scribe-Server:
- I wonder if it could make sense to attempt to tie them to a matching Wikidata lexeme, but I'm still unsure as this likely could get messy.
- Is there anything else we could use that makes sense?
Also, would doing this even make sense?

Open for discussion! :blush::eyes:

andrewtavis commented 7 months ago

Hey @wkyoshida 👋 FYI I made a new issue in iOS that speaks to this even being something that we could include in the app data files 😊 See https://github.com/scribe-org/Scribe-iOS/issues/400. What that's saying is when we have a verb conjugation not showing up, this could actually be a link to the Wikidata page for the given lexeme such that the person could then enter in the conjugation and have it show up in the next data download :)

wkyoshida commented 6 months ago

It was decided in the dev sync to go ahead and already at least implement the first idea proposed in this issue:

For nouns, verbs, and prepositions, this is likely the Wikidata lexemes.

Created a different issue, #101, to track the work for this and actually decided to leave this issue open to continue the discussion on potential ideas for the second point:

For translations, autosuggestions, and emoji keywords...

Grabbing the lexemes though will already be a useful addition :grin:

scribe-org / Scribe-Data