Fix `LL-LanguagesGenderData.json` to include languages with one gender being empty

hugolpz commented 1 year ago

Hello @felixknds ,

Do you think you could look at this issue some days before Wikimania ?

LanguagesGallery secret preview > search Esperanto : missing data for Esperanto.
./data/LL-LanguagesRecordingsGender.json : 100 languages > no presence of Q25 (Esperanto)

./data/LL-LanguagesSpeakers.json : 210 languages > presence of Q25 (Esperanto)

"iso": "epo",
"wikidata": "Q143",
"language": "Q25",
"records": "33842",
"speakers": "18",
"speakersMales": "17",
"speakersFemales": "0",
"speakersOthers": "1"
},

Le breton est dans le meme cas :

{
"iso": "bre",
"wikidata": "Q12107",
"language": "Q209",
"records": "1391",
"speakers": "5",
"speakersMales": "4",
"speakersFemales": "0",
"speakersOthers": "1"
},

UI

I do a speakersFemales+speakersOthers in the UI for now, so the 1 in Gender split actually means 0 (females) + 1 (others)

Screenshot_2023-06-03-14-57-39-65_40deb401b9ffe8e1df2f1cc5ba480b12

Faulty sparql

I suspect this SPARQL excludes any language where either males or females equal 0, so only 100 languages are returned.

./sparql/LL-LanguagesRecordingsGender.sparql

LLQS

https://lingualibre.org/bigdata/#query - for tests

hugolpz commented 1 year ago

@felixknds Still some weirdities.

Languages below have known male leading contributors yet appears as recordsOthers, wrongly :

SELECT (SUBSTR(STR(?languageId),32) AS ?languageQid) ?languageName
WHERE {
  VALUES ?languageName { "Afrikaans" "Breton" "Esperanto" "Gascon" "Occitan" "Salentino" "Sicilian" } # Target values
  ?languageId 
    prop:P2 entity:Q4 ;          # Filter: P2 'instance of' is Q4 'language' AND
    rdfs:label ?languageLabel .  # Assign value label into ?languageLabel
  BIND ( STRLANG(?languageName, "en") AS ?languageLabel ) # Bind filter by English
}

Given languages, return speakers with genders

SELECT (SUBSTR(STR(?language),32) AS ?languageQid) ?languageLabel (SUBSTR(STR(?speaker),32) AS ?speakerQid) ?speakerLabel ?genderLabel
WHERE {
  VALUES ?language { entity:Q209 }
  ?speaker prop:P2 entity:Q3 .  # P2 'instance of' is Q3 'speaker'
  ?speaker prop:P4 ?language .  # P4 'language' is Q209 'Esperanto'
  ?speaker prop:P8 ?gender .
  # Labels
  SERVICE wikibase:label {
    bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en" .
  } 
}

Given speaker name, return Qid

SELECT ?speakerName (SUBSTR(STR(?speakerId),32) AS ?speaker)  ?genderLabel
WHERE {
  VALUES ?speakerName { "Lepticed7" "XANA000" "ThonyVezbe" "Davidgrosclaude" } # Assign value: one or multiple values
  # note: need to comment BIND
  BIND ( STRLANG(?speakerName, "en") AS ?speakerLabel )
  # Grammatical note: ';' allows to chain actions 
  ?speakerId prop:P2 entity:Q3 ;        # Filter: P2 'instance of' is Q3 'speaker'.
             rdfs:label ?speakerLabel . # Filter by value: label equal ?speakerLabel's value
  ?speakerId prop:P8 ?gender;
             rdfs:label ?speakerLabel .
    SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }
}

Given language Qid and User Qid, gives number of recordings in that language

SELECT (SUBSTR(STR(?language),32) AS ?languageId) ?languageLabel (SUBSTR(STR(?speaker),32) AS ?speakerId)  ?speakerLabel (COUNT(?audio) AS ?audios)
WHERE {
  VALUES ?language { entity:Q25 entity:Q209 entity:Q259 entity:Q311 entity:Q930 }  # Assign value: Q25 'Esperanto' Q259 'Sicilian' into ?language  
  VALUES ?speaker { entity:Q51319 entity:Q584098 entity:Q687891 entity:Q1976 }   # Assign value: Q445757 'SangeetaRH‎' into ?speaker 
  ?audio prop:P5 ?speaker .   # Filter: P5 'speaker' is Q445757 'SangeetaRH‎'
  ?audio prop:P4 ?language .  # Filter: P4 'language' is Q34 'Marathi'
  ?audio prop:P2 entity:Q2 .  # Filter: P2 'instance of' is Q2 'record'
  # Add labels
  SERVICE wikibase:label {bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en"} 
}
GROUP BY ?language ?languageLabel ?speaker ?speakerLabel  # Sorting first groups per language and speaker

Conclusion

Yet these languages show no speakersMale in LL-LanguagesRecordingsGender.json nor in the language gallery. These male audios appears as recordingsOthers

hugolpz commented 1 year ago

After observation... It seems that somehow, whenever recordsFemale is null, all recordsMale are counted as recordsOthers.

felixknds commented 1 year ago

So, it seems as though OPTIONAL { INCLUDE %females } does not quite do what I thought it would do, so I moved the OPTIONAL clause into the subqueries instead of the outer query.