Open MattMacs opened 3 years ago
As of Version 0.2.0 other language support is prepared. Which language(s) would you like to see supported and what is your usecase? It would be very helpful if you specify a concrete scenario in which your need is shown.
Hi, what is the state for supporting other languages?
I would be interested in German as well.
@JakobMiksch - you might want to first try out to create the locations.db using a wikidata query with a german query. Also it might be possible to switch to a "dynamic" wikidata query for the resolution of locations. It's a trade off for the cache size or the query speed. The data is in principle available via the "label" field of the location hierarchy within wikidata.
Again - what is your usecase - e.g. how often do you need to lookup data - what is the expected response time and so on. Designing a solution will depend on knowing what your needs are.
Need any help on this?
This is what I am looking for
Hugginface NER models are pretty slow for this task. No experience with Spacy.
@astuanax could you please create a pull request for language specific wikidata queries. We also might try using a different endpoint such as QLever for make sure we don't run into timeouts.
A first step would be to externalize the queries to yaml files using the named queries feature of the pyLodStorage QueryManager https://github.com/WolfgangFahl/pyLoDStorage/blob/ef3ff1148a5addb6974d2bc0eac1004a1ba37beb/lodstorage/query.py#L498
Ok, I'll have a look at the specific wikidata queries to start with.
the tool https://www.npmjs.com/package/wikidata-taxonomy might be helpful: human settlement is currently our base class which might be far to broad for most real-life applications.
wdtaxonomy Q486972 -c -d -m =
will show your all specialization classes
the following queries will show the top 200 classes with more than 700 instances:
# relevant human settlement classes in wikidata
# based on wdtaxonomy Q486972 -c -d -m = -s
# WF 2021-08-25
SELECT ?item ?itemLabel ?itemDescription ?instances ?sites WITH {
SELECT DISTINCT ?item { ?item wdt:P279* wd:Q486972 }
} AS %items WHERE {
INCLUDE %items .
{
SELECT ?item (count(distinct ?element) as ?instances) {
INCLUDE %items.
OPTIONAL { ?element wdt:P31 ?item }
} GROUP BY ?item
}
{
SELECT ?item (count(distinct ?site) as ?sites) {
INCLUDE %items.
OPTIONAL { ?site schema:about ?item }
} GROUP BY ?item
}
SERVICE wikibase:label {
bd:serviceParam wikibase:language "en"
}
FILTER(?instances>700)
} ORDER BY DESC (?instances)
LIMIT 200
sparqlquery -qp wikidata.yaml -qn Top200HumanSettlementClasses -f github
based on wdtaxonomy Q486972 -c -d -m = using https://www.npmjs.com/package/wikidata-taxonomy
# relevant human settlement classes in wikidata
# based on wdtaxonomy Q486972 -c -d -m = -s
# WF 2021-08-25
SELECT ?item ?itemLabel ?itemDescription ?instances ?sites WITH {
SELECT DISTINCT ?item { ?item wdt:P279* wd:Q486972 }
} AS %items WHERE {
INCLUDE %items .
{
SELECT ?item (count(distinct ?element) as ?instances) {
INCLUDE %items.
OPTIONAL { ?element wdt:P31 ?item }
} GROUP BY ?item
}
{
SELECT ?item (count(distinct ?site) as ?sites) {
INCLUDE %items.
OPTIONAL { ?site schema:about ?item }
} GROUP BY ?item
}
SERVICE wikibase:label {
bd:serviceParam wikibase:language "en"
}
FILTER(?instances>700)
} ORDER BY DESC (?instances)
LIMIT 200
item | itemLabel | itemDescription | instances | sites |
---|---|---|---|---|
Q13100073 | village-level division in China | People's Republic of China subdivision level | 588606 | 11 |
Q486972 | human settlement | community of any size, in which people live | 570340 | 80 |
Q532 | village | small clustered human settlement smaller than a town | 272870 | 168 |
Q56436498 | village in India | administrative division below the subdistrict level in the South Asian country | 145541 | 0 |
Q5084 | hamlet | small settlement in a rural area | 117270 | 57 |
Q985488 | residential community of the People's Republic of China | administrative territorial entity of the People's Republic of China | 90953 | 10 |
Q2225692 | fourth-level administrative division in Indonesia | Indonesian fourth administrative division, part of kecamatan, with a limited level of government administration. It has various national (wilayah) and local (including gampong, kampung, desa, pekon, kelurahan, nagari, etc.) designations | 81294 | 0 |
Q3558970 | village of Poland | type of populated place | 56082 | 0 |
Q484170 | commune of France | France territorial subdivision for municipalities | 45582 | 75 |
Q253019 | Ortsteil | named subdivision or section of a human settlement in Germany, Austria or Switzerland | 41777 | 11 |
Q26211545 | desa | type of administrative district in Indonesia | 41655 | 5 |
Q3257686 | locality | place of human settlement | 40859 | 7 |
Q17343829 | unincorporated community in the United States | settlement in the United States without its own local government | 39123 | 0 |
Q23925393 | Douar | administrative division of Morocco | 32026 | 7 |
Q123705 | neighborhood | geographically localised community within a larger city, town or suburb | 27091 | 53 |
Q21672098 | village of Ukraine | type of Ukrainian administrative division | 26523 | 2 |
Q735428 | town in China | People's Republic of China township-level subdivision | 21612 | 21 |
Q1529096 | village in Turkey | rural administrative division in Turkey | 19894 | 8 |
Q2116450 | manor estate | estate in land to which is incident the right to hold a manorial court | 19255 | 17 |
Q634099 | rural settlement of Russia | type of municipal unit in Russia | 19121 | 19 |
Q1134686 | frazione | territorial subdivision of a comune in Italy | 19093 | 32 |
Q2514025 | posyolok | type of settlement in Belarus, Kazakhstan, Russia and Ukraine | 18720 | 20 |
Q15303838 | municipality seat | location of the main elements of the government of a municipality | 18219 | 0 |
Q65661087 | village in Belarus | type of rural settlement in Belarus | 17320 | 0 |
Q18663566 | dissolved municipality of Japan | former municipality in Japan | 15246 | 0 |
Q3957 | town | settlement that is smaller than a city but bigger than a large village (or borough in some areas) | 14609 | 85 |
Q20541692 | settlement in Galicia, Spain | 14591 | 0 | |
Q20202352 | locality of Mexico | entity below a municipality, identified by INEGI | 12680 | 2 |
Q61089180 | part of municipality in the Czech Republic | municipes of Czech Republic | 11509 | 2 |
Q17051044 | mahalle | type of subdivision in Turkey | 11022 | 3 |
Q498162 | census-designated place | statistical concentration of population defined by the United States Census Bureau | 10733 | 42 |
Q131596 | farm | area of land for farming, or, for aquaculture, lake, river or sea, including various structures | 10110 | 95 |
Q1093829 | city of the United States | administrative division at the local government level in the United States | 9732 | 7 |
Q262166 | municipality of Germany | lowest official level of territorial division in Germany | 9608 | 35 |
Q515 | city | large permanent human settlement | 9168 | 235 |
Q23012917 | village of Burkina Faso | Situé dans la région de la boucle du mouhoun, sapala est un gros village qui est à 32km de la commune rurale de yaba et à 25km de Toma. | 8369 | 0 |
Q7930614 | village of Taiwan | fourth level administrative subdivisions of Taiwan | 7996 | 7 |
Q747074 | commune of Italy | municipal divisions of Italy | 7875 | 54 |
Q2074737 | municipality of Spain | administrative territorial entity of Spain | 6986 | 27 |
Q188509 | suburb | residential or mixed use area, either existing as part of a city or urban area or as a separate area | 6737 | 78 |
Q4285979 | gampong | Indonesian village administrative division in Aceh province, equivalent of "desa" in the rest of Indonesia | 6694 | 4 |
Q350895 | abandoned village | village that has been deserted | 6614 | 15 |
Q15127012 | town of the United States | administrative division at the local government level in the United States | 6471 | 4 |
Q5153359 | municipality of the Czech Republic | administrative territorial division of the Czech Republic | 6260 | 6 |
Q12488913 | kampung of Papua | type of village in Papua | 6248 | 2 |
Q2023000 | khutor | type of hamlet | 6181 | 29 |
Q22674925 | former populated place | settlement that no longer exists, e.g. because it was abandoned, destroyed or it became part of another settlement | 5698 | 1 |
Q3184121 | municipality of Brazil | administrative division of the states in Brazil | 5568 | 25 |
Q3055118 | single entity of population | Spanish statistical unit of human settlements | 5293 | 3 |
Q19730508 | former municipality | municipality that no longer exists | 5248 | 0 |
Q15630849 | village of Bulgaria | type of village | 5059 | 0 |
Q106505070 | Settlement of protohistoric times | settlement of cultures that have knowledge of writing (protohistory) | 5021 | 0 |
Q22580836 | village in Latvia | 4824 | 2 | |
Q51049922 | village in Estonia | type of settlement in Estonia | 4458 | 0 |
Q14839548 | minor locality in Sweden | small place with 50-199 inhabitants in Sweden | 4358 | 8 |
Q7551933 | mahallah | city subdivision or neighbourhood in parts of the Arab world, Balkans, Western and South Asia and nearby countries | 4222 | 25 |
Q15284 | municipality | administrative division having corporate status and usually powers of self-government or jurisdiction | 4212 | 106 |
Q5327369 | chōchō | subdivision of a city in Japan | 4090 | 3 |
Q1484611 | buurtschap | small inhabited place with its own name but often without a clear centre such as a church or market square | 3628 | 8 |
Q61878 | barangay | administrative division in the Philippines | 3524 | 39 |
Q19658107 | neighborhood of Brazil | administrative subdivision of municipalities in Brazil | 3417 | 0 |
Q70208 | municipality of Switzerland | smallest government division in Switzerland | 3360 | 27 |
Q152081 | concentration camp | camp in which people are imprisoned or confined, commonly in large groups, without trial | 3335 | 73 |
Q1549591 | big city | city with a population of at least 100,000 | 3231 | 20 |
Q74047 | ghost town | city depopulated of inhabitants and that often stays practically intact | 3158 | 55 |
Q751708 | village in the United States | administrative division at the local government level in the United States | 3148 | 10 |
Q490329 | dong of South Korea | lowest administrative unit of districts throughout Souh Korea | 2998 | 0 |
Q6784672 | municipality of Slovakia | municipality in Slovakia | 2884 | 9 |
Q34841063 | seat of the local council | location of the local council of a rural locality of Romania | 2861 | 1 |
Q2082774 | shieling | mountain pasture | 2839 | 20 |
Q2590631 | municipality of Hungary | 2813 | 4 | |
Q12813115 | urban area in Sweden | Def: population >= 200, < 200 m between buildings, etc | 2713 | 12 |
Q11722303 | rural municipality of Sweden and Finland | former type of municipality in Sweden (1863–1970) and Finland (1865–1977) | 2628 | 6 |
Q695850 | airbase | aerodrome used by a military force for the operation of military aircraft | 2613 | 40 |
Q17540983 | open air site | 2548 | 2 | |
Q56398194 | core municipal part with the same name as the municipality in the Czech Republic | 2480 | 0 | |
Q1852859 | populated place in the Netherlands | Officially designated place ("woonplaats") (city/town/village) in the Netherlands | 2467 | 2 |
Q21869758 | delegated commune | 2409 | 8 | |
Q15661340 | ancient city | city in historical records | 2383 | 0 |
Q667509 | municipality of Austria | local administrative unit in Austria | 2366 | 9 |
Q1952852 | municipality of Mexico | municipio: mexican administrative country subdivisions | 2320 | 24 |
Q106492558 | Settlement of prehistoric times | 2288 | 0 | |
Q72030539 | farmstead | the buildings and service areas associated with a farm | 2194 | 0 |
Q16352482 | dispersed settlement in Latvia | type of rural settlement in Latvia | 2104 | 1 |
Q106491339 | Neolithic settlement | 2101 | 0 | |
Q42744322 | urban municipality of Germany | municipality with town privileges in Germany (may e.g. legally prepend "Stadt" to its name) | 2065 | 2 |
Q106491386 | Q106491386 | 2021 | 0 | |
Q188913 | plantation | long artificially established forest, farm or estate, where crops are grown for sale | 1994 | 58 |
Q12225020 | uzlah | local administrative division of Yemen | 1982 | 4 |
Q1138414 | hermitage | an isolated building in Europe | 1925 | 4 |
Q2066754 | manor | historical manor estate in the Baltic area | 1849 | 7 |
Q3624938 | city district of Brazil | geographical and administrative region in Brazil | 1726 | 4 |
Q62049 | county seat | administrative center or seat of government for a county or civil parish | 1691 | 34 |
Q42211429 | township of Minnesota | administrative territorial entity of Minnesota | 1664 | 0 |
Q20757176 | parish municipality of Denmark | 1649 | 1 | |
Q17205621 | township of Iowa | township in Iowa, United States | 1647 | 0 |
Q21507948 | former village | village that no longer exists, e.g. because it was deserted, destroyed or it became part of another settlement | 1626 | 1 |
Q9035798 | township of Pennsylvania | type of municipality in the U.S. state of Pennsylvania | 1588 | 2 |
Q3504085 | rural municipality of Poland | administrative division of Poland | 1586 | 6 |
Q2661988 | urban settlement in Russia | type of a municipal formation in Russia | 1583 | 14 |
Q2989398 | commune of Algeria | administrative division of Algeria | 1548 | 12 |
Q2039348 | municipality of the Netherlands | second-level administrative division in the Netherlands | 1522 | 22 |
Q104841013 | hromada | administrative unit in Ukraine | 1513 | 13 |
Q902814 | border town | town or city on a border between larger political areas | 1494 | 10 |
Q24764 | municipality of the Philippines | administrative division of the Philippines | 1492 | 17 |
Q2583015 | ghetto in Nazi-occupied Europe | settlement set up in Nazi-occupied Europe by Nazi Germany for Jews and sometimes Romani | 1481 | 18 |
Q16363214 | small village in Latvia | type of rural settlement in Latvia | 1457 | 1 |
Q17198545 | township of Illinois | civil township subdivision of most Illinois counties | 1442 | 0 |
Q4632675 | dwelling place | 1400 | 2 | |
Q17205735 | township of Kansas | 1399 | 0 | |
Q17318027 | rural commune of Morocco | administrative division of Morocco | 1352 | 0 |
Q1802801 | rural municipality of Austria | type of Austrian rural municipality | 1349 | 1 |
Q28371991 | village of Yemen | type of populated place | 1342 | 1 |
Q15092344 | urban area in Norway | urban unit definition in Norway issued by Statistics Norway (SSB): population at least 200, less than 50 m between buildings, etc | 1339 | 1 |
Q245016 | military base | facility directly owned and operated by or for the military | 1327 | 48 |
Q1348006 | city block | central element of urban planning and urban design; smallest area that is surrounded by streets | 1320 | 37 |
Q17198620 | township of Ohio | 1311 | 0 | |
Q1989945 | agrotown | type of rural settlement in Belarus | 1306 | 11 |
Q685309 | former municipality of Switzerland | type of municipality of Switzerland | 1305 | 0 |
Q56557504 | city of Iran | city and type of administrative subdivision of Iran | 1271 | 0 |
Q7930989 | city/town | human settlement | 1264 | 0 |
Q15975450 | smallholding | type of rural locality | 1145 | 9 |
Q17205774 | township of Michigan | 1111 | 0 | |
Q2555896 | municipality of Colombia | decentralized subdivisions of the Republic of Colombia | 1108 | 11 |
Q27654220 | abandoned hamlet | ghost town | 1096 | 0 |
Q16830604 | census town of India | administrative territorial entity of India | 1076 | 0 |
Q17201685 | township of Indiana | 1009 | 0 | |
Q777120 | borough of Pennsylvania | type of municipality in the U.S. state of Pennsylvania | 989 | 7 |
Q4291381 | amalgamated hromada | former administrative unit in Ukraine | 970 | 2 |
Q190928 | shipyard | place where ships are repaired and built | 966 | 51 |
Q33146843 | municipality of Catalonia | municipality of Catalonia, Spain | 947 | 2 |
Q1985797 | municipal unit of Greece | municipality subdivision in Greece | 935 | 3 |
Q106071004 | town of New York | form of local government in the state of New York | 921 | 0 |
Q15141625 | subdistrict municipality | type of municipality in Thailand | 915 | 1 |
Q28328984 | village in Armenia | 912 | 0 | |
Q148837 | polis | ancient Greek social and political organisation | 901 | 71 |
Q57058 | municipality of Croatia | second-level administrative subdivision in Croatia | 895 | 16 |
Q12015918 | colony | administrative unit of Poland | 885 | 3 |
Q7216840 | urban-type settlement in Ukraine | official type of settlement between a town and a agricultural village | 878 | 7 |
Q2989400 | commune of Morocco | administrative territorial entity in Morocco | 869 | 5 |
Q106505045 | Q106505045 | 867 | 0 | |
Q12161100 | cortijo | 866 | 5 | |
Q26742250 | former municipality of Estonia | former administrative division of Estonia | 856 | 0 |
Q106504960 | Q106504960 | 835 | 0 | |
Q16355778 | average size village in Latvia | type of rural settlement in Latvia | 813 | 1 |
Q4379239 | Q4379239 | 809 | 7 | |
Q815324 | town municipality of Turkey | type of administrative region in Turkey, large village with a municipality | 803 | 7 |
Q15921247 | constituent locality | level 3 type of administrative entity of the SIRUTA system issued by the Romanian National Institute for Statistics | 790 | 0 |
Q666943 | associated commune | France territorial subdivision of a larger commune but retaining certain autonomy | 787 | 8 |
Q1708422 | settlement site | site of a former settlement of archaeological interest | 786 | 8 |
Q562061 | market municipality | municipality in some German-speaking regions; usually has a historical legal right to hold markets and fairs | 777 | 17 |
Q882149 | nagari | semi-autonomous Minangkabau people regional administrative unit in West Sumatra, Indonesia | 769 | 6 |
Q424857 | ōaza | subdivision of a village or town in Japan | 758 | 3 |
Q1059478 | town of Japan | administrative division of Japan | 757 | 23 |
Q88205 | castrum | ancient Roman fortification | 749 | 38 |
Q2989457 | urban-type settlement | type of urban populated place | 734 | 41 |
Q494721 | city of Japan | administrative division of Japan | 731 | 39 |
Q329683 | industrial zone | area for development of industry | 729 | 29 |
Q19886692 | neighbourhood of São Paulo | 716 | 0 | |
Q3559024 | village of Senegal | 713 | 2 | |
Q1743100 | church town of Sweden | district of cottages around a church, traditionally used by visiting churchgoers | 712 | 5 |
Is your feature request related to a problem? Please describe. I would like to use geograpy3 in my current project, but as I'm working with sentences in German, it doesn't work...
Describe the solution you'd like Support other languages as well.