Option to filter tokens by languages

[tailtq]

Hi everyone, I'm using the /parser/search API to search a location using free-text. Can we have another option besides lang to filter tokens by only a specific language? Thanks in advance.

Use-cases

I searched for the keyword Holland, Michigan and it returned Holland but in the War language (Austroasiatic language used by the minority of people in Bangladesh and India). When mapping into English, it turned into Baraga, which caused confusion to the users.

I have debugged and found the problem within this function: https://github.com/pelias/placeholder/blob/master/lib/Queries.js#L83-L105

As well as two queries match_subject_distinct_subject_ids and match_subject_autocomplete_distinct_subject_ids

Proposal

In my opinion, we should define an option to specify which languages or all of them should be used. For example, we can pass a search_language parameter with a value like "eng,fra" to filter tokens in English and French. If this option is ignored, we can filter all languages.

The SQL queries after editing should be:

SELECT DISTINCT( t1.id ) AS subjectId
FROM tokens AS t1
  JOIN fulltext AS f1 ON f1.rowid = t1.rowid
WHERE f1.fulltext MATCH $subject
AND t1.lang IN ('eng', 'fra')
-- AND t1.tag NOT IN ( 'colloquial' )
ORDER BY t1.id ASC
LIMIT $limit

SELECT DISTINCT( t1.id ) AS subjectId
FROM tokens AS t1
  JOIN fulltext AS f1 ON f1.rowid = t1.rowid
WHERE f1.fulltext MATCH $subject
AND t1.lang IN ('eng', 'fra')
-- AND t1.tag NOT IN ( 'colloquial' )
ORDER BY t1.id ASC
LIMIT $limit

[tailtq]

@missinglink Could you please take a look?

[missinglink]

I don't understand the issue completely, your description says:

I searched for the keyword Holland, Michigan and it 
returned Holland but in the War language 
(Austroasiatic language used by the minority of 
people in Bangladesh and India). When mapping 
into English, it turned into Baraga, which caused 
confusion to the users.

However when I enter Holland, Michigan in the demo I don't see Baraga: https://placeholder.demo.geocode.earth/demo/#eng

What am I missing?