add highlight fragment scoring mode

[dohykim]

when highlight, "order" : "score", the returned fragments are not match with that I expected. In lately Lucene, WeightedFieldFragList is added and It seems work better. I think WeightedFieldFragList has more chances to return various term fragments. it considers distinct term weight and terms per fragment norm. It's hard and unefficient to take it in my application. because of number of fragment limit.

package org.elasticsearch.search.highlight; In line 117 to 122 WeightedFieldFragList than SimpleFieldFragList. or may has ES more highlight order option that use WeightedFieldFragList?

[s1monw]

can you provide an example of what is not in the order you expect and why and preferably what you would it expect to be?

[dohykim]

11933505226509.hwp.txt

I use elasticsearch to service fulltext search on office documents. Users upload their office document (excel, pdf ... etc) and search with multi keyword AND operator. I expected the more distinct highlighted term in fragments get better score. fragment scoring version Query Coordination

The problem rise when execute 'high TF term + low TF term multi keyword search'. in example, I search on linked file by keyword "test acceptable".

I tested in ES-1.5.2, but I see use of SimpleFragListBuilder not changed in current version.

Query :

post english/typeMain/_search { "size": 1, "fields": [ "highlight", "fileName" ], "query": { "match": { "contents": { "query": "test acceptable", "operator": "and" } } }, "highlight": { "order": "score", "fields": { "contents": { "fragment_size": 128, "number_of_fragments": 1000 } } } }

Ordering result, Original =======

"hits": { "total": 16, "max_score": 0.3831197, "hits": [ { "_index": "english", "_type": "typeMain", "_id": "12419", "_score": 0.3831197, "highlight": { "contents": [ "OPIc rubrics and test items, OPIc criteria, OPIc test preparation, OPIc tests in use, and English speaking tests. It was found that", "Bradshaw, J. “Test-takers reactions to a placement test.” Language Testing, 7 (1990): 13-30. Brown, A. “The role of test-taker feedback", "from the test institution. (Chung-Ang University) Key Words: ACTFL proficiency guidelines, oral proficiency testing, test-takers ", "of Foreign Languages Testing Committee - Korea (ATCK). It is now considered by test administrators and test takers that a good OPIc", "overviews the OPIc testing procedure, he or she performs tasks in an actual test. During this process, the test-taker can listen to", "about the test (Kenyon & Malabonga, 2001). Last, the computer technology facilitates test results and ratings. Once OPIc test-taking", "considered under real test conditions (Bachman, 1990). Test validation had been left to professionals concerned with test development and",

"grown regarding how examinees perceive tests, including whether a given test is acceptable to its users or not (Davies et al., 1999)",

"several considerations as perceived by test-takers. First, test-takers perceptions include test validity. It has been suggested that", "difficulty are now accepted by test-developers when undertaking task revision (Alderson, 1998). Second, test-takers reactions are", "Examinees perceptions may assist in test revision and overall acceptability as well as test validity from their perspectives. III", "response time. In the following parts, test-takers perceptions on OPIc test format, preparation and test results in use will be presented", "differences between test-takers attitudes with regard to test preparation and the intent of OPIc. Table 7. OPIc Test Preparation Statement", "8 40.\u001f\u001fAn OPIc test can be a substitute for \u001fthe TOEIC test. 3.29 1.06 47.6 22.0 4.3 English Speaking Tests Table 9 shows general", "discussed OPIc test-takers perceptions regarding ACTFL and ACTFL proficiency guidelines, OPIc test format, test preparation, and", "Korea, English speaking tests are frequently used, although test-specific strategies are used mainly to raise test scores rather than", " Hence there is a major gap between test preparation and the intent of the test itself. Test-takers should be provided with sufficient", "develop proficiency testing materials, to use oral proficiency test scores appropriately, and to teach to the test. As noted in Table",

And I switch SimpleFragListBuilder to WeightedFragListBuilder and build. (org.elasticsearch.search.highlight.FastVectorHighlighter.java)

` if (field.fieldOptions().numberOfFragments() == 0) { fragListBuilder = new SingleFragListBuilder();

                if (!forceSource && mapper.fieldType().stored()) {
                    fragmentsBuilder = new SimpleFragmentsBuilder(mapper, field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                } else {
                    fragmentsBuilder = new SourceSimpleFragmentsBuilder(mapper, context, hitContext, field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                }
            } else {
                fragListBuilder = field.fieldOptions().fragmentOffset() == -1 ? new WeightedFragListBuilder() : new WeightedFragListBuilder(field.fieldOptions().fragmentOffset());
                if (field.fieldOptions().scoreOrdered()) {
                    if (!forceSource && mapper.fieldType().stored()) {
                        fragmentsBuilder = new ScoreOrderFragmentsBuilder(field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                    } else {
                        fragmentsBuilder = new SourceScoreOrderFragmentsBuilder(mapper, context, hitContext, field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                    }
                } else {
                    if (!forceSource && mapper.fieldType().stored()) {
                        fragmentsBuilder = new SimpleFragmentsBuilder(mapper, field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                    } else {
                        fragmentsBuilder = new SourceSimpleFragmentsBuilder(mapper, context, hitContext, field.fieldOptions().preTags(), field.fieldOptions().postTags(), boundaryScanner);
                    }
                }

} `

Result : "hits": { "total": 16, "max_score": 0.3831197, "hits": [ { "_index": "english", "_type": "typeMain", "_id": "12419", "score": 0.3831197, "highlight": { "contents": [ "grown regarding how examinees perceive tests, including whether a given test is acceptable to its users or not (Davies et al., 1999)",_ "difficulty are now accepted by test-developers when undertaking task revision (Alderson, 1998). Second, test-takers reactions are", "Examinees perceptions may assist in test revision and overall acceptability as well as test validity from their perspectives. III", "OPIc rubrics and test items, OPIc criteria, OPIc test preparation, OPIc tests in use, and English speaking tests. It was found that", "Bradshaw, J. “Test-takers reactions to a placement test.” Language Testing, 7 (1990): 13-30. Brown, A. “The role of test-taker feedback", "from the test institution. (Chung-Ang University) Key Words: ACTFL proficiency guidelines, oral proficiency testing, test-takers ", "of Foreign Languages Testing Committee - Korea (ATCK). It is now considered by test administrators and test takers that a good OPIc", "overviews the OPIc testing procedure, he or she performs tasks in an actual test. During this process, the test-taker can listen to", "about the test (Kenyon & Malabonga, 2001). Last, the computer technology facilitates test results and ratings. Once OPIc test-taking", "considered under real test conditions (Bachman, 1990). Test validation had been left to professionals concerned with test development and", "several considerations as perceived by test-takers. First, test-takers perceptions include test validity. It has been suggested that", "response time. In the following parts, test-takers perceptions on OPIc test format, preparation and test results in use will be presented", "differences between test-takers attitudes with regard to test preparation and the intent of OPIc. Table 7. OPIc Test Preparation Statement", "8 40.\u001f\u001fAn OPIc test can be a substitute for \u001fthe TOEIC test. 3.29 1.06 47.6 22.0 4.3 English Speaking Tests Table 9 shows general", "discussed OPIc test-takers perceptions regarding ACTFL and ACTFL proficiency guidelines, OPIc test format, test preparation, and", "Korea, English speaking tests are frequently used, although test-specific strategies are used mainly to raise test scores rather than", " Hence there is a major gap between test preparation and the intent of the test itself. Test-takers should be provided with sufficient", "develop proficiency testing materials, to use oral proficiency test scores appropriately, and to teach to the test. As noted in Table", "informed of appropriate and ethical test content and test procedures. This study has shown how Korean test-takers perceive ACTFL and OPIc", "feedback in the test development process: test-takers reactions to a tape-mediated test of proficiency in spoken Japanese.” Language", "preparation and test results currently in use. The source of data for this research is a questionnaire filled in by test-takers. Results", "commercialized tests such as Oral Proficiency Interview-computer (OPIc), TOEIC Speaking and TOEFL-IBT. As in other tests of speaking", "speaking assessment, the number of OPIc test-takers has been increasing every year since the test was first administrated in 2007 by American",

[clintongormley]

Tested on 6.0 with unified highlighting - the unified highlighter is not returning the result matching the most different words with lowest IDF first:

``` PUT t/t/1 { "text": "(Chung-Ang University)\n\nShin, Dong-il․Kim, Na-hee․Kang, Suk-joo. “Korean Examinees Perceptions of Testing Spoken English: A Move Toward Proficiency.” Studies in English Language & Literature. 36.2(2010): 263-288. The purpose of this study is to investigate Korean test-takers’ perceptions on ACTFL Proficiency Guidelines and Oral Proficiency Interview-computer rubrics, and to understand Korean examinees’ beliefs regarding OPIc preparation and test results currently in use. The source of data for this research is a questionnaire filled in by test-takers. Results are presented in the following categories: ACTFL and ACTFL proficiency guidelines, OPIc in general, OPIc rubrics and test items, OPIc criteria, OPIc test preparation, OPIc tests in use, and English speaking tests. It was found that Korean examinees did not have enough information on ACTFL, proficiency levels, OPI and proficiency movement, although they had positive perceptions regarding OPIc rubrics. It is suggested that test-takers need to acquire sufficient information on ACTFL Proficiency Guidelines and OPIc preparation and speaking strategies from the test institution. (Chung-Ang University)\n\nKey Words: ACTFL proficiency guidelines, oral proficiency testing, test-takers’ perception, English speaking tests, proficiency movement\n\n\nI. Introduction\n\nEnglish educators in Korea are currently facing critical issues when assessing speaking proficiency. Face-to-face interviews, and semi-direct speaking test formats, are challenging conventional but indirect methods of assessing speaking proficiency. English has often been used to communicate with foreign speakers in work places as well as in academic contexts, and schools and companies have come to evaluate employees’ speaking proficiency through commercialized tests such as Oral Proficiency Interview-computer (OPIc), TOEIC Speaking and TOEFL-IBT. As in other tests of speaking assessment, the number of OPIc test-takers has been increasing every year since the test was first administrated in 2007 by American Council on the Teaching of Foreign Languages Testing Committee - Korea (ATCK). It is now considered by test administrators and test takers that a good OPIc score is necessary to get a good or better job. \nWhat makes OPIc distinct from conventional norm-referenced tests is that it measures performance by criteria described in ACTFL Proficiency Guidelines. In other words, OPIc does not compare a speaker’s performance with others’ performance. A speech sample is rated only according to criteria on the rating scale outlined in the ACTFL Proficiency Guidelines (ACTFL, 2008). Thus, it is helpful to understand these guidelines in order to prepare for OPIc. In current journals, books or academic papers, however, there has been no systematic investigation concerning how correctly Korean test-takers have recognized instructions within ACTFL Proficiency Guidelines or OPIc, and how well they understand guidelines in OPIc test-preparation activities. \nThe purpose of this study is to investigate Korean test-takers’ perceptions of ACTFL proficiency guidelines and OPIc. Also, it is intended to evaluate test-takers’ thoughts concerning OPIc preparation and the OPIc test results in use. The source of data in this study is questionnaires completed by OPIc test-takers. \n \nII. Literature Review\n \n2.1 ACTFL Proficiency Guidelines\nIn the United States, defining and assessing language proficiency was initiated in the 1950s by government agencies including the Foreign Service Institute (FSI), in order to ascertain their employees’ language proficiency and place them in foreign service positions based on their proficiency levels (Hadley, 2001). Along with the Interagency Language Roundtable (ILR), which serves to describe language proficiency levels for government contexts, FSI designed an oral proficiency test based on a proficiency scale. In 1979, a U.S. President’s Commission report on Foreign Language and International Studies recommended that language proficiency, especially oral proficiency, goals and guidelines, be formulated to structure and measure the results of foreign language instruction. Based on its findings, it was determined that standardized evaluation instruments help foreign language educators develop methodologies and curricula based on defined outcomes, and also enable foreign language students to measure their own progress. \nAs demands for proficiency standards grew, institutions like Educational Testing Service (ETS) and ACTFL undertook projects to define language learners’ level of language proficiency for the purpose of developing an analogous scale used by ILR of FSI in academic settings (Lowe & Stansfield, 1988). With grants and support from U.S. government, ACTFL Provisional Guidelines were published in 1982, and revised ACTFL Proficiency Guidelines for Speaking, Reading, Listening, and Writing were disseminated in specific detail in 1986. Also, based on ACTFL Proficiency Guidelines, the ACTFL OPI (Oral Proficiency Interview) was developed by ACTFL. \nIn 1999, ACTFL Proficiency Guidelines-Speaking were revised in accord with the development of oral proficiency testing. These guidelines describe language proficiency in spoken language, from Superior to Novice Low, within a hierarchical order. Except for Superior, each major level is further divided into three sublevels of High, Mid, and Low. Picture 1 shows the ACTFL Rating Scale, which comprises four major levels of language performance. Each description represents a range of ability, while sublevels represent a more specific definition in terms of quality and quantity of control of the speaker for the functions required at each level. \n\nFigure 1. ACTFL Rating Scale (ACTFL, 1999, p. 12)\n\nThe introduction to ACTFL Proficiency Guidelines and the ACTFL OPI formed the foundation of the ACTFL Proficiency Movement. In other words, it was created by testing approaches, which later influenced teaching practice (Hadley, 2001). Swender (2009) states that “it was shift from what teachers do in the classroom, to what students are able to do with the language they have learned. Also, the discussion is no longer about what is the best method for teaching languages, but rather, what does the data on proficiency outcomes tell us about effective instructional and curricular design? Thus, it has dramatically changed the way in which languages are taught and tested in the United States” (p. 6). \nThe impact of the ACTFL proficiency movement has implications for English education in Korea. Although the demand for oral proficiency testing is increasing, test-preparation still focuses on memorizing language forms and pronunciation. It appears there is a big gap between the purpose of testing (to assess language proficiency for real world applications) and test preparation. However, ACTFL may have a positive impact in Korea in that it provides the language profession and educators with new insights for assessing speaking proficiency. \n2.2 The ACTFL OPIc \nThe ACTFL OPI is a one-to-one telephonic or face-to-face interactive medium conducted by an interviewer in the target language. Since its development, it has been used worldwide for testing oral language proficiency. In addition, with increased usage in schools, commercial enterprises, and government agencies, OPIc has been developed to incorporate computer technology (ACTFL, 2008). \nThe goal of OPIc is the same as the OPI: to measure how well an examinee speaks a language by comparing the speaker’s performance with criteria in ACTFL Proficiency Guidelines – Speaking (ACTFL, 1999). Table 1 shows OPIc assessment criteria. \n \nTable 1. Assessment Criteria (ACTFL, 2008, p. 4)\nProficiency\nLevel\nGlobal tasks\nand functions\nContext/\nContent\nAccuracy\nText Type\nAdvanced\nNarrate and describe in major time frames and deal effectively with an unanticipated complication.\nMost informal and some formal settings/Topics of personal and general interest.\nUnderstood without difficulty by speakers unaccustomed to dealing with nonnative speakers.\nParagraphs\nIntermediate\nCreate with language initiate, maintain, and bring to close simple conversations by asking and responding to simple questions.\nSome informal settings and a limited number of transactional situations/Predictable, familiar, topics related to daily activities.\nUnderstood, with some repetition, by speakers accustomed to dealing with nonnative speakers.\nDiscrete\nsentences\nNovice\nCommunicate minimally with formulaic and rote utterance, lists and phrases.\nMost common informal settings/Most common aspects of daily life.\nMay be difficult to understand, even for speakers accustomed to dealing with nonnative speakers.\nIndividual words and phrases\n\nOPIc is criterion-referenced assessment consisting of four major categories: \n\nthe global tasks or functions performed with the language (ex. asking and answering simple questions, narrating, describing, etc.); the social contexts (ex. in a restaurant in Mexico) and the content areas (ex. ordering a meal) in which the language can be used; the accuracy with which the tasks are performed (ex. The grammar, vocabulary, pronunciation, fluency, sociolinguistic appropriateness or acceptability of what is being said with in a certain setting, and the use of appropriate strategies for discourse management); the oral text types that result from the performance of the tasks (ex. discrete words and phrases, sentences, paragraphs, or extended discourse). Each speaker performance is rated on four major categories by trained and certified raters (Swender, 1999, p. 2). \n\nOPIc measures oral proficiency up to the Advanced Low level on the ACTFL scale. For example, picture 2 shows the difference between the OPI and the OPIc using the scale. As indicated, OPIc has seven levels including three major levels.\n \nFigure 2. OPI and OPIc on the ACTFL Scale (ACTFL, 2008)\n\n \nIn testing, OPIc takes advantages of computer technology in semi-direct procedures based on ACTFL Proficiency Guidelines. First, OPIc allows examinees to choose topics based on their own interests and experience. Here OPIc has five phases: selecting a language, background survey, self-assessment, overview of OPIc, and the actual test. The procedure begins with one’s language selection. In phase 2, the background survey is implemented based on the candidate’s interests, occupation and experience. It shows how OPIc takes advantage of computer technology to be adaptive by presenting a more learner-centered assessment. \nSecond, OPIc allows examinees to self-assess their proficiency levels through a series of questions. Before the actual tasks begin, an examinee chooses his or her estimated proficiency level (Novice, Intermediate and Advanced) for a sample task. This helps examinees perform in a more flexible and less threatening environment. It also may reduce test-takers’ anxiety, which can inhibit their best performance (Malabonga, Kenyon & Carpenter, 2005). \nThird, OPIc allows examinees to control the time they need to respond to tasks. After an examinee overviews the OPIc testing procedure, he or she performs tasks in an actual test. During this process, the test-taker can listen to questions twice and even control their response time. Even though there is a maximum time limit, it is sufficient to prepare for and respond to each task. Through this procedure, examinees have more positive feelings about the test (Kenyon & Malabonga, 2001). \nLast, the computer technology facilitates test results and ratings. Once OPIc test-taking is complete, speaking samples are sent to ACTFL certified OPIc raters using the Internet. These raters listen to each sample and find the best match between a sample and the assessment criteria of the rating scale. In addition, OPIc employs a reliable assessment conducted by ACTFL-certified OPIc raters. \n \n2.3 Test-taker Perceptions \nFrom a traditional academic perspective, examinees’ perceptions of tests were regarded in terms of face validity and were not considered under real test conditions (Bachman, 1990). Test validation had been left to professionals concerned with test development and administration. However, interest has grown regarding how examinees perceive tests, including whether a given test is acceptable to its users or not (Davies et al., 1999). In this regard, there are several considerations as perceived by test-takers. \nFirst, test-takers’ perceptions include test validity. It has been suggested that if examinees have negative perceptions regarding a test, they do not perform well (Elder, Iwashita & McNamara, 2002). Consequently, if attitudes interfere with performance, test scores may be invalid (Elder & Lynch, 1996; Nevo, 1985; Spolsky, 1995). Messick (1989) stated that test-taker perceptions are therefore necessary as a crucial source of evidence for construct validity. Because of their importance, perceptions of task difficulty are now accepted by test-developers when undertaking task revision (Alderson, 1998). \nSecond, test-takers’ reactions are important when developing test formats or task types. Many researchers have suggested that feedback is useful in test development or revision (Bachman & Palmer, 1996; Brown, 1993; Shohamy, 1982; Zeidner, 1990). Findings indicate that test-takers have preferences for certain types of tasks: they tend to perceive easier tasks as those to be more interesting. Since examinee reaction to task difficulty is related to actual task performance (Bradshaw, 1990; Scott, 1986; Zeidner, 1990; Zeidner & Bensoussan, 1988), this information may be useful for test revision (Alderson, 1988; Kenyon & Stansfield, 1991).\nThird, test-takers’ perceptions are important for assessment. Related research indicates their involvement can enhance assessment criteria (Birenbaum, 1996; Wolf, Bixby, Glenn & Gardner, 1991), while Bensoussan and Kreindler (1990) suggest that scoring considerations obtained from feedback are valuable. Also, Cohen (1993) revealed that making a rating key based on feedback was useful, while involving examinees in test design and development promotes later positive impact (Bachman & Palmer, 1996). \nWhile a rich literature exists concerning test-takers’ perceptions in language testing generally, there is little research regarding the newly-developed OPIc. Accordingly, one objective of this current study is to examine feedback to ACTFL proficiency and OPIc. Examinees’ perceptions may assist in test revision and overall acceptability as well as test validity from their perspectives.\n \nIII. Method\n \nThis section addresses three aspects of the study: developing the survey instrument, administering the questionnaire to OPIc examinees, and analyzing data. \n \n3.1 Procedure\n After taking the OPIc test, examinees were asked to fill out the questionnaire. The instrument of the study was a survey questionnaire about examinees’ perception on ACTFL Proficiency Guidelines, the OPIc test, and English speaking tests. There were 45 items divided into three sub sections. The first section, ACTFL Proficiency Guidelines were concerned with how well Korean test-takers recognize and understand ACTFL Proficiency Guidelines. Then, in the second section regarding OPIc, the respondents were asked to indicate how they perceive the OPIc test in overall impression, test format, task type, OPIc preparation, and score use. Finally, the respondents were given questions about whether they perceive English speaking tests reliable and helpful on teaching and learning. \nThe questionnaire was presented in Korean in order to avoid any possible misinterpretation and misunderstanding from the English terms. Examinees were asked to indicate the degree to which they agreed with each statement using a five point scale: 5 = Strongly Agree, 4 = Agree, 3 = Moderate, 2 = Disagree, and 1 = Strongly Disagree. The questionnaire was administered to 82 OPIc examinees at a computer based test center in Seoul, Korea. They were not allowed to carry the questionnaire into the test-taking center, and all responded after they finished the test and came out of the test room. \n\n3.2 Participants \n37 (54.88%) of 82 examinees were office workers, while the remainder consisted of 16 college students (19.5%) and 21 job applicants (25.6%). The response rate by age group was 24% in the group aged 20 to 25, 26.7% in the group aged 26 to 30, 21.3% in the group aged 31 to 35, 28% in the group aged 36-40. The response rate by gender was 64.6% for males and 13.4% for females. On the average, 56.1% of the respondents responded to take OPIc to check his/her English speaking proficiency in the basic. Among the rest, 41.5 % answered that they took OPIc simply to use its score in job applications or for promotion materials. \n \nTable 2. Respondents Information\n\n \nQuestionnaire Respondents\nTotal Number\n82\nGender\n \nMale\n53 (64.6%)\nFemale\n11 (13.4%)\nNot marked \n18 (22%) \nReason to Take OPIc Test\n \nTo check his/her English speaking proficiency\n46 (56.1%)\nTo use in job application or promotion\n34 (41.5%)\nOthers\n2 (2.4%)\nIV. Results\n \n4.1 ACTFL and Proficiency Guidelines\nThe ACTFL Proficiency Guidelines and the Proficiency Movement have played an important role in the history of foreign language education (Rifkin, 2003). As mentioned, the proficiency guidelines were the basis for development of the ACTFL Proficiency Movement, which has been a shift in focus from memorizing language forms to using languages for real world purposes. Also, based on the ACTFL Proficiency Guidelines-Speaking, the OPI(c) measures how well a person speaks in general. \nDespite their usefulness, Korean examinees did not have enough information overall on ACTFL, the Proficiency Guidelines, Proficiency Movement, and OPI (Table 3). For example, 52.4% of respondents either disagreed or strongly disagreed that they “have heard about ACTFL”, while only 24.4% agreed or strongly agreed. Also, 75.6% of respondents disagreed or strongly disagreed that they “have heard about ACTFL Proficiency Movement, and only 18.3% believed that they “have heard about ACTFL Proficiency Guidelines”. \nSince 2006, there has been rapid growth regarding the OPIc test in Korea. However, it is not directly related to awareness-raising of ACTFL-driven guidelines or the movement. Few test-takers were aware who developed the OPIc, where it came from in the mandate of the Proficiency Movement, or what ACTFL Proficiency Guidelines or the OPI look like. They simply understood more about OPIc testing generally.\n \n" } ```

GET t/_search
{
  "size": 1,
  "_source": false,
  "query": {
    "match": {
      "text": {
        "query": "test acceptable",
        "operator": "and"
      }
    }
  },
  "highlight": {
    "type": "unified",
    "order": "score",
    "fields": {
      "text": {
        "fragment_size": 128,
        "number_of_fragments": 1000
      }
    }
  }
}

Returns:

• "First, test-takers’ perceptions include test validity.",
• "Second, test-takers’ reactions are important when developing test formats or task types.",
• "Test validation had been left to professionals concerned with test development and administration.",
• "Also, it is intended to evaluate test-takers’ thoughts concerning OPIc preparation and the OPIc test results in use.",
• "It is now considered by test administrators and test takers that a good OPIc score is necessary to get a good or better job.",
• "Examinees’ perceptions may assist in test revision and overall acceptability as well as test validity from their perspectives.",
• "Third, test-takers’ perceptions are important for assessment.",
• FIRST MENTION OF ACCEPTABLE "However, interest has grown regarding how examinees perceive tests, including whether a given test is acceptable to its users or not (Davies et al., 1999).",
• "It is suggested that test-takers need to acquire sufficient information on ACTFL Proficiency Guidelines and OPIc preparation and speaking strategies from the test institution.",
• etc

[jimczi]

Tested on 6.0 with unified highlighting - the unified highlighter is not returning the result matching the most different words with lowest IDF first:

When using the plain mode of the unified highlighter the IDF of the terms is missing. When using the unified_postings mode of the highlighter, the IDF is taking in account and the snippet with the term "acceptable" is ranked first. I'll open a ticket in Lucene since it should be feasible to retrieve the IDF of the terms even if we're using the plain highlighter (as long as the field is indexed).

[jimczi]

And just to be clear, the unified highlighter treats each document as a corpus and each passage as a document. The IDF in this context is the number of times the term appears in the document and the TF is the number of times the term appears inside the passage that we're trying to score.

[dohykim]

I just got there's new unified highlighter. what I used is fvh. and I see fvh as a one mode in unified highlighter.

I want to say one thing more about fvh highlighter. That is a method [getBestFragments] in lucene. ES use this in FastVectorHighlighter.java line 143 and 146. It's lucene's method that get one field's highlight result. It loads a termvector of doc in segment file in storage maybe. It loads [number of highlight field] time and very slow when query has many highlight field and even doc is big it worse. Even don't care about the field is matched.(I guess Fetch phase is independent form Query phase). I think it's speacial use case. I sets many highlight fields because of requirement that our business service multi-lingual environment.

I suffered a performance issue. I have suggested this but some guy says to me "go to lucene". I think this issue also concern with ES. If yours improve highlighter things this time, This inefficiency should take account of. I had solved this issue in my service with load termvector once in ES code and lucene [getBestFragments] method receive termvector object. So just "one load termvector operation" per "one document". I also created in lucene JIRA. https://issues.apache.org/jira/browse/LUCENE-7397?filter=-2

[jimczi]

Closing this, the unified highlighter returns the expected snippet.