Currently failing tests

[dianashk]

### search
  ✘ regression [2] "{"text":"brooklyn, ny"}": score 3 out of 4
  diff:
    label
      expected: Brooklyn, Kings County, NY, USA
      actual:   Brooklyn, New York, NY, USA
  ✘ regression [7] "{"text":"billerica"}": score 1 out of 2
  diff:
    label
      expected: Billerica, Middlesex County, MA, USA
      actual:   Billerica, River Pines, MA, USA
  ✘ regression [8] "{"text":"billerica, ma"}": score 1 out of 2
  diff:
    label
      expected: Billerica, Middlesex County, MA, USA
      actual:   Billerica, River Pines, MA, USA
  ✘ regression [10] "{"text":"chelsea, new york"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [11] "{"text":"soho, new york"}": score 2 out of 4
  diff:
    label
      expected: Soho, New York County, NY, USA
      actual:   Soho, Brooklyn, NY, USA
    label
      expected: SoHo, Manhattan, NY, USA
      actual:   Soho, Brooklyn, NY, USA
  ✘ regression [1425586777012:3] "{"text":"paris"}": score 1 out of 2
  diff:
    label
      expected: Paris, France
      actual:   Paris, Paris-4E-Arrondissement, France
  ✘ regression [1425586777012:6] "{"text":"london"}": score 3 out of 6
  diff:
    label
      expected: London, Greater London, United Kingdom
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, Middlesex, Canada
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, United Kingdom
      actual:   London, City of Westminster, United Kingdom

### San Francisco
  ✘ regression [5] "{"text":"san francisco"}": score 2 out of 3
  diff:
    region
      expected: California
      actual:   Atlántida

### quattroshapes popularity
  ✘ regression [1] "{"layers":"coarse","text":"chelsea"}": score 3 out of 4
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3] "{"layers":"coarse","text":"ridgewood"}": score 3 out of 4
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA
  ✘ regression [1-1] "{"layers":"coarse","text":"chelsea, ny"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3-1] "{"layers":"coarse","text":"ridgewood, ny"}": score 1 out of 2
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA

### Brooklyn
  ✘ regression [3] "{"text":"brooklyn","layers":"locality,localadmin","boundary.country":"USA"}": score 2 out of 3
  diff:
    region
      expected: New York
      actual:   Iowa
  ✘ regression [7] "{"text":"brooklyn","layers":"locality,localadmin"}": score 2 out of 3
  diff:
    region
      expected: New York
      actual:   Victoria

### autocomplete admin areas
  ✘ regression [2] "{"text":"new york"}": score 7 out of 8
  diff:
    label
      expected: New York, USA
      actual:   New York, Brooklyn, NY, USA
  ✘ regression [7] "{"text":"wales"}": score 1 out of 2
  diff:
    label
      expected: Wales, United Kingdom
      actual:   Powys, United Kingdom
  ✘ regression [11] "{"text":"madrid"}": score 4 out of 5
  diff:
    layer
      expected: locality
      actual:   venue

### admin lookup
  ✘ regression [17] "{"text":"ringland","layers":"neighbourhood"}": score 5 out of 6
  diff:
    county
      expected: Newport City
      actual:
  ✘ regression [18] "{"text":"california","layers":"region"}": score 8 out of 9
  diff:
    source
      expected: whosonfirst
      actual:   geonames
  ✘ regression [5] "{"point.lat":51.515049,"point.lon":-0.093229,"layers":"coarse"}": score 2 out of 3
  diff:
    region
      expected: London
      actual:
  ✘ regression [8] "{"text":"crown heights","layers":"coarse"}": score 6 out of 8
  diff:
    label
      expected: Crown Heights, Kings County, NY, USA
      actual:   Crown Heights, Poughkeepsie, NY, USA
    county
      expected: Kings County
      actual:   Dutchess County

### address parsing
  ✘ regression [3] "{"text":"450 w 37th st, new york, ny 11232"}": score 13 out of 14
  diff:
    unexpected result found
  ✘ regression [4] "{"text":"starbucks 10010"}": score 10 out of 14
  diff:
    name
      expected: Starbucks
      actual:   100 East 22nd Street
    housenumber
      expected: 304
      actual:   100
    street
      expected: Park Avenue South
      actual:   East 22nd Street
    label
      expected: Starbucks, Manhattan, NY, USA
      actual:   100 East 22nd Street, Manhattan, NY, USA

### address matching
  ✘ regression [2] "{"text":"507 hackney rd"}": score 6 out of 7
  diff:
    name
      expected: Hackney Cycles
      actual:   507 Hackney Road
  ✘ regression [3] "{"text":"49 Kay Street"}": score 2 out of 3
  diff:
    unexpected result found

Aggregate test results Pass: 129 Fail: 24 Placeholders: 0 Regressions: 25 Took 81119ms Test success rate 86%

[dianashk]

Related issues filed:

Chelsea, NY Soho, NY

https://github.com/whosonfirst/whosonfirst-data/issues/226

[dianashk]

Crown Heights, NY

not clear what's happening since the record from WOF is in the index as can be seen here however search doesn't surface it. Popularity is set to 1 so maybe it's just not able to score high enough to surface above the other results.

[dianashk]

London

https://github.com/whosonfirst/whosonfirst-data/issues/225

[dianashk]

search

  ✘ regression [10] "{"text":"chelsea, new york"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [11] "{"text":"soho, new york"}": score 3 out of 4
  diff:
    label
      expected: Soho, New York County, NY, USA
      actual:   Soho, Brooklyn, NY, USA
  ✘ regression [1425586777012:3] "{"text":"paris"}": score 1 out of 2
  diff:
    label
      expected: Paris, France
      actual:   Paris, Paris-4E-Arrondissement, France
  ✘ regression [1425586777012:6] "{"text":"london"}": score 3 out of 6
  diff:
    label
      expected: London, Greater London, United Kingdom
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, Middlesex, Canada
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, United Kingdom
      actual:   London, City of Westminster, United Kingdom

San Francisco

  ✘ regression [5] "{"text":"san francisco"}": score 2 out of 3
  diff:
    region
      expected: California
      actual:   Atlántida

quattroshapes popularity

  ✘ regression [1] "{"layers":"coarse","text":"chelsea"}": score 3 out of 4
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3] "{"layers":"coarse","text":"ridgewood"}": score 3 out of 4
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA
  ✘ regression [1-1] "{"layers":"coarse","text":"chelsea, ny"}": score 1 out of 2
  diff:
    label
      expected: Chelsea, New York County, NY, USA
      actual:   Chelsea, London, United Kingdom
  ✘ regression [3-1] "{"layers":"coarse","text":"ridgewood, ny"}": score 1 out of 2
  diff:
    label
      expected: Ridgewood, Queens County, NY, USA
      actual:   Ridgewood, Bergen County, NJ, USA

labels

  ✘ regression [12] "{"text":"New Zealand"}": score 1 out of 2
  diff:
    label
      expected: New Zealand
      actual:   New Zealand, Kaikoura District

autocomplete admin areas

  ✘ regression [2] "{"text":"new york"}": score 2 out of 4
  diff:
    label
      expected: New York, NY, USA
      actual:   New York, Brooklyn, NY, USA
    label
      expected: New York City, Manhattan, NY, USA
      actual:   New York, Brooklyn, NY, USA
  ✘ regression [3] "{"text":"london"}": score 2 out of 4
  diff:
    label
      expected: London, Greater London, United Kingdom
      actual:   London, City of Westminster, United Kingdom
    label
      expected: London, Middlesex, Canada
      actual:   London, City of Westminster, United Kingdom

admin lookup

  ✘ regression [5] "{"point.lat":51.515049,"point.lon":-0.093229}": score 2 out of 3
  diff:
    region
      expected: London
      actual:   City and County of the City of London
  ✘ regression [7] "{"text":"mission, ca","layers":"coarse"}": score 7 out of 8
  diff:
    priorityThresh is 5 but found at position 7

address parsing

  ✘ regression [3] "{"text":"450 w 37th st, new york, ny 11232"}": score 13 out of 14
  diff:
    unexpected result found

Aggregate test results
Pass: 146
Fail: 23
Placeholders: 0
Regressions: 15
Took 78292ms
Test success rate 92%

[orangejulius]

Ok, we're now down to just a few tests failing. They roughly fall into 3 categories, with a few other misc. failures

Categories

• [x] County is Brooklyn, should be Manhattan Will be fixed with the next build thanks to whosonfirst/whosonfirst-data#228
• [x] Geonames countries have neighborhood values fixed in pelias/wof-admin-lookup#37, which was just released in pelias-wof-admin-lookup-1.4.0 and will be in the next build
• [ ] London (the city) contains a region called "The City of Westminister" Tracked in whosonfirst/whosonfirst-data#225. We can probably ship without fixing this at first?

Individual failures

• [x] SF, CA ranks fourth in a search for san francisco, ca Needs looking into Fixed by https://github.com/pelias/schema/pull/111
• [ ] Unexpected zip code found in a search for 450 w 37th st, new york, ny 11232 This one tests that if we specify a brooklyn address with zip code, that an address which is identical except for zip code, and is in Manhattan, does not appear. Needs some more investigation, and we should probably fix it since the bad result is first. Tracked in pelias/api#456
• [x] coarse search for mission, ca does not return the mission district high enough. I believe this is a bad test since there are many places in CA called mission. I've proposed a new test, which we pass, in #201
• [ ] search for 49 Kay Street returns unexpected result 49 De Kay Street Ideally address matching would not return streets that are similar but clearly wrong, so we should investigate why this result is showing up now. But this seems like another we could wait on fixing (correct results are still first)