FATAL ERROR: 138 regression(s) detected.

uerdemiryukatech commented 1 year ago

Hi there,

I installed Pelias Planet build on my server with prepare extracts from Geoearth rather than preparing myself but i got some errors when i ran test script. What can be wrong in my configuration?

Describe the bug

Results are under the bellow:

Pass: 503 Improvements: 14 Expected Failures: 101 Placeholders: 0 Regressions: 138 Total tests: 756 Took 19113ms Test success rate 81.75%

Environment (please complete the following information):

OS: Ubuntu 20
Docker versions: Docker version 20.10.21, build baeda1f

geoops commented 1 year ago

I encountered something similar on a planet build. Here is an example regression:

✘ regression [7] "/v1/search?sources=wof&text=Città del Vaticano, Vatican": score 3 out of 6
  diff:
    name
      expected: Vatican City
      actual:   Città del Vaticano
    locality
      expected: Vatican City
      actual:   Città del Vaticano
    country
      expected: Vatican City
      actual:   Vatican

When I compare my local output to the public and dev APIs they seem to have the same "regressions" https://pelias.github.io/compare/#/v1/search?sources=wof&text=Citt%C3%A0+del+Vaticano%2C+Vatican

michaelkirk commented 11 months ago

Sorry for the possible double post... GH UI is giving me grief.

These are integration tests testing the entire stack with real input data. Since the input data can change (planet-latest.osm.pbf, latest WOF, etc.), it follows that the expected output will change.

I just ran the tests against north-america and get a handful of failures:

Aggregate test results
Pass: 233
Improvements: 18
Expected Failures: 28
Placeholders: 0
Regressions: 84
Total tests: 363
Took 10415ms
Test success rate 76.86%

FATAL ERROR: 84 regression(s) detected.

Most of them seem like the kind of thing I'd expect from changing input data. e.g.:

tizen sdk: places
  ✘ [maps_service_search_place] "/v1/nearby?boundary.circle.radius=10&categories=food:pizza&point.lat=40.235771&point.lon=-74.937712": score 1 out of 2
  diff:
    name
      expected: Jule's Thin Crust
      actual:   Jule's Thin Crust Pizza

There's an argument to be made for pinning the tests to known input data, and then updating the inputs & tests in lockstep, but that might be more work for the maintainers.

full output of north-america tests

``` Tests for: http://api:4000/v1 (docker) address interpolation ✘ regression [1] "/v1/search?text=475f west 26th street, new york": score 0 out of 7 diff: layer expected: address actual: locality source expected: mixed actual: whosonfirst housenumber expected: 475 actual: street expected: West 26th Street actual: match_type expected: interpolated actual: fallback accuracy expected: point actual: centroid 'New York, NY, USA' is not close enough: distance is 7874m but should be under 50m ✘ regression [1a] "/v1/search?size=1&text=475f west 26th street, new york": score 0 out of 7 diff: layer expected: address actual: locality source expected: mixed actual: whosonfirst housenumber expected: 475 actual: street expected: West 26th Street actual: match_type expected: interpolated actual: fallback accuracy expected: point actual: centroid 'New York, NY, USA' is not close enough: distance is 7874m but should be under 50m ✘ regression [2] "/v1/search?text=257 west 26th street, new york": score 0 out of 7 diff: layer expected: address actual: locality source expected: mixed actual: whosonfirst housenumber expected: 257 actual: street expected: West 26th Street actual: match_type expected: interpolated actual: fallback accuracy expected: point actual: centroid 'New York, NY, USA' is not close enough: distance is 7453m but should be under 50m ✘ [2a] "/v1/search?text=257 west 26th street": score 4 out of 7 diff: source expected: mixed actual: openaddresses match_type expected: interpolated actual: exact '257 West 26th Street, Northampton, PA, USA' is not close enough: distance is 127200m but should be under 50m address matching ✔ [1] "/v1/search?text=30 w 26 st" ✔ [3] "/v1/search?size=1&text=49 Kay Street" ✔ [4] "/v1/search?text=461 W Main St, Cheshire, 06410" ✘ regression [5] "/v1/search?text=190 dean street brooklyn": score 0 out of 1 diff: label expected: 190 Dean Street, Brooklyn, New York, NY, USA actual: 190 Dean Street, Brooklyn, NY, USA address parsing ✘ regression [1] "/v1/search?text=101 saint marks pl 10009": score 12 out of 13 diff: label expected: 101 Saint Marks Place, Manhattan, New York, NY, USA actual: 101 Saint Marks Place, New York, NY, USA ✔ [2] "/v1/search?text=1 water st manhattan ny" ✔ [2.2] "/v1/search?text=1 water st manhattan ny" ✘ regression [3] "/v1/search?size=1&text=450 37th st, new york, ny 11232": score 12 out of 13 diff: label expected: 450 37th Street, Brooklyn, New York, NY, USA actual: 450 37th Street, Brooklyn, NY, USA ✔ [4] "/v1/search?text=starbucks 10010" ✘ regression [5] "/v1/search?text=455 43rd st new york ny 11232": score 12 out of 13 diff: label expected: 455 43rd Street, Brooklyn, New York, NY, USA actual: 455 43rd Street, Brooklyn, NY, USA ✘ regression [6] "/v1/search?text=1 main st ny 11201": score 12 out of 13 diff: label expected: 1 Main Street, Brooklyn, New York, NY, USA actual: 1 Main Street, Brooklyn, NY, USA ✘ regression [7] "/v1/search?text=186 Tuskegee St SE Atlanta GA": score 8 out of 11 diff: name expected: 186 Tuskegee Street Southeast actual: 186 Tuskegee Street SE street expected: Tuskegee Street Southeast actual: Tuskegee Street SE label expected: 186 Tuskegee Street Southeast, Atlanta, GA, USA actual: 186 Tuskegee Street SE, Atlanta, GA, USA ✔ [8] "/v1/search?text=3122 16th St San Francisco, CA 94103" ✔ [9] "/v1/search?text=3010 20th St San Francisco CA 94110" ✘ regression [10] "/v1/search?text=3577 Jackson St San Francisco, CA 94118": score 7 out of 11 diff: name expected: 3577 Jackson St actual: 3577 Jackson Street postalcode expected: 94118 actual: 94118.0 street expected: Jackson St actual: Jackson Street label expected: 3577 Jackson St, San Francisco, CA, USA actual: 3577 Jackson Street, San Francisco, CA, USA ✘ regression [11] "/v1/search?text=whole foods NY": score 4 out of 5 diff: name expected: Whole Foods actual: Whole Foods Market ✔ [11] "/v1/search?text=301 Commons Park S, Stamford, CT 06902" address type ✘ regression [1] "/v1/search?focus.point.lat=51.53177&focus.point.lon=-0.06672&text=102 Fleet Street": score 1 out of 4 diff: gid expected: openstreetmap:address:node:1401849738 actual: openaddresses:address:us/nj/statewide:595383c77dad8aa8 source expected: openstreetmap actual: openaddresses id expected: node:1401849738 actual: us/nj/statewide:595383c77dad8aa8 ✘ regression [2] "/v1/search?focus.point.lat=51.219671&focus.point.lon=-0.779389&text=Shepherd and Flock": score 2 out of 4 diff: gid expected: openstreetmap:venue:way:338854694 actual: openstreetmap:venue:node/7039565624 id expected: way:338854694 actual: node/7039565624 ✘ regression [3] "/v1/search?focus.point.lat=51.219671&focus.point.lon=-0.779389&text=22 Moor Park Lane": score 1 out of 4 diff: gid expected: openstreetmap:address:way:338854694 actual: openaddresses:address:us/ny/statewide:4ab9b02cd6aa86e0 source expected: openstreetmap actual: openaddresses id expected: way:338854694 actual: us/ny/statewide:4ab9b02cd6aa86e0 admin lookup ✔ [2] "/v1/search?layers=county&text=bronx" ✔ [3] "/v1/search?layers=county&text=Lake, MT" ✘ regression [4] "/v1/reverse?point.lat=-40.806533324215565&point.lon=144.8101043701172": no results returned ✔ [7] "/v1/search?layers=coarse&text=mission, san francisco" ✘ regression [8] "/v1/search?layers=coarse&text=crown heights": score 7 out of 8 diff: label expected: Crown Heights, Brooklyn, New York, NY, USA actual: Crown Heights, Brooklyn, NY, USA ✔ [9] "/v1/search?layers=coarse&text=palo alto" ✔ [9-1] "/v1/search?layers=coarse&text=palo alto, ca" ✔ [13] "/v1/search?layers=neighbourhood&text=Vista Center" ✔ [14] "/v1/search?layers=neighbourhood&text=Yadkin Trail" ✔ [15] "/v1/search?layers=neighbourhood&text=Hillside, Santa Clara" ✔ [18] "/v1/search?layers=region&text=california" ✘ regression [19] "/v1/search?text=5 West 4th Avenue Canada": score 5 out of 6 diff: street expected: W 4th Ave actual: West 4th Avenue autocomplete acushnet antiques ✘ regression [1] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=acus": score 0 out of 1 diff: name expected: Acushnet River Antiques Showroom actual: Acushnet ✔ [2] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=antiq" ✔ [3] "/v1/autocomplete?focus.point.lat=41.652889&focus.point.lon=-70.922898&text=antique" ✔ [4] "/v1/autocomplete?text=acushnet an" autocomplete admin areas ✔ [1] "/v1/autocomplete?text=brooklyn" ✔ [1-1] "/v1/autocomplete?text=broo" ✔ [1-2] "/v1/autocomplete?text=brook" ✔ [1-3] "/v1/autocomplete?text=brookl" ✔ [1-4] "/v1/autocomplete?text=brookly" ✔ [1-5] "/v1/autocomplete?text=brooklyn" ✘ [2] "/v1/autocomplete?text=new york (2 expectations)": score 2 out of 3 diff: label expected: New York City, Manhattan, NY, USA actual: New York, NY, USA ✔ improvement [3] "/v1/autocomplete?text=london" ✔ [4] "/v1/autocomplete?text=san francisco" ✔ improvement [5] "/v1/autocomplete?text=victoria" ✔ [9] "/v1/autocomplete?text=california" autocomplete boundary.country ✔ [1] "/v1/autocomplete?boundary.country=USA&text=London" autocomplete boundary.rect ✘ regression [1] "/v1/autocomplete?boundary.rect.min_lat=19&boundary.rect.max_lat=20&boundary.rect.min_lon=-99&boundary.rect.max_lon=-98&text=san francisco": score 2 out of 3 diff: name expected: San Francisco actual: San Francisco Acuautla autocomplete daly city ✔ improvement [1] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=dal" ✔ [2] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly" ✔ [3] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly " ✔ [4] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly c" ✔ [5] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly ci" ✔ [6] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly cit" ✔ [7] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=daly city" autocomplete focus ✘ regression [1] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=DiDi dumpling": score 0 out of 1 diff: label expected: DiDi Dumpling, Manhattan, New York, NY, USA actual: DiDi Dumpling, New York, NY, USA ✘ [2] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=union square": score 0 out of 1 diff: label expected: Union Square, New York County, NY, USA actual: Union Square, Elizabeth, NJ, USA ✘ [3] "/v1/autocomplete?focus.point.lat=37.769316&focus.point.lon=-122.484223&text=union square": score 0 out of 1 diff: label expected: Union Square, San Francisco County, CA, USA actual: Union Square, San Francisco, CA, USA ✘ regression [5] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=hard rock cafe": score 0 out of 1 diff: label expected: Hard Rock Cafe, Manhattan, New York, NY, USA actual: Hard Rock Cafe, New York, NY, USA ✔ improvement [24] "/v1/autocomplete?focus.point.lat=37.743618&focus.point.lon=-122.426117&text=zoo" ✔ [25] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=1 madison ave" ✘ regression [26] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=w 26th st": score 3 out of 4 diff: '5th Avenue & West 26th Street, New York, NY, USA' is not close enough: distance is 191m but should be under 100m ✘ [27] "/v1/autocomplete?focus.point.lat=41.05343&focus.point.lon=-73.53873&text=New York": score 4 out of 5 diff: 'New York, NY, USA' is not close enough: distance is 3463m but should be under 100m ✘ [28] "/v1/autocomplete?focus.point.lat=41.05343&focus.point.lon=-73.53873&text=New York, NY": score 4 out of 5 diff: 'New York, NY, USA' is not close enough: distance is 3463m but should be under 100m ✔ [29] "/v1/autocomplete?focus.point.lat=40.744131&focus.point.lon=-73.990424&text=San Francisco" autocomplete jitter ✘ regression [1.1] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dea": score 3 out of 4 diff: priorityThresh is 2 but found at position 5 ✘ regression [1.2] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean": score 3 out of 4 diff: priorityThresh is 2 but found at position 3 ✘ regression [1.3] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean ": score 3 out of 4 diff: priorityThresh is 2 but found at position 3 ✔ [1.4] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean s" ✔ [1.5] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean st" ✔ [1.6] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean str" ✔ [1.7] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean stre" ✔ [1.8] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean stree" ✔ [1.9] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street" ✔ [1.10] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street " ✔ [1.11] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street b" ✔ [1.12] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street br" ✔ [1.13] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street bro" ✔ [1.14] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street broo" ✔ [1.15] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brook" ✔ [1.16] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brookl" ✔ [1.17] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brookly" ✔ [1.18] "/v1/autocomplete?focus.point.lat=41.15232&focus.point.lon=-72.25119&text=190 dean street brooklyn" Autocomplete POI ✔ improvement [1-1] "/v1/autocomplete?text=Target Eureka CA" ✔ improvement [1-2] "/v1/autocomplete?text=Target Eureka California" ✔ [1-3] "/v1/autocomplete?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target" ✔ improvement [1-4] "/v1/autocomplete?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target Eureka" ✔ [2] "/v1/autocomplete?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=didi" autocomplete stop words ✔ [1] "/v1/autocomplete?text=world trade center" ✔ [3] "/v1/autocomplete?text=moscone west" autocomplete street centroids ✘ regression [3] "/v1/autocomplete?sources=osm&layers=street&text=w 26 st, new york": no results returned autocomplete street fallback ✘ regression [2] "/v1/autocomplete?text=w 26th st, new york (2 expectations)": score 4 out of 8 diff: street expected: West 26th Street actual: layer expected: street actual: venue street expected: West 26th Street actual: layer expected: address actual: venue Autocomplete Streets ✔ [1-1] "/v1/autocomplete?text=90 Vermilyea (2 expectations)" ✔ [1-2] "/v1/autocomplete?text=90 Vermilyea A (2 expectations)" ✔ [1-2] "/v1/autocomplete?text=90 Vermilyea Av (2 expectations)" ✔ [1-3] "/v1/autocomplete?text=90 Vermilyea Ave (2 expectations)" ✔ [1-4] "/v1/autocomplete?text=90 Vermilyea Aven (2 expectations)" ✔ [1-5] "/v1/autocomplete?text=90 Vermilyea Avenue (2 expectations)" ✘ [2-1] "/v1/autocomplete?text=424 Clay Av": score 2 out of 3 diff: street expected: Clay Ave actual: Clay Avenue ✘ regression [2-2] "/v1/autocomplete?text=424 Clay Ave (2 expectations)": score 6 out of 7 diff: street expected: Clay Ave actual: Clay Avenue ✔ [2-3] "/v1/autocomplete?text=424 Clay Avenu" ✔ [2-4] "/v1/autocomplete?text=424 Clay Avenue" ✔ [3-1] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker St" ✔ [3-2] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Str" ✔ [3-3] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Stre" ✔ [3-4] "/v1/autocomplete?focus.point.lat=42.102222&focus.point.lon=-75.911667&text=198 Baker Street" ✔ [4-1] "/v1/autocomplete?text=451 Fog Hill" ✔ [4-2] "/v1/autocomplete?text=451 Fog Hill rd" ✔ [4-3] "/v1/autocomplete?text=451 Fog Hill Road" ✘ regression [9] "/v1/autocomplete?text=190 dean street brooklyn": score 0 out of 1 diff: label expected: 190 Dean Street, Brooklyn, New York, NY, USA actual: 190 Dean Street, Brooklyn, NY, USA autocomplete ✘ regression [1] "/v1/autocomplete?text=DiDi Dumpling": score 0 out of 1 diff: label expected: DiDi Dumpling, Manhattan, New York, NY, USA actual: DiDi Dumpling, New York, NY, USA poi vs. admin areas balance ✔ [1] "/v1/autocomplete?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=starbucks" ✔ [2] "/v1/search?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=starbucks" ✔ [3] "/v1/autocomplete?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=mcdonalds" ✔ [4] "/v1/search?focus.point.lat=40.743987&focus.point.lon=-73.990417&text=mcdonalds" ✔ [5] "/v1/autocomplete?focus.point.lat=40.74686681162143&focus.point.lon=-73.98983001708986&text=katzs deli" Brooklyn ✔ [1] "/v1/search?boundary.country=USA&text=brooklyn" ✔ [2] "/v1/search?layers=borough&boundary.country=USA&text=brooklyn" ✔ [3-autocomplete] "/v1/autocomplete?boundary.country=USA&text=brooklyn" ✔ [4] "/v1/search?text=brooklyn" ✔ [5] "/v1/search?layers=borough&text=brooklyn" ✔ [6-autocomplete] "/v1/autocomplete?text=brooklyn" confidence score ✔ [1] "/v1/search?text=1 West 72nd St, New York, NY, 10023" encoding, or fun with diacritics ✔ [1] "/v1/search?text=mexico city" exact matches ✔ [1] "/v1/search?text=100 20th street" ✔ [2] "/v1/search?text=40 20th street" ✔ [3] "/v1/search?text=120 42nd Street" ✔ [4] "/v1/search?text=358 41st street, ny" ✔ [6] "/v1/search?text=1359 54 street, ny" ✔ [7] "/v1/search?text=310 7 street, ny" ✔ [8] "/v1/search?text=921 83 street, ny" ✔ [9] "/v1/search?text=518 3 street, ny" fallback queries ✘ regression [3] "/v1/search?text=101 california, san francisco": score 4 out of 5 diff: street expected: California actual: California Street International addresses ✔ [6] "/v1/search?boundary.country=CAN&text=490 Sussex Drive Ottawa K1N 1G8" ✘ regression [8] "/v1/search?boundary.country=MEX&text=Prolongación Ave. Alfonso Reyes #150 Col. Valle del Poniente Santa Catarina 66196": score 2 out of 3 diff: locality expected: Ciudad Santa Catarina actual: Santa Catarina ✘ [10] "/v1/search?text=Av. Juárez, Guadalajara, México": score 2 out of 3 diff: name expected: AVENIDA JUÁREZ actual: GUADALAJARA ✔ [12] "/v1/search?boundary.country=CAN&text=615 MacLeod Trail S.E. 10th Floor Calgary T2G 4T8" ✘ regression [14] "/v1/search?boundary.country=MEX&text=Paseo de la Reforma Colonia Cuauhtemoc Mexico D.F. 06500": score 2 out of 4 diff: name expected: PASEO DE LA REFORMA actual: MEXICO CITY layer expected: street actual: locality ✘ [15] "/v1/search?boundary.country=MEX&text=Progreso 175 Col. Americana Guadalajara 44160": score 2 out of 3 diff: name expected: PROGRESO 175 actual: AMERICANA labels ✔ [1] "/v1/search?text=San Francisco, San Francisco County, CA" ✘ regression [2] "/v1/search?text=30 West 26th Street, Manhattan, NY": score 0 out of 1 diff: label expected: 30 West 26th Street, Manhattan, New York, NY, USA actual: 30 West 26th Street, New York, NY, USA ✘ regression [4] "/v1/search?text=California, CA": score 0 out of 1 diff: label expected: California, USA actual: California, US, El Salvador ✘ [15] "/v1/search?sources=geonames&layers=localadmin&text=city of new york": Unexpected status code 400 ✔ [24] "/v1/search?text=national air and space museum, washington dc" landmarks ✘ [14] "/v1/search?text=statue of liberty": score 0 out of 1 diff: label expected: Statue of Liberty, Manhattan, NY, USA actual: Statue of Liberty Museum, New York, NY, USA missing postcodes ✔ [1] "/v1/search?text=267 e 10th st, new york, ny 10010" ✘ [2] "/v1/search?text=440 S Combee Rd, Lakeland, FL 33801": score 8 out of 9 diff: locality expected: Crystal Lake actual: Lakeland GET /place ✘ regression [1] "/v1/place?ids=openstreetmap:venue:node:2562438641": no results returned ✘ regression [2] "/v1/place?ids=osm:venue:node:2562438641": no results returned ✘ regression [3] "/v1/place?ids=openstreetmap:venue:way:5013364": no results returned ✘ regression [4] "/v1/place?ids=whosonfirst:region:85688525,whosonfirst:locality:101928027 (2 expectations)": score 7 out of 11 diff: gid expected: whosonfirst:locality:101928027 actual: whosonfirst:region:85688525 layer expected: locality actual: region id expected: 101928027 actual: 85688525 name expected: Cape Town actual: North Dakota ✘ regression [5] "/v1/place?ids=openstreetmap:address:node:2420772655": no results returned ✘ [6] "/v1/place?ids=whosonfirst:locality:123": no results returned Placeholder alt names ✔ [1] "/v1/search?lang=es&text=553 s main st red lion pa usa" ✔ [7] "/v1/search/structured?address=285 Fulton St&borough=Manhattan&locality=New York®ion=NY&postalcode=10007&lang=pl" ✘ regression [8] "/v1/reverse?lang=pt&point.lat=40.712748&point.lon=-74.013296": score 3 out of 4 diff: priorityThresh is 2 but found at position 7 ✔ [9] "/v1/reverse?layers=coarse&lang=pt&point.lat=40.712748&point.lon=-74.013296" ✔ improvement [10] "/v1/nearby?lang=pt&point.lat=40.712748&point.lon=-74.013296" ✔ [12] "/v1/search?boundary.circle.lat=31.030974&boundary.circle.lon=-98.326329&boundary.circle.radius=125&lang=es&text=beverly hills" ✔ [13] "/v1/search?boundary.rect.min_lon=-106.645646&boundary.rect.min_lat=25.837164&boundary.rect.max_lon=-93.508039&boundary.rect.max_lat=36.500704&lang=sr&text=beverly hills" ✔ [14] "/v1/search?focus.point.lon=-82.985586&focus.point.lat=39.98072&lang=ru&text=springfield" ✔ [18] "/v1/search?boundary.country=US&lang=bg&text=damascus" ✔ [23] "/v1/search?text=30 west 26th street, nueva york, ny" ✔ [24] "/v1/search?text=1220 calle de lago Сокорро Nouveau-Mexique" ✔ [25] "/v1/search?text=280 Bloor Street West টরোন্টো オンタリオ州 کانادا" General Placeholder requests ✔ [1] "/v1/search?text=port saint lucie, fl" ✔ [2] "/v1/search?text=port st. lucie, fl" ✔ [3] "/v1/search?text=Sault Sainte Marie, Ontario" ✔ [4] "/v1/search?text=Sault Ste Marie, Ontario" ✘ [5] "/v1/search?text=Fort Hood, TX": score 0 out of 3 diff: gid expected: whosonfirst:locality:101726923 actual: whosonfirst:county:102087383 confidence expected: 0.6 actual: 0.4 name expected: Fort Hood actual: Hood County ✘ [6] "/v1/search?text=Ft Hood, TX": score 0 out of 3 diff: gid expected: whosonfirst:locality:101726923 actual: whosonfirst:county:102087383 confidence expected: 0.6 actual: 0.4 name expected: Fort Hood actual: Hood County ✔ [7] "/v1/search?text=Mount Pleasant, SC" ✔ [8] "/v1/search?text=Mt Pleasant, SC" ✔ [9] "/v1/search?text=Carmel-by-the-Sea, CA" ✔ [10] "/v1/search?text=Carmel by the Sea, CA" ✔ [13] "/v1/search?text=City of Socorro, NM" ✔ [14] "/v1/search?text=L.A. C.A." ✔ [23] "/v1/search?layers=county&text=Socorro, NM" ✔ [26] "/v1/search?boundary.rect.min_lat=31.346514&boundary.rect.max_lat=36.987116&boundary.rect.min_lon=-109.049381&boundary.rect.max_lon=-103.006901&text=Las Vegas" Placeholder should obey geometric filters ✘ regression [2] "/v1/search?boundary.country=CA&text=2000 main st, berlin": score 1 out of 2 diff: name expected: 2000 Main St actual: 2000 Main Street Sorting rules for admin-only placeholder responses ✔ [402] "/v1/search?text=San Francisco" ✔ [600] "/v1/search?text=New Mexico" ✔ [601] "/v1/search?text=British Columbia" ✔ [602] "/v1/search?text=California" ✔ [603] "/v1/search?text=Ontario, CA" ✔ [700] "/v1/search?text=Manhattan" ✔ improvement [701] "/v1/search?text=Queens" ✔ [800] "/v1/search?text=Chelsea" ✔ [900] "/v1/search?text=Socorro, NM" ✔ [1100] "/v1/search?text=Maui" ✔ [1200] "/v1/search?text=San Fernando Valley" ✔ [1201] "/v1/search?text=Mission District" ✔ [1300] "/v1/search?text=Flatiron District" ✘ regression [1301] "/v1/search?text=Brockley": score 1 out of 2 diff: gid expected: whosonfirst:neighbourhood:85860571 actual: whosonfirst:locality:1292928207 ✔ [1302] "/v1/search?text=Haight Ashbury" ✔ [1400] "/v1/search?boundary.country=USA&text=Felton (3 expectations)" Result properties ✔ [1] "/v1/search?focus.point.lon=-73.944108&focus.point.lat=40.815263&layers=venue&text=IHOP" ✘ regression [2] "/v1/autocomplete?sources=wof&text=London": score 2 out of 5 diff: gid expected: whosonfirst:locality:101750367 actual: whosonfirst:locality:101735809 id expected: 101750367 actual: 101735809 source_id expected: 101750367 actual: 101735809 ✔ [3] "/v1/reverse?layers=venue&point.lat=40.744022&point.lon=-73.989231" ✘ regression [4] "/v1/place?ids=openstreetmap:venue:node:4509430011": no results returned quattroshapes popularity ✔ [1] "/v1/search?layers=neighbourhood&text=chelsea" ✔ [1-1] "/v1/search?layers=coarse&text=chelsea, ny" ✔ [2] "/v1/search?layers=neighbourhood&text=williamsburg" ✔ [2-1] "/v1/search?layers=neighbourhood&text=williamsburg, ny" ✔ [3] "/v1/search?layers=coarse&text=ridgewood" ✔ [3-1] "/v1/search?layers=coarse&text=ridgewood, ny" reverse non-coarse ✔ [1] "/v1/reverse?point.lat=40.038123&point.lon=-76.304096" ✘ regression [2] "/v1/reverse?point.lat=34.35755&point.lon=-119.126008": score 0 out of 2 diff: layer expected: street actual: county unexpected property found from {"layer":"county"} ✔ [3] "/v1/reverse?point.lat=39.898111&point.lon=-76.607958" ✔ [4] "/v1/reverse?layers=venue,locality&point.lat=39.898111&point.lon=-76.607958" ✔ [5] "/v1/reverse?layers=address,locality&point.lat=39.898111&point.lon=-76.607958" ✘ regression [6] "/v1/reverse?layers=street,locality&point.lat=39.898111&point.lon=-76.607958": score 0 out of 2 diff: layer expected: street actual: locality unexpected property found from {"layer":"locality"} reverse coarse ✔ [1] "/v1/reverse?layers=coarse&point.lat=40.74426&point.lon=-73.9904" ✘ regression [2] "/v1/reverse?layers=coarse&point.lat=40.042013&point.lon=-76.301163": score 5 out of 7 diff: layer expected: neighbourhood actual: locality neighbourhood expected: Rossmere actual: ✔ [5] "/v1/reverse?layers=coarse&point.lat=33.214722&point.lon=-106.363415" ✔ [10] "/v1/reverse?layers=neighbourhood&point.lat=40.74426&point.lon=-73.9904" ✔ [11] "/v1/reverse?layers=borough&point.lat=40.74426&point.lon=-73.9904" ✘ [12] "/v1/reverse?layers=locality&point.lat=40.74426&point.lon=-73.9904": score 4 out of 5 diff: county expected: New York County actual: Kings County ✔ [13] "/v1/reverse?layers=localadmin&point.lat=40.042013&point.lon=-76.301163" ✔ [14] "/v1/reverse?layers=county&point.lat=40.74426&point.lon=-73.9904" ✔ [16] "/v1/reverse?layers=region&point.lat=40.74426&point.lon=-73.9904" ✔ [19] "/v1/reverse?layers=country&point.lat=40.74426&point.lon=-73.9904" ✔ [20] "/v1/reverse?point.lat=34.119423&point.lon=-106.284351" ✘ regression [27] "/v1/reverse?layers=marinearea&point.lat=38.048214&point.lon=-76.294635": Unexpected status code 400 reverse coordinate wrapping ✔ [3] "/v1/reverse?point.lat=40.72956780913899&point.lon=-73.99463653564453" ✔ [4] "/v1/reverse?point.lat=40.72956780913899&point.lon=286.00536346435547" ✔ [5] "/v1/reverse?point.lat=40.72956780913899&point.lon=-433.99463653564453" ✔ [5] "/v1/reverse?point.lat=40.72956780913899&point.lon=-793.9946365356445" ✔ [6] "/v1/reverse?point.lat=40.744131&point.lon=286.009575" ✔ [7] "/v1/reverse?point.lat=40.744131&point.lon=-433.990425" ✔ [8] "/v1/reverse?point.lat=139.255869&point.lon=106.009575" San Francisco ✔ [1] "/v1/search?boundary.country=USA&text=san francisco" ✔ [2] "/v1/search?layers=locality&boundary.country=USA&text=san francisco" ✔ [3] "/v1/search?layers=locality,localadmin&boundary.country=USA&text=san francisco" ✔ [4-autocomplete] "/v1/autocomplete?boundary.country=USA&text=san francisco" ✔ [5] "/v1/search?text=san francisco" ✔ [6] "/v1/search?layers=locality&text=san francisco" ✔ [7] "/v1/search?layers=locality,localadmin&text=san francisco" ✔ [8-autocomplete] "/v1/autocomplete?text=san francisco" search ✘ regression [1] "/v1/search?text=brooklyn": score 0 out of 1 diff: label expected: Brooklyn, New York, NY, USA actual: Brooklyn, NY, USA ✘ regression [2] "/v1/search?text=brooklyn, ny": score 0 out of 1 diff: label expected: Brooklyn, New York, NY, USA actual: Brooklyn, NY, USA ✔ [4] "/v1/search?text=philadelphia" ✔ [4-1] "/v1/search?text=philadelphia, pa" ✘ regression [5] "/v1/search?text=new york, new york (2 expectations)": score 3 out of 5 diff: gid expected: whosonfirst:county:102081863 actual: whosonfirst:locality:85977539 name expected: New York County actual: New York ✔ [5:2] "/v1/search?text=new york city" ✔ [5:3] "/v1/search?text=new york city, usa" ✔ [6] "/v1/search?text=130 dean street brooklyn, ny" ✔ [7] "/v1/search?text=billerica" ✔ [8] "/v1/search?text=billerica, ma" ✔ [9] "/v1/search?text=15 call street billerica, ma" ✘ [1425586777012:0] "/v1/search?text=union sq": score 4 out of 7 diff: name expected: Union Square actual: Union Square Dining county expected: San Francisco County actual: Santa Clara County label expected: Union Square, San Francisco County, CA, USA actual: Union Square Dining, San Jose, CA, USA ✔ [1425586777012:2] "/v1/search?text=portland" ✔ [1425586777012:2-1] "/v1/search?text=portland, oregon" ✔ [10] "/v1/search?text=chelsea, new york" ✔ [11] "/v1/search?text=soho, new york" ✔ [15] "/v1/search?text=101 saint marks pl, new york" ✘ [18] "/v1/search?text=newark airport": score 7 out of 10 diff: name expected: Newark Liberty International Airport actual: Marriott Newark Airport neighbourhood expected: Dayton actual: label expected: Newark Liberty International Airport, Newark, NJ, USA actual: Marriott Newark Airport, Newark, NJ, USA ✔ [19] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&layers=address&sources=openstreetmap&text=30 West 26th Street" ✘ regression [20] "/v1/search?boundary.country=FRA&text=statue of liberty": no results returned ✘ regression [1426636804303:51] "/v1/search?text=4th and King san francisco": score 1 out of 2 diff: name expected: San Francisco 4th & King Street Station actual: 4th & King ✘ [24] "/v1/search?focus.point.lat=37.743618&focus.point.lon=-122.426117&text=zoo": score 2 out of 3 diff: priorityThresh is 1 but found at position 9 ✔ [25] "/v1/search?size=1&text=82 Whitney Ave, Hamilton, Ontario" searchAbbvs ✘ [1] "/v1/search?text=412 Saint Patrick St, donaldsonville, la": score 2 out of 5 diff: layer expected: address actual: locality housenumber expected: 412 actual: street expected: St Patrick St actual: ✘ regression [2] "/v1/search?text=412 St Patrick St, donaldsonville, la": score 2 out of 5 diff: layer expected: address actual: locality housenumber expected: 412 actual: street expected: St Patrick St actual: search city/country ✔ [20] "/v1/search?text=truth or consequences, usa" ✔ [21] "/v1/search?text=toronto, ca" ✔ [22] "/v1/search?text=yakima, us" ✘ regression [31] "/v1/search?text=seattle, canada": score 0 out of 3 diff: layer expected: country actual: locality country expected: Canada actual: United States country_a expected: CAN actual: USA search coarse ✘ regression [1] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=brooklyn": score 0 out of 1 diff: label expected: Brooklyn, New York, NY, USA actual: Brooklyn, NY, USA ✔ [2] "/v1/search?focus.point.lat=40.744243&focus.point.lon=-73.990342&text=new york" search geodisambiguation ✔ [1] "/v1/search?sources=wof&text=Aliquippa, PA" ISO2->ISO3 internal conversion ✘ regression [1] "/v1/search?text=1 Main St, Richlandtown, PA, US": score 4 out of 7 diff: layer expected: address actual: locality street expected: Main St actual: housenumber expected: 1 actual: ✘ regression [4] "/v1/search?text=22 Lloyd George Ave, Toronto Ontario CA": score 4 out of 5 diff: street expected: Lloyd George Ave actual: Lloyd George Avenue ✔ [5] "/v1/search?text=327 Rincon de Romos, Aguascalientes, MX" search layers ✔ [1] "/v1/search?layers=venue&text=london" search_poi ✘ [searchpoi-1] "/v1/search?text=Target Eureka CA": score 4 out of 5 diff: priorityThresh is 1 but found at position 2 ✘ [searchpoi-2] "/v1/search?text=Target Eureka California": score 4 out of 5 diff: priorityThresh is 1 but found at position 3 ✘ [searchpoi-3] "/v1/search?focus.point.lat=40.801944&focus.point.lon=-124.163611&text=Target": score 3 out of 4 diff: locality expected: Eureka actual: Torrance search_postalcodes ✔ improvement [searchpostal-1] "/v1/search?text=90210" ✔ improvement [searchpostal-4] "/v1/search?text=not-an-address, 90210" search street centroids ✘ regression [3] "/v1/search?sources=osm&layers=street&text=w 26 st, new york": score 1 out of 6 diff: name expected: West 26th Street actual: New York street expected: West 26th Street actual: layer expected: street actual: locality source expected: openstreetmap actual: whosonfirst 'New York, NY, USA' is not close enough: distance is 7550m but should be under 1500m structured address parsing ✔ [1] "/v1/search/structured?address=1396 Dual 40 Hwy&locality=Hagerstown®ion=MD&postalcode=21740&country=United States" ✘ [1.1] "/v1/search/structured?address=1396 Dual 40 Hwy&locality=Hagerstown®ion=MD&postalcode=21740&country=United States": score 5 out of 8 diff: layer expected: address actual: postalcode housenumber expected: 1396 actual: street expected: Dual Highway actual: structured geocoding ✘ regression [1] "/v1/search/structured?address=1090 N Charlotte Street&locality=Lancaster®ion=PA": score 10 out of 11 diff: priorityThresh is 1 but found at position 2 ✘ regression [2] "/v1/search/structured?address=North Charlotte Street&locality=Lancaster®ion=PA": score 6 out of 9 diff: layer expected: street actual: locality name expected: North Charlotte Street actual: Lancaster street expected: North Charlotte Street actual: ✘ regression [3] "/v1/search/structured?address=12200000 Calle de Lago&locality=Socorro®ion=NM": score 6 out of 9 diff: layer expected: street actual: locality name expected: Calle de Lago actual: Socorro street expected: Calle de Lago actual: ✔ [4] "/v1/search/structured?address=1090 Trescube Boulevard&locality=Lancaster®ion=PA" ✘ regression [5] "/v1/search/structured?address=1338 Kobbe Ave&locality=San Francisco®ion=CA": score 6 out of 7 diff: name expected: 1338 Kobbe Ave actual: 1338 Kobbe Avenue ✘ regression [7] "/v1/search/structured?address=5 russian hill pl&locality=San Francisco®ion=CA": score 6 out of 7 diff: name expected: 5 Russian Hill Pl actual: 5 Russian Hill Place ✘ regression [8] "/v1/search/structured?address=1090 n charlotte st": score 1 out of 2 diff: name expected: 1090 N Charlotte St actual: 1090 North Charlotte Street ✔ [100] "/v1/search/structured?neighbourhood=Dumbo&borough=Brooklyn" ✔ [101] "/v1/search/structured?neighbourhood=Chelsea" ✔ [102] "/v1/search/structured?neighbourhood=Noho" ✘ regression [200] "/v1/search/structured?locality=Manhattan (2 expectations)": score 18 out of 19 diff: priorityThresh is 3 but found at position 5 ✔ [201] "/v1/search/structured?borough=Manhattan&locality=New York®ion=NY" ✔ [300] "/v1/search/structured?locality=Socorro®ion=NM" ✔ [301] "/v1/search/structured?locality=New York®ion=NY&country=USA" ✘ regression [401] "/v1/search/structured?sources=wof&locality=Zumbehl": score 3 out of 5 diff: layer expected: localadmin actual: locality name expected: Zumbehl Township actual: Zumbehl ✘ regression [402] "/v1/search/structured?sources=wof&locality=Aastad&country=United States": score 5 out of 7 diff: layer expected: localadmin actual: locality name expected: Aastad Township actual: Aastad ✘ regression [403] "/v1/search/structured?sources=wof&locality=Bloominggrove®ion=Ohio": score 5 out of 7 diff: layer expected: localadmin actual: locality name expected: Bloominggrove Township actual: Bloominggrove ✔ improvement [500] "/v1/search/structured?county=Lancaster (2 expectations)" ✔ [501] "/v1/search/structured?county=Lancaster®ion=PA" ✔ [700] "/v1/search/structured?region=PA" ✔ [701] "/v1/search/structured?region=New Mexico" ✔ [1000] "/v1/search/structured?country=USA" ✔ [1001] "/v1/search/structured?country=US" ✔ [1002] "/v1/search/structured?country=United States" ✘ regression [1100] "/v1/search/structured?venue=police&address=1090 N Charlotte Street&locality=Lancaster®ion=PA": score 3 out of 4 diff: name expected: Lancaster Bureau of Police actual: Lancaster City Bureau of Police ✘ regression [1101] "/v1/search/structured?venue=nightlife&address=1090 N Charlotte Street&locality=Lancaster®ion=PA": score 4 out of 5 diff: priorityThresh is 1 but found at position 2 ✘ regression [1102] "/v1/search/structured?venue=police&address=N Charlotte Street&locality=Lancaster®ion=PA": score 3 out of 4 diff: name expected: Lancaster Bureau of Police actual: Lancaster City Bureau of Police ✘ regression [1103] "/v1/search/structured?venue=unsupported venue&address=1090 N Charlotte Street&locality=Lancaster®ion=PA": score 5 out of 6 diff: priorityThresh is 1 but found at position 2 ✘ regression [1104] "/v1/search/structured?venue=unsupported venue&address=N Charlotte Street&locality=Lancaster®ion=PA": score 2 out of 4 diff: layer expected: street actual: locality street expected: North Charlotte Street actual: ✔ [1105] "/v1/search/structured?venue=unsupported venue&locality=Lancaster®ion=PA" ✘ [1202] "/v1/search/structured?address=412 saint patrick st&locality=donaldsonville": score 2 out of 5 diff: layer expected: address actual: locality housenumber expected: 412 actual: street expected: St Patrick St actual: structured_postalcodes ✔ improvement [structuredpostal-1] "/v1/search/structured?postalcode=90210" ✔ improvement [structuredpostal-3] "/v1/search/structured?address=not-an-address&postalcode=90210" ✔ [structuredpostal-4] "/v1/search/structured?postalcode=83278&country=United States" tizen sdk: geocode ✔ [maps_service_geocode_inside_area] "/v1/search?boundary.rect.min_lon=-80.947265625&boundary.rect.min_lat=41.343824581185686&boundary.rect.max_lat=42.407234661551875&boundary.rect.max_lon=-78.90380859375&text=30 w 26th street" ✔ [maps_service_geocode_by_structured_address] "/v1/search/structured?address=30 w 26th street®ion=new york&postalcode=10010" ✔ [maps_service_reverse_geocode] "/v1/reverse?layers=address&point.lat=40.744022&point.lon=-73.989231" tizen sdk: places ✘ [maps_service_search_place] "/v1/nearby?boundary.circle.radius=10&categories=food:pizza&point.lat=40.235771&point.lon=-74.937712": score 1 out of 2 diff: name expected: Jule's Thin Crust actual: Jule's Thin Crust Pizza ✔ improvement [maps_service_search_place_by_area] "/v1/nearby?boundary.rect.min_lat=40.235974&boundary.rect.min_lon=-74.937158&boundary.rect.max_lat=40.228095&boundary.rect.max_lon=-74.944906&categories=finance&point.lat=40.235771&point.lon=-74.937712" ✔ improvement [maps_service_search_place_by_list] "/v1/nearby?boundary.rect.min_lat=40.235974&boundary.rect.min_lon=-74.937158&boundary.rect.max_lat=40.228095&boundary.rect.max_lon=-74.944906&categories=finance&point.lat=40.235771&point.lon=-74.937712" ✘ [maps_service_get_place_details] "/v1/place?ids=openstreetmap:venue:node:223078192": no results returned university ✘ regression [0] "/v1/search?boundary.country=USA&text=Teachers College, Columbia University": score 0 out of 2 diff: name expected: Teachers College actual: District of Columbia Teachers College region expected: New York actual: District of Columbia ✘ [1] "/v1/search?boundary.country=USA&text=Washington University in St. Louis": score 1 out of 2 diff: name expected: Washington University actual: Washington University in St. Louis ✔ [2] "/v1/search?boundary.country=USA&text=Miami University" ✘ regression [3] "/v1/search?text=University of Hawaii at Hilo": score 1 out of 2 diff: name expected: University of Hawai‘i at Hilo actual: University of Hawaii - Hilo Main Campus ✘ regression [4] "/v1/search?text=Universidad de Guadalajara, Guadalajara, Jalisco, México": score 3 out of 4 diff: name expected: CUCEA (Universidad de Guadalajara) actual: Preparatoria 10, Universidad de Guadalajara ✘ regression [5] "/v1/search?text=Universidad Autonoma de Ciudad Juarez": score 0 out of 2 diff: name expected: Universidad Autónoma de Ciudad Juárez actual: Rectoría Universidad Autónoma de Tamaulipas region expected: Chihuahua actual: Tamaulipas ✔ [6] "/v1/search?text=Oxford College, oxford, ga" ✘ regression [9] "/v1/search?text=Ohio State University": score 1 out of 2 diff: localadmin expected: Bath actual: Bath Township ✔ [10] "/v1/search?text=Antioch University Seattle" ✔ [11] "/v1/search?text=Union college, kentucky" ✔ improvement [12] "/v1/search?text=Union college kentucky" washington dc ✘ regression [1] "/v1/search?text=1705 P St NW, Washington, dc": score 4 out of 5 diff: street expected: P Street Northwest actual: P Street NW ✘ regression [2] "/v1/search?text=1705 P St NW, Washington, district of columbia": score 4 out of 5 diff: street expected: P Street Northwest actual: P Street NW ✔ [3] "/v1/search?text=busboys and poets, washington dc" ✘ regression [4] "/v1/search?text=641 S Street NW, washington, district of columbia": score 4 out of 5 diff: street expected: S Street Northwest actual: S Street NW Who's on First counties ✔ improvement [2] "/v1/search?sources=wof&text=Potter, PA" ✔ [3] "/v1/search?sources=wof&text=Maui, Hawaii" Who's on First countries ✔ [1] "/v1/search?sources=wof&text=United States" ✔ [2] "/v1/search?sources=wof&text=Canada" ✔ [6] "/v1/search?sources=wof&text=USA" Who's on First localadmins ✔ [2] "/v1/search?sources=wof&layers=localadmin&text=Norwood Young America, Minnesota, US" ✘ regression [3] "/v1/search?sources=wof&text=Zickrick, SD": score 4 out of 6 diff: layer expected: localadmin actual: locality name expected: Zickrick Township actual: Zickrick Who's on First localities ✔ [1] "/v1/search?sources=wof&text=New York, New York" ✔ [2] "/v1/search?sources=wof&text=Socorro, New Mexico" ✔ [3] "/v1/search?sources=wof&text=Truth or Consequences, NM" ✔ [5] "/v1/search?sources=wof&text=Carmel-by-the-Sea, California" ✔ [6] "/v1/search?sources=wof&layers=locality&text=Springfield" Who's on First neighbourhoods ✔ [1] "/v1/search?sources=wof&text=Crown Heights, Brooklyn, New York" ✔ [2] "/v1/search?sources=wof&text=Reparto Vista Alegre, Cuba" ✔ [3] "/v1/search?sources=wof&text=Nuevo Cristóbal, Panama" Who's on First regions ✔ [1] "/v1/search?sources=wof&text=Pennsylvania" ✔ [2] "/v1/search?sources=wof&text=New Mexico" ✔ [3] "/v1/search?sources=wof&text=Newfoundland and Labrador" ✔ [4] "/v1/search?sources=wof&text=Prince Edward Island" Aggregate test results Pass: 233 Improvements: 18 Expected Failures: 28 Placeholders: 0 Regressions: 84 Total tests: 363 Took 10415ms Test success rate 76.86% FATAL ERROR: 84 regression(s) detected. ```

pelias / docker

FATAL ERROR: 138 regression(s) detected. #304