search

slashP / Vali

GNU General Public License v3.0
25 stars 1 forks source link

readme

Vali - create GeoGuessr locations like a pro

The creator of An Arbitrary World, An Arbitrary Rural World, Dirty World, IntersectionGuessr ++ brings you Vali - the next evolution in computer generating GeoGuessr maps. With this tool you can create "An Arbitrary Rural Southern Europe", "Coastal Sri Lanka", "Skewed Africa" or something brand new. More than 100 million possible locations are available - be creative.

What is Vali?

Vali uses a massive pool of pre-generated Google street view locations combined with specific data from OpenStreetMap and Nominatim to provide a tool that can generate locations based on your preferences. The creation and update of the locations happens outside of Vali and is not part of this repository. You provide a specification of what type of locations you want in the form of a JSON file, and Vali will give you locations you can upload to map-making.app or GeoGuessr.

First time setup

Getting started

Commands

Buliding blocks

Go directly to full examples or properties if you prefer not to read.

Selecting countries

You need to specify which countries you want to include in the countryCodes array. Example:

{
  "countryCodes": ["EC", "CO", "PE", "BO"]
}

Distribution strategy

You can select between different strategies for distributing locations. Option 1 is FixedCountByMaxMinDistance where you specify how many locations you want in total and each subdivision will get their locations spread as far away from each other as possible. You can set a minimum minimum distance to set a lower limit on how close each location can be to each other. This is the recommended approach for most. Example:

{
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 45000,
    "minMinDistance": 200
  }
}

Option 2 is MaxCountByFixedMinDistance where you specify how far you want each location to be from each other in FixedMinDistance and it finds the maximum amount of locations still upholding the subdivision distribution. Example:

{
  "distributionStrategy": {
    "key": "MaxCountByFixedMinDistance",
    "FixedMinDistance": 200
  }
}

Specific country distribution

You can specify a country distribution of your choice. All numbers are relative (not representing location counts) and must be integers. Example:

{
  "countryDistribution": {
    "NO": 200,
    "SE": 220,
    "FI": 200,
    "DK": 140
  }
}

Use vali countries --country "NO,SE,FI,DK" to output the default distribution, so you can copy it and make adjustments suitable to your map.

Specific subdivision distribution(s)

You can specify one or more subdivision distribution to override the default one(s). All numbers are relative (not representing location counts) and must be integers. You can output the default subdivision distribution as JSON with the command vali subdivisions --country "NO" (or vali subdivisions --country "NO" --text to get a textual representation) Example:

{
  "subdivisionDistribution": {
    "NO": {
      "NO-03": 200,
      "NO-15": 1500
    },
    "SE": {
      "SE-AB": 1000,
      "SE-AC": 1500,
      "SE-BD": 2000
    }
  }
}

Use vali subdivisions --country "NO" to output the default distribution, so you can copy it and make adjustments suitable to your map.

Include or exclude subdivisions

Sometimes it can be handy to include or excludle a few subdivisions. F.ex. include only African Spain or exclude European Türkiye. Examples:

{
  "subdivisionInclusions": {
    "ES": [
      "ES-CN",
      "ES-CE",
      "ES-ML"
    ]
  }
}
{
  "subdivisionExclusions": {
    "TR": [
      "TR-22",
      "TR-39",
      "TR-59",
      "TR-34"
    ]
  }
}

Location filtering

Locations can be filtered globally, per country or per subdivision.

Properties

Property Description
Surface [OSM] Surface on the road within a 3m radius.
Buildings10 [OSM] Number of buildings within a 10m radius.
Buildings25 [OSM] Number of buildings within a 25m radius.
Buildings100 [OSM] Number of buildings within a 100m radius.
Buildings200 [OSM] Number of buildings within a 200m radius.
Roads0 [OSM] Number of roads connecting this point. Meant for selecting intersection locations.
Roads10 [OSM] Number of roads within a 10m radius.
Roads25 [OSM] Number of roads within a 25m radius.
Roads50 [OSM] Number of roads within a 50m radius.
Roads100 [OSM] Number of roads within a 100m radius.
Roads200 [OSM] Number of roads within a 200m radius.
Tunnels10 [OSM] Number of tunnels within a 10m radius. Locations are filtered on 0 tunnels by default.
Tunnels200 [OSM] Number of tunnels within a 200m radius.
IsResidential [OSM] Whether any road is marked with landuse="residential" within a 100m radius.
ClosestCoast [OSM] Distance to closest coastline in meters. Only works up to ~10 000 meters. Can be not set (null)
ClosestLake [OSM] Distance to closest lake in meters. Only works up to ~10 000 meters. Can be not set (null).
ClosestRiver [OSM] Distance to closest river in meters. Only works up to ~10 000 meters. Can be not set (null).
ClosestRailway [OSM] Distance to closest railway in meters. Only works up to ~10 000 meters. Can be not set (null).
HighwayType [OSM] Text representing the highway type. See road types below for possible values. If Roads0 is larger than 1 (location is at an intersection) HighwayType can have multiple values, aka HighwayType eq 'Living_street' and HighwayType eq 'Residential' is valid.
Month [Google] The month of the coverage. Integer.
Year [Google] The year of the coverage. Integer.
Lat [Google] The latitude of the location. Number.
Lng [Google] The longitude of the location. Number.
Heading [Google] The default heading of the location. Corresponds to one of the "arrows". Number.
DrivingDirectionAngle [Google] The direction of the front of the Google car. Integer between 0 and 359.
ArrowCount [Google] The number of arrows, mostly corresponds to the possible number of directions you can go in. Integer.
Elevation [Google] Meters above sea level.
DescriptionLength [Google] The length of the "description" field in Google's API. Used to distinguish (estimate) trekker coverage. If you want to include all coverage, add DescriptionLength gt -1 to your respective location filter.
CountryCode [Nominatim] Two character ISO 3166 country code.
SubdivisionCode [Nominatim] ISO 3166-2 code for the subdivision. Mostly corresponding to data available at ISO_3166-2 (but with some exceptions.)
County [Nominatim] Municipality/county name where available.

Operators

Operator Description
eq Equal to
neq Not equal to
lt Less than
lte Less than or equal
gt Greater than
gte Greater than or equal
and Logical AND
or Logical OR
- Minus
+ Plus
/ Divide
* Multiply

Road types

With these building blocks we can write queries/expressions to filter out certain locations. Some examples:

Query Description
Lat gt 66.6 Above the polar circle.
Month gte 9 and Month lte 11 Autumn coverage.
Year lte 2010 Gen2.
Roads200 eq 1 and Buildings200 eq 0 Rural coverage.
ClosestCoast lt 100 Coastal coverage.
Buildings25 gte 3 and Buildings100 gte 6 Urban-ish coverage?
Surface eq 'gravel' or Surface eq 'fine_gravel' Gravel roads?

Filter locations globally

{
  "globalLocationFilter": "ClosestCoast lt 100 and Buildings100 gte 4"
}

Filter locations per country

{
  "countryLocationFilters": {
    "GR": "ClosestCoast lt 100 and Buildings100 gte 4"
  }
}

Filter locations per subdivision

{
  "subdivisionLocationFilters": {
    "GR": {
      "GR-M": "ClosestCoast lt 50",
      "GR-L": "ClosestCoast lt 100"
    }
  }
}

Location preference filtering

Sometimes you want a certain percentage of your map to contain one type of locations. Vali can help you try and achieve that. It's called preference filters and can be applied globally/per country or per subdivision. The filtering is applied after any location filtering as described above. Example showing how to achieve 25 % locations on unpaved roads and filling in the rest with any location:

{
  "globalLocationPreferenceFilters": [
    {
      "expression": "Surface neq 'paved' and Surface neq 'asphalt'",
      "percentage": 25,
      "fill": false
    },
    {
      "expression": "*",
      "percentage": null,
      "fill": true
    }
  ]
}
{
  "countryLocationPreferenceFilters": {
    "ES": [
      {
        "expression": "Surface neq 'paved' and Surface neq 'asphalt'",
        "percentage": 25,
        "fill": false
      },
      {
        "expression": "*",
        "percentage": null,
        "fill": true
      }
    ]
  }
}
{
  "subdivisionLocationPreferenceFilters": {
    "ES": {
      "ES-AN": [
        {
          "expression": "Surface neq 'paved' and Surface neq 'asphalt'",
          "percentage": 25,
          "fill": false
        },
        {
          "expression": "*",
          "percentage": null,
          "fill": true
        }
      ]
    }
  }
}

Location output adjustments

You can adjust the zoom on locations with globalZoom (range 0-3.6) and set the pitch with globalPitch (range -90 to 90). Heading can be set with globalHeadingExpression or countryHeadingExpressions. Examples:

{
  "output": {
    "globalHeadingExpression": "DrivingDirectionAngle + 90"
  }
}
{
  "output": {
    "countryHeadingExpressions": {
      "FR": "DrivingDirectionAngle + 90",
      "GB": "DrivingDirectionAngle + 270"
    }
  }
}
{
  "output": {
    "globalHeadingExpression": "",
    "globalPitch": -30,
    "globalZoom": 2.4
  }
}

Pano verification and selection

Vali offers functionality for verifying each location against Google streetview APIs to ensure no unofficial coverage and/or to select specific/non-default panorama ids. When panoVerificationStrategy is not empty, each location is checked and the specified strategy for pano selection is applied. The resulting map will then have locations with panoId set. Example:

{
  "countryCodes": [
    "LU"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 15000,
    "minMinDistance": 50,
    "treatCountriesAsSingleSubdivision": []
  },
  "globalLocationFilter": "",
  "output": {
    "panoVerificationStrategy": "Newest"
  }
}

Available strategies: Newest, Random, RandomNotNewest, RandomAvoidNewest, RandomNotOldest, RandomAvoidOldest, SecondNewest, Oldest, SecondOldest

Country code expansions

Most places that accept country code, can also accept special keywords that will expand into multiple country codes. Possible values:

Country distributions from famous maps

Default is acw. Possible values for distributionStrategy->countryDistributionFromMap:

Tagging locations

You can tag locations in output.locationTags. Available tags:

Full examples

With all the building blocks described above we can start creating real, serious maps.

A very skewed Europe

{
  "countryCodes": [
    "europe"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 25000,
    "minMinDistance": 200,
    "countryDistributionFromMap": "acw"
  },
  "output": {
    "countryHeadingExpressions": {
      "lefthandtraffic": "DrivingDirectionAngle + 270",
      "righthandtraffic": "DrivingDirectionAngle + 90"
    }
  }
}

A coastal Asia

{
  "countryCodes": [
    "asia"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 25000,
    "minMinDistance": 200,
    "countryDistributionFromMap": "acw"
  },
  "globalLocationFilter": "ClosestCoast lt 100"
}

An Arbitrary Rural World

{
  "countryCodes": [
    "*"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 111000,
    "minMinDistance": 50,
    "countryDistributionFromMap": "aarw",
    "treatCountriesAsSingleSubdivision": ["IL,UG,AE,QA,TN,AD,CW,DO,GH,GL,JO,KG,LA,MK,MT,SN,SG,TW"]
  },
  "globalLocationFilter": "Buildings200 eq 0 and Roads200 eq 1",
  "globalLocationPreferenceFilters": [
    {
      "expression": "Surface eq 'dirt' or Surface eq 'earth' or Surface eq 'fine_gravel' or Surface eq 'grass' or Surface eq 'gravel' or Surface eq 'ground' or Surface eq 'sand'",
      "percentage": 30,
      "locationTag": "dirty",
      "minMinDistance": 500
    },
    {
      "expression": "*",
      "percentage": null,
      "fill": true,
      "locationTag": "fill"
    }
  ],
  "panoIdCountryCodes": [],
  "output": {
    "locationTags": [
      "Year",
      "Month",
      "Season",
      "Elevation500"
    ]
  }
}

An Antenna Focused Gen3 Bulgaria

{
  "countryCodes": [
    "BG"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 12000,
    "minMinDistance": 200
  },
  "globalLocationFilter": "ArrowCount eq 2 and Year gt 2011 and Year lt 2019 and DrivingDirectionAngle neq 0",
  "output": {
    "globalHeadingExpression": "DrivingDirectionAngle + 180",
    "globalPitch": -30
  }
}

A Mostly Non-urban Terminus Hungary

{
  "countryCodes": [
    "HU"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 5000,
    "minMinDistance": 200
  },
  "globalLocationFilter": "ArrowCount eq 1 and Buildings100 eq 0"
}

A Degenerated Oceania

{
  "countryCodes": [
    "oceania"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 10000,
    "minMinDistance": 25
  },
  "countryDistribution": {
    "AS": 10,
    "AU": 10,
    "GU": 10,
    "MP": 10,
    "NZ": 10,
    "PN": 10,
    "US": 10,
    "UM": 10
  }
}

An Off-grid US

{
  "countryCodes": [
    "US"
  ],
  "distributionStrategy": {
    "key": "FixedCountByMaxMinDistance",
    "locationCountGoal": 80000,
    "minMinDistance": 500
  },
  "globalLocationFilter": "(DrivingDirectionAngle gt 10 and Heading gt 10 and DrivingDirectionAngle lt 80 and Heading lt 80) or (DrivingDirectionAngle gt 100 and Heading gt 100 and DrivingDirectionAngle lt 170 and Heading lt 170) or (DrivingDirectionAngle gt 190 and Heading gt 190 and DrivingDirectionAngle lt 260 and Heading lt 260) or (DrivingDirectionAngle gt 280 and Heading gt 280 and DrivingDirectionAngle lt 350 and Heading lt 350)"
}

Please remember

Potentially asked questions

Can you turn it into a webpage?

I don't think so. Currently it's a very resource intensive program. Downloading/processing 15 GB of data is not something a web site is suitable for.

Why is Vali a command line application?

Because I have very little interest in creating a user interface for it, especially before it gains any kind of popularity.

Can you include more locations/roads in country X?

Maybe. The best chance for that is if you generate a JSON or csv with "Filter by minimum distance from locations" set to 1 km with as many locations as possible in country X and send it to me on discord. Only one country per file.

Why are there (relatively) few locations on straight stretches of road?

Each location in the location pool corresponds to an OSM "node". Nodes exist to describe roads and other features, so naturally there will be more locations in places with curves, buildings, other roads etc.

Run in Docker

Create a Dockerfile with the following content:

FROM mcr.microsoft.com/dotnet/sdk:8.0
ENV PATH="${PATH}:/root/.dotnet/tools"

then bulid the image.

docker build . -t vali-image

run it

docker run -it vali-image

and start using Vali as normal

dotnet tool install --global vali
...