Build for other countries

[december1990]

hey there, I managed to build and test the portland-metro project successfully and now I want to build the project for other countries that aren't listed in projects folder, for example build for the great britain, how am I to do that ?

thanks

[missinglink]

Hi @december1990, please don't open multiple issues for the same thing (https://github.com/pelias/pelias/issues/923).

You mentioned in your issue description that you've resolved https://github.com/pelias/docker/issues/279, if that's the case please close the issue.

Regarding your question, you can copy one of the existing project directories and modify it to build any region. There are some values which you'll need to change in the pelias.json file, but the docker-compose.yml file doesn't need to be modified.

in particular the imports.geonames.countryCode, imports.openstreetmap.download.sourceURL (and corresponding imports.openstreetmap. import.filename), import.openaddresses.files, whosonfirst.countryCode and whosonfirst.importPlace variables.

the documentation for this is sadly lacking at the moment but I will try to write a blog post about it to explain what all the settings are and how to configure them.

[JosephKuchar]

Hi,

I thought I'd piggyback on this issue instead of creating another. I'm planning to create a build for Canada, and I had a question about the best practices for incorporating openaddresses data. Several of the examples I've looked at seem to reference local downloads of the OA data as opposed to harvesting from a URL - is this the norm? Canada has something approaching 100 sources in OA, if I recall correctly, so I'm hoping there's an alternative to downloading them all individually.

Thanks, Joseph

[missinglink]

Hi @JosephKuchar unfortunately the code for openaddresses downloads pre-dates the OA team having an API by several years.

It would be nice to modernize the code by having it call their API for a list of sources and then filter by prefix, or whatever.

I started writing something to do exactly that but then got busy with other things, you might be able to use my code to either scrape a list of sources or to open a PR to add support for it.

[missinglink]

const _ = require('lodash');
const OA = require('@openaddresses/lib');
const config = require('pelias-config');

class OpenAddressesAPI {
  constructor() {
    this.config = _.get(config.generate(), 'imports.openaddresses', {});
    this.client = new OA(_.pick(this.config, 'token'));
  }

  // remove file extensions from 'source'
  normalize(source) {
    if (!_.isString(source)) { return source; }
    return source.replace(/\.[^/.]+$/, '')
  }

  // return the http url for a specific job id
  url(job) {
    return `${this.client.url}/api/job/${job}/output/source.geojson.gz`
  }

  // if the 'validated' mode is enabled (for financial supporters only)
  isValidatedModeEnabled() {
    return _.get(this.config, 'validated') === true
  }

  async lookup() {
    // files listed in pelias.json
    const files = _.get(this.config, 'files', []);

    // lookup each source from the remote API
    return await Promise.all(files.map(async (filename) => {

      // normalize 'source' property
      // support the 'validated' property for financial supporters
      const payload = {
        source: this.normalize(filename),
        layer: 'addresses',
        validated: this.isValidatedModeEnabled()
      };

      // request extended info and return the first result
      const versions = await this.client.cmd('data', 'list', payload);
      return _.isArray(versions) && !_.isEmpty(versions) ? _.head(versions) : {};
    }));
  }
}

module.exports = OpenAddressesAPI

[JosephKuchar]

Thanks @missinglink ! I actually ended up writing a short Python script to loop through OpenAddresses's Canadian sources listed on their github, then query those to the OpenAddresses Data and Jobs APIs to collect all the data.

I'm encountering an issue now with the installation process: when I run pelias compose pull I get a permissions error: File "/snap/docker/1779/lib/python3.6/site-packages/dotenv/main.py", line 54, in _get_stream with io.open(self.dotenv_path, encoding=self.encoding) as stream: PermissionError: [Errno 13] Permission denied: './.env'

The drive it's being installed on is generally one that requires root permissions to make modifications to, but I gave myself ownership of the pelias folder and all of its contents, and even after explicitly setting the .env file's permissions to be read and write by everyone this error is persisting. Any ideas?

[missinglink]

pelias compose pull is just a thin wrapper over docker-compose pull.

What happens when you run the latter command? Presumably the same error?

[JosephKuchar]

Aha, thanks - that also failed, and I see now the issue is docker was installed with Snap, so I'll need to just reinstall it properly.

[december1990]

Hi

I want to build geocoding service for great britain, how can i get the right value(s) for whosonfirst.importPlace for great britain ? since pbf file for this area is too large I get an error when running "pelias prepare all" command, so polylines can't be created. do I need polylines and interpolation to use geocoding service ? if yes, after downloading prepared .csv file should I run prepare all again or I can go to the next command ?