developmentseed / road-planning-datapipeline

Data pipeline for the Road Planning tool
MIT License
0 stars 1 forks source link

AADT segments #14

Open danielfdsilva opened 4 years ago

danielfdsilva commented 4 years ago

Getting the data

From population file remove entries with nulls eg: "properties": { "ID": 55, "NOM": null, "ID_DEP": null, "ID_COM": null, "POP Ville": null }

Preparing the pairs

Run OD generator:

1) All settlements to the closest populated place named: settle-to-pop.json

yarn od generate -m closest populated-places.geojson population.geojson

2) Populated place to other populated places named: pop-to-pop.json

yarn od generate -m closest populated-places.geojson population.geojson

Cleanup data

Problem: since form pop-to-pop.json both origin and destinations are the same we need to remove the same-to-same entries

const fs = require('fs');

const odpairs = JSON.parse(fs.readFileSync('pop-to-pop.json'));

odpairs.pairs = odpairs.pairs.reduce(
  (acc, p) => (p.o !== p.d ? acc.concat(p) : acc),
  []
);

fs.writeFileSync('pop-to-pop_.json', JSON.stringify(odpairs));

Problem: Merging both odpairs since the pairs are index based.

const pop2pop = JSON.parse(fs.readFileSync('pop-to-pop.json')); const settle2pop = JSON.parse(fs.readFileSync('settle-to-pop.json'));

const existingOriginsCount = settle2pop.origins.features.length;

// Add pop2pop origins to settle2pop. settle2pop.origins.features.push(...pop2pop.origins.features); // Add pop2pop pairs updating indexes. const updatedPairs = pop2pop.pairs.map(p => ({ o: p.o + existingOriginsCount, d: p.d }));

settle2pop.pairs.push(...updatedPairs);

fs.writeFileSync('merged.json', JSON.stringify(settle2pop));


### Generate aadt-segments

Follow instructions in https://github.com/developmentseed/road-planning-datapipeline/tree/feature/aadt-segments/aadt-segments

## Result files
Uploaded to `s3://rr-data-haiti/aadt-segments/`

aws s3 ls s3://rr-data-haiti/aadt-segments/ 2020-03-11 11:21:22 113305 error.json 2020-03-11 11:21:22 2693279 odpairs.json 2020-03-11 11:21:23 19107402 result.json

olafveerman commented 4 years ago

This is great, thanks for the detailed documentation @danielfdsilva

Do you want to update this ticket so Xavier can start working on this?