Add route geometries

[vgeorge]

Changes:

• Add migration to create a table for the route geometries
• Add seed task to ingest geometries
• Change inbound/outbound routes to include route geometries

How to test:

• Download updated data as described in the README
• Run seed task, which should take a while (30 mins on my M1 machine)
• Start the development server
• Inspect inbound/outbound routes, they should include the geojson features for the routes

@nerik this is ready for a first review.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
food-system-digital-twin	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 13, 2023 0:49am

[nerik]

Yipeee, routes, thanks @vgeorge !

A couple of comments/questions:

1. The routes seed fails at some point on my machine:

   Processed 4820000 features: 4.066s
   Error while executing "/Users/erik/Work/ds/food-twin/seeds/004-add-routes.js" seed: Unexpected end of JSON input
   Error: Error while executing "/Users/erik/Work/ds/food-twin/seeds/004-add-routes.js" seed: Unexpected end of JSON input
   at Seeder._waterfallBatch (/Users/erik/Work/ds/food-twin/node_modules/knex/lib/migrations/seed/Seeder.js:118:23)
   SyntaxError: Unexpected end of JSON input
   at JSON.parse (<anonymous>)
   at /Users/erik/Work/ds/food-twin/seeds/004-add-routes.js:25:28
   at Array.map (<anonymous>)
   at processBatch (/Users/erik/Work/ds/food-twin/seeds/004-add-routes.js:24:36)
   at Object.exports.seed (/Users/erik/Work/ds/food-twin/seeds/004-add-routes.js:63:11)
   at runMicrotasks (<anonymous>)
   at processTicksAndRejections (node:internal/process/task_queues:96:5)
   at async Seeder._waterfallBatch (/Users/erik/Work/ds/food-twin/node_modules/knex/lib/migrations/seed/Seeder.js:115:9)

   The routes table, however, has data at the end of this, and I managed to get them on the map. About 50% do show up, but I don't know if that's due to the seed failing or if the data is incomplete. I can ask tonight whether we are supposed to have all of them.

2. I see that the original geometries come in polyline format. Just out of curiosity, why do we convert them to a PostGIS geometry? Could the polyline stored as plain text be taking less space? (we can easily decode the polyline on the client side)

3. Simplification:

The lines are overly simplified, which generates ugly artefacts when two routes overlap. I don't see any simplification happening on the geometries on our side, but can you confirm? We should ask for (much) less simplified geometries. I'm worried about ingestion time which could increase tenfold. Could using raw polylines be a solution to this?

[vgeorge]

@nerik thanks for the review! It makes sense to use the encoded polyline as we can use it in the frontend. I've changed the migration and fixed the seed workflow. Can you please do another review and update the front-end to use the new format? Steps:

• run ./node_modules/.bin knex migrate:down
• yarn migrate
• yarn seed or ./node_modules/.bin/knex seed:run --specific=004-add-routes.js if you want to update just the routes

About data coverage, it should return what is available in the database, unless there is some error reported in the terminal. I'll do a quick count of O/D pairs and report via chat.

On simplification, the seed process was using the exact same GeoJSON object available in the source file. I'm not sure if with the encoded polyline will be different, I suspect this was deliberate to make the data size more manageable.

[nerik]

Thanks, I'll merge 👍 Can confirm that some routes are missing and that oversimplification is baked into the polylines.