Importing Berlin

[jvolker]

I've used ./importer --oneshot=/ with the binary version and this (extracted) OSM file: http://download.geofabrik.de/europe/germany/berlin-latest.osm.bz2

The importer crashes with Unknown turn restriction no_entry:

- Running convert_osm on /Users/jerry/Downloads/berlin-latest.osm
Read /Users/jerry/Downloads/berlin-latest.osm (1,040)... 162.1816s
OSM doc has 5522394 nodes, 818090 ways, 14293 relations
processing OSM nodes (5,522,394)... 4.4821s
processing OSM ways (818,090)... 6.3400s
Relation 1653527 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 2107105 has unhandled member role inner, ignoring it
thread 'main' panicked at 'Unknown turn restriction no_entry', map_model/src/raw.rs:373:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
dropping Timer while doing progress processing OSM relations, due to panic?

[dabreegster]

The OSM importing code is extremely strict about stuff I haven't seen before, to prevent regressions when converting regularly used maps. This one is easy to fix (edit map_model/src/raw.rs:373) if you've built from source. You're bound to hit a few more issues though. I'll download the Berlin map and fix whatever I encounter.

[dabreegster]

The entire Berlin area is huge. Could we start with a smaller area? Not neighborhood small, maybe the general city center area, just not 1.1GB. :) I use geojson.io or geoman.io to draw polygon boundaries.

[jvolker]

Wow, thanks for doing this.

Sorry. I didn't know how much this file covered.

Could we start with a smaller area?

Of course, how about this for the core city centre:

"coordinates": [
          [
            [
              13.361434936523436,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.49532344352079
            ]
          ]
        ]

and this for a larger part of the city which is surrounded by the circle train:

"coordinates": [
          [
            [
              13.271141052246094,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.45935663683681
            ]
          ]
        ]

I can try to create those filter files.

[jvolker]

Filter files: berlin-polys.zip

I've tried another time with berlin-centre.poly but am still getting the same error.

[dabreegster]

Thanks! I got the large one working. You can put this file in data/system/maps: https://www.dropbox.com/s/4uia1pk9aj1d8gd/berlin.bin?dl=0 My GPU half-melted during loading and my mouse was briefly unresponsive, but in the end, only took 20s to load. :)

I'll commit a fix in a moment that deals with almost all of the issues. There are a few intersections where seemingly different combinations of roads wind up with exactly the same turn geometry, which breaks some assertions. I want to figure out why that's happening instead of relaxing that code permanently.

[dabreegster]

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat. It's been quite painful mapping them in Seattle. Odd that the coverage in Berlin is so spotty; IIRC from a prior visit, there's definitely more than just these areas

[jvolker]

Wow! That's amazing. Thanks a lot for your quick help.

Unfortunately, abstreet is crashing when trying to load the Berlin map. output.txt contains:

Loading map ../data/system/maps/berlin.bin

Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0005s

../data/system/maps/berlin.bin is missing or corrupt. Check https://github.com/dabreegster/abstreet/blob/master/docs/dev.md and file an issue if you have trouble.

invalid value: integer `512`, expected variant index 0 <= i < 8

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat.

That's great!

[dabreegster]

I found what I think is a problem with Bayernring in OSM:

• https://www.openstreetmap.org/way/722013503
• https://www.openstreetmap.org/way/722013169 The first one is an overlapping road of the second. I think the second one needs to be split at the point where the road attributes are different. Are you part of the OSM community in Berlin? If not, I can attempt a fix or raise the issue on the appropriate slack.

The crash is because the binary map format has changed since Sunday. That's the price of rapid iteration. :( If you can build the game crate from source, it'll handle the new format. If not, https://github.com/dabreegster/abstreet/suites/866601914/artifacts/10031908 is a Mac binary from last night that should work

[jvolker]

The new Mac binary is working with the Berlin map. Fantastic! Thanks so much.

Are you part of the OSM community in Berlin?

No sorry, I'm not. But I'm part of CityLAB Berlin. I should introduce myself properly in the next few days. And maybe my colleagues can answer some of your questions on the Berlin data. Let me know if you have any more questions.

Thanks so much! 🙏 That was the quickest support.

[jvolker]

About the parking data, my colleague @Lisa-Stubert mentioned an official data set from 2014 that contains this data in some format. It might be useful to fill in the gaps on OSM.

FIS-Broker > Straßenbefahrung > Parkfläche > WFS address

The data set contains polygons. @Lisa-Stubert mentioned we might need lines instead.

[dabreegster]

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?

Seattle has a perhaps similar dataset, blockface. I'm matching that to the nearest side of each road here. We could do something similar for this dataset.

Ideally the data winds up in OSM, so anybody can use it, and it can be maintained more easily. OSM has strict rules around importing data automatically. Regardless of an import, theres a tool in A/B Street to update parking tags in OSM a little more conveniently than other editors.

[jvolker]

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?

I'm not familiar with the workflow yet, but a former colleague has written this guide on how to use that data source: https://lab.technologiestiftung-berlin.de/projects/fisbroker-to-qgis/en/

Apparently you can also load it via command line (with GDAL installed): ogr2ogr -f gpkg Parkflächen.gpkg WFS:"https://fbinter.stadt-berlin.de/fb/wfs/data/senstadt/s_Parkflaeche"

I'm going to get back to this next week.

[Lisa-Stubert]

https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=k_StraDa@senstadt&bbox=383640,5815972,386377,5817593 this is the direct link to the data set of the "Straßenbefahrung". It contains all possible areas and objects of road traffic for Berlin. The dark grey areas are the parking areas. They can only be downloaded via the WFS service @jvolker posted.

[jvolker]

I've just tried to load the berlin map in A/B street compiled from source on Windows and received this error:

switch map...
Wrote ../data/player/camera_state/montlake.json
load map...
Loading map ../data/system/maps/berlin.bin
Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0001smemory allocation of 14934484003840414750 bytes failederror: process didn't exit successfully: `C:\Users\PC\Desktop\AB-Street\abstreet\target\release\game.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

Do you have any idea what could cause this? Thanks.

[dabreegster]

The binary format of the map changed yesterday.

I can regenerate the Berlin map and give you a new dropbox link in the next ~30 mins

[dabreegster]

https://www.dropbox.com/s/2l2muvha0dubuvp/berlin.bin?dl=0

The simulation crashes 3m58s in. I can investigate soon; I suspect this is the same as https://github.com/dabreegster/abstreet/issues/143#issuecomment-654488564 -- there are some bus stops mis-tagged in OSM. I'll improve the error message to find these better, or filter out the invalid ones.

[dabreegster]

Fixed Berlin map: https://www.dropbox.com/s/2l2muvha0dubuvp/berlin.bin?dl=0

The possible bus route issues:

Route Bus M76: U Walther-Schreiber-Platz <=> S Lichtenrade has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 43238974, i1: OriginalIntersection { osm_node_id: 27433695 }, i2: OriginalIntersection { osm_node_id: 27434791 } }
Route Bus 277: S+U Hermannstraße <=> Marienfelde, Stadtrandsiedlung has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 391673769, i1: OriginalIntersection { osm_node_id: 3726792172 }, i2: OriginalIntersection { osm_node_id: 3708789255 } }
Route Bus N50: U Tierpark <=> Buchholz-West/Hugenottenplatz has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 626027608, i1: OriginalIntersection { osm_node_id: 5910363710 }, i2: OriginalIntersection { osm_node_id: 2519032893 } }
Route Bus X54: S+U Pankow <=> U Hellersdorf has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 4783453, i1: OriginalIntersection { osm_node_id: 30625193 }, i2: OriginalIntersection { osm_node_id: 2101949795 } }
Route Bus N5: S+U Alexanderplatz <=> Hellersdorf, Riesaer Straße has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 139877314, i1: OriginalIntersection { osm_node_id: 270718978 }, i2: OriginalIntersection { osm_node_id: 7312391564 } }
Route Buslinie M29 has two bus stops seemingly out of order somewhere on OriginalRoad { osm_way_id: 451544125, i1: OriginalIntersection { osm_node_id: 294494421 }, i2: OriginalIntersection { osm_node_id: 845075600 } }

These stops are just skipped for now. Not high priority to figure out what's wrong here.

[jvolker]

Thanks. It's working again.

[dabreegster]

Just checking, are you blocked on me for anything besides importing and matching parking data? (which I'm not likely to get to anytime super soon)

[jvolker]

Thanks for asking @dabreegster. It seems the most important thing is to get trip data to get a more realistic simulation. We are currently looking into data sources and are going to post them here.

[tori-d]

Hey Dustin!

So we've done some brainstorming and come up with some additional data sources that might be useful for the project. Maybe you can take a look and give us an indication which data sources you see as more or less useful, or what's still missing?

The biggest thing we don't have right now is any sort of traffic pattern model like the SoundCast model. It's unclear to me to what extent this data exists in Berlin at all, and to the extent it does, it's not available as open data and we currently lack the contacts to rustle up that data. Obviously we'll keep exploring that, because it's something we're very interested in, but I don't know how soon we would have something concrete to offer there, if that even happens at all.

So what do we have?

• Public transit data (GTFS): https://daten.berlin.de/datensaetze/vbb-fahrplandaten-gtfs (available as a regularly-updated upload; there is also an API, but it requires first contacting the local transit operator to get access)
• Car traffic speed data: https://movement.uber.com/explore/berlin/speeds/query?dt%5Btpb%5D=ALL_DAY&dt%5Bwd;%5D=1,2,3,4,5,6,7&dt%5Bdr%5D%5Bsd%5D=2020-03-01&dt%5Bdr%5D%5Bed%5D=2020-03-31&ff=&lang=en-US (this is a dataset made available by Uber; shows average speeds over various road segments in both directions – could be useful as a proxy for traffic flows?)
• Traffic volume data: https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=wmsk_07_01verkmeng2014@senstadt&bbox=388185,5818529,395105,5822530 (This is a very general dataset, available via WFS, showing estimated number of vehicles driving over major streets in a 24 hour period – this data is based on traffic counts originally gathered in 2014, so it's pretty outdated)

What might we have?

I looked at the dataset containing information on "blockfaces" to see what it contained (wasn't familiar with this term myself). Seems like it includes data on things like zones (e.g. loading and unloading zones, no parking zones, etc.) as well as metered parking/parking fees. Not sure what out of the blockfaces dataset is most relevant for A/B Street – if it's just about knowing where there is streetside parking, then the aforementioned "Straßenbefahrung" dataset is probably our best bet there. If you also want information on no-parking zones etc., I think we can possibly also get those out of the Straßenbefahrung dataset, but I'm not certain. We definitely won't be to get get good data on where parking is metered (and under what circumstances); there isn't currently a comprehensive city-wide dataset for that.

We may be able to rustle up some data on off-street parking – @Lisa-Stubert wanted to check out another large geospatial dataset we have to see if parking lots and parking garages are in it (although if they are, my guess is we won't have data on capacities, but maybe I'll be proven wrong).

Let me know what questions you have. If there's an acute need for data that has not yet been addressed, also let me know about that and we can go sleuthing...

[dabreegster]

Thanks for doing this research!

• The GTFS upload will be useful eventually. The transit modelling needs lots of work first; currently just a single bus is created per route, and it runs all day. No need for API access; the schedule information is used when building the map as a one-time process.
• Speed and traffic volume could be helpful. I've seen research papers before talking about how to use volume data to infer a set of start/end points that would cause the measured volume counts. I have no familiarity with the algorithms for doing this; it'd be future work to implement them or find an existing tool that does it.
• Straßenbefahrung: It sounds like this is the equivalent of the Seattle blockface data indeed. We could use it just to infer the existence of space along the shoulder of the road for street parking. The data in Seattle wasn't adequate; it says "no parking restrictions" on narrow roads that physically had no space for parking. So I use it as a first pass, manually look at areas of the map with odd road geometry, then fix upstream in OpenStreetMap using a tool.
• There's no modelling of metered parking, resident-only restrictions, 2 hours max, etc yet. No immediate plans to figure this out either.
• Offstreet parking with garages/lots could be helpful. We can estimate capacity from the geometry. Some lots are already in OSM: https://www.openstreetmap.org/way/669540789

I think the biggest thing that would help generate better trip data would be census. In the US, there are census tracts like this that chop up the area into pretty small regions, then assign a bunch of demographics to each tract -- mainly a population count. Then there's the American Commuter Survey that has questions like "how many cars owned per household" and "how many trips > 5 miles taken per day." The sample size is probably really small, but it's better than nothing, and I think the survey results are correlated with census tracts. So from this, the really simple trip generation model would take the number of people inside a tract, randomly distribute them to houses in that area (using OSM's residential vs commercial building tagging), guess a workplace/school/common destination for them using stats on how far people in that area travel, and so on.

I see https://en.wikipedia.org/wiki/Demographics_of_Berlin, and drilling down, https://en.wikipedia.org/wiki/Tiergarten,_Berlin is a small area with a population of ~12.5k. So I think we could use this! I haven't looked for where the wiki article pulls data; if there's a more consolidated format that ideally has the GPS coordinates of each neighborhood with this population count, that'd be a huge step forward.

[tori-d]

With respect to the Census Tracts / finding a corollary in Berlin:

So there are various administrative units that Berlin as a city is broken up into. A coworker actually did a nice breakdown of them (in English) here: https://lab.technologiestiftung-berlin.de/projects/spatial-units/en/. The example you found, Tiergarten, is an example of an Ortsteil/locality. Of the top of my head, I'm not actually sure what the smallest possible unit is (there are a few different systems for breaking up the city that are used in different concepts, so it's not necessarily straightforward to arrange all of the units from smallest to biggest. I guess what we want is the smallest possible unit for which we also have demographic data?

I think the smallest unit for which we would have overall population data is a Planungsraum/Planning Area. The population data is available here: https://daten.berlin.de/datensaetze/einwohnerinnen-und-einwohner-berlin-lor-planungsr%C3%A4umen-am-31122018

We also have the geometry of the Planungsräume available in various geospatial formats: https://data.technologiestiftung-berlin.de/dataset/lor_planungsgraeume/en (I think this file is slightly outdated though, we'll have to update it first).

I don't know if we'll be able to find additional census-like data for that level of specificity, however. Within Berlin itself I haven't seen any sort of survey data around mobility habits.

@Lisa-Stubert found this dataset, which probably has what we want? https://daten.clearingstelle-verkehr.de/279/ It includes a regional level, where an area is broken up into a grid of squares that are at least 500x500m and have at least 500 residents in them. The data, as far as I can tell, would include information on the average trip length from a given cell, whether the people living there own cars, etc. Unfortunately, it is not available as open data and I suspect it is only available for a fee. Moreover, the data seems like it is likely to be fairly complex, so even if we were to acquire it, I think it might be beyond our resources at least to integrate it into the program.

I can keep poking around to see if I find anything else at the Berlin level, but at least right now, we definitely have population data for sub-units of Berlin.

[dabreegster]

The spatial breakdown article is quite helpful! Thanks for finding the datasets. I think these will work for figuring out how many residents to assign to buildings. Would you mind sanity checking my interpretation of the population CSV -- Google Translate seems fine, but you never know. I found Körnerstraße in the KML, with spatial_name of 01011104. That matches RAUMID in the population csv. The E_E column is total residents, so that's 4759 in that area, right?

The 500x500m trip length dataset may be helpful, but if it's not open, not a priority.

Here's a proposal for some next steps:

• I'll add Berlin to the set of maps I regularly convert when the map format changes. The larger boundary suggested here turned out a bit too big, so I'll start with the city core. If anyone wants to use geojson.io or geoman.io to draw multiple boundaries, just send the coordinates along. The regions can overlap slightly, they don't have to cover everything (so not a strict partition in the mathematical set sense) -- an example is:

• I'll make the import pipeline grab the shapefile and population CSV, then emit a file that'll let us visualize the regions with the population count.

• I'll start researching approaches to reasonably distribute the number of people in a region into individual buildings. @matkoniecz came up with some heuristics for guessing building sizes using OSM tags.

[dabreegster]

First two steps done. The importer grabs the shapefile and the population CSV, then adds a num_residents attribute to the shapefile. Here's a way to quickly visualize it:

I've started including Berlin in the default binary release, because the city center is tiny (only adds 20MB). This'll hopefully make it easier to iterate without any of you needing to build from source or run the updater. I'll post the build here when it's done; github is having partial outages right now.

[dabreegster]

Linux: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329941 Mac: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329942 Windows: https://github.com/dabreegster/abstreet/suites/924669830/artifacts/11322491

To follow the video above, you'll also need https://www.dropbox.com/s/pbovhfr6zw0s0ob/planning_areas.bin.zip?dl=0. Unzip so that data/input/berlin/planning_areas.bin exists

[tori-d]

The spatial breakdown article is quite helpful! Thanks for finding the datasets. I think these will work for figuring out how many residents to assign to buildings. Would you mind sanity checking my interpretation of the population CSV -- Google Translate seems fine, but you never know. I found Körnerstraße in the KML, with spatial_name of 01011104. That matches RAUMID in the population csv. The E_E column is total residents, so that's 4759 in that area, right?

Yep, this is the correct interpretation!

[jvolker]

Thanks a lot, @dabreegster.

Linux: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329941 Mac: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329942 Windows: https://github.com/dabreegster/abstreet/suites/924669830/artifacts/11322491

Those versions have a "trips between home and work" traffic setting. Is this already using the population count?

[jvolker]

Two quick observations:

• I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.
• The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

[matkoniecz]

I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.

Are there also unusually quiet areas? (for now Kraków has some weird hot spots, especially pedestrian hot spots and some unusually quiet areas, mostly as indirect results of issues that I already reported)

The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

For me it is also consuming plenty of resources with laggy simulation. Not sure whatever it is simply result of many trips, or is there something about traffic generator that triggers unusually poor performance.

[dabreegster]

Those versions have a "trips between home and work" traffic setting. Is this already using the population count?

Not yet, it was added a little later. I'll regenerate the build and post new links soon, unless you get the Rust compiler updated successfully first.

I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.

I'm seeing quite a few possibly unusual patterns. In the morning rush hour, there are 20,000 pedestrian trips and climbing, but under 1000 drivers. This is the logic used to pick driving/walking/biking/transit. If you had to guess a trip mode in Berlin based on any factors you can imagine, what kind of heuristics would make sense?

Some of the weird issues include cars backing up at Torstraße because of pedestrians getting the right-of-way at crosswalks here. Likely a bug with the stop sign policy, and also exasperated by A/B Street inferring crosswalks at all possible crossings and ignoring data in OSM right now.

There's also a massive number of residents matched to a single building in the SW corner. From the population layer:

I think it's because the Nollendorfplatz area has 16,000 residents, but the polygon is mostly out-of-bounds. Maybe it'd be better to just linearly scale the number of residents based on the percent area of the polygon overlapping with the map boundary.

The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

It's more that the number of residents is huge -- 360,000 -- and the trip generation tries to make everybody experience a morning and afternoon rush. There are a number of simulation issues also causing problems, like the stop sign / crosswalk problem I mentioned. And I have never run with so many pedestrians at once; I'll look into profiling and optimizing code dealing with them.

[matkoniecz]

in the morning rush hour, there are 20,000 pedestrian trips and climbing, but under 1000 drivers. This is the logic used to pick driving/walking/biking/transit.

I noticed something similar in Kraków, I attributed it to transit/drivers trip spawns failing for some reason, what seems supported by logs. That is why I keep reporting issues blocking bus/car/bicycle travel.

In addition in Kraków many car trips fail because there was no space to park the car (during mapgen), maybe the same happens here. And there are some degenerate trips like "walk across map, drive 10 meters, walk across map back to the start" that inflate number of pedestians.

[jvolker]

unless you get the Rust compiler updated successfully first.

Got the game compiled but the city center map is gone. How can I reimport it? import.sh --map berlin seems to quit unsuccessfully.

[jvolker]

Are there also unusually quiet areas?

The area I've mentioned is unusually quiet in A/B street compared to what I've seen in reality. Could it be because it's relatively close to the border? How are the work targets generated?

Some of the weird issues include cars backing up at Torstraße because of pedestrians getting the right-of-way at crosswalks here. Likely a bug with the stop sign policy

That particular crossing does have traffic lights. Stop signs are only relevant in case the traffic lights are out of order.

[dabreegster]

Got the game compiled but the city center map is gone. How can I reimport it? import.sh --map berlin seems to quit unsuccessfully.

Try ./import.sh --raw --map --city=berlin. Confusingly, the city is always assumed to be seattle, unless you override it. I'll make the tool smarter and realize that berlin_center implies a city.

Could it be because it's relatively close to the border? How are the work targets generated?

Possibly the number of residents matched to the area is low or there aren't enough workplaces around. (layer > amenities will show you those.) I can dig in later today.

That particular crossing does have traffic lights. Stop signs are only relevant in case the traffic lights are out of order.

https://www.openstreetmap.org/node/3792544915 Ah, there are two styles for tagging traffic signals. This is using the more detailed representation showing where the lights are for each incoming road. I can add support for this soon.

[jvolker]

This is the logic used to pick driving/walking/biking/transit.

Trying to understand the logic:

• What does "transit" mean in that case? Public transport?
• What distance applies to the following?

  if rng.gen_bool(0.005) {
      // low chance for really, really dedicated cyclists
      return TripMode::Bike;
  }
  if rng.gen_bool(0.3) {
      // try transit if available, will
      // degrade into walk if not available
      return TripMode::Transit;
  }

We'll see what data we have for it. Otherwise we could make a guess.

If I understand correctly, currently:

• there are no car or public transport trips for distances shorter than 3miles/4.8km? I think there would be.
• 0.5% would cycle if the trip was further than 3miles/4.8km? I think it would be more.

[dabreegster]

"Transit" is public transport, yes.

The distance is measured by the walking route. Sometimes the distance could be much longer or shorter by driving, but usually this is an OK proxy.

Your understanding about the rest seems correct. This is a function we've kind of made up out of vague intuitions, and tuned for Kraków. Mode choice is a complicated topic I don't have an understanding of yet. If you have alternative ideas or come across any academic papers that are easy to understand, implementation is the easy part.

[jvolker]

Try ./import.sh --raw --map --city=berlin.

I think, it's still the same unfortunately. Another windows opens, displays some console output and closes again. In A/B street Berlin is still not visible.

[dabreegster]

You're running from the git repo, not the binary release, right? I don't understand why a second terminal is opening; are you typing that command into a first mac terminal? You could add > log.txt at the end to redirect output to a file, if that helps.

[jvolker]

You're running from the git repo, not the binary release, right?

Yep.

are you typing that command into a first mac terminal?

No, it's a PC running Windows.

You could add > log.txt at the end to redirect output to a file, if that helps.

The file is empty, unfortunately.

[jvolker]

[matkoniecz]

Maybe 2> log.txt (stdeer capture) will work?

[jvolker]

Maybe 2> log.txt (stdeer capture) will work?

No, nothing, unfortunately.

[matkoniecz]

This is a function we've kind of made up out of vague intuitions, and tuned for Kraków

"tuned" is overstatement, it was more of "AB Street is no longer crashing and there are some pedestrians, cars and cyclists" and "problem of bicyclist traffic jams, with bicycle traffic over 100 times reality, is reduced to 5 times reality" (it is not overstatement, bicycle traffic in one iteration was so large that it caused nearly instant city-wide gridlock).

Feel free to change numbers or tweak it based solely on Berlin, do not worry about Krakow (though if you want you may test it also there).

[dabreegster]

Ah, maybe running a separate shell script on Windows launches a new terminal. Try:

RUST_BACKTRACE=1 cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin

I'm not sure about Windows syntax, but you could also try . import.sh --raw --map --city=berlin. The leading . means run all of the commands in a file, but ignore the first line that says what shell to execute with.

Also I noticed you're doing --city=berlin_center; it should be --city=berlin. The difference isn't interesting when a city only has one map, but we'll probably split into multiple maps at some point, like Seattle. The city name is berlin and the only map there is called berlin_center, currently.

[jvolker]

Thanks, the following has worked:

set RUST_BACKTRACE=1 && cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin

But I'm now getting the following errors:

   Compiling importer v0.1.0 (C:\Users\PC\Desktop\AB-Street\abstreet\importer)
warning: unreachable expression
  --> importer\src\main.rs:96:13
   |
91 | /             panic!(
92 | |                 "Can't do --scenario or --scenario_everyone without the scenarios feature \
93 | |                  compiled in"
94 | |             );
   | |______________- any code following this expression is unreachable
95 |               // Nonsense to make the type-checker work
96 |               (Some(true), Some(true))
   |               ^^^^^^^^^^^^^^^^^^^^^^^^ unreachable expression
   |
   = note: `#[warn(unreachable_code)]` on by default

warning: unused variable: `maybe_popdat`
  --> importer\src\main.rs:80:10
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |          ^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_popdat`
   |
   = note: `#[warn(unused_variables)]` on by default

warning: unused variable: `maybe_huge_map`
  --> importer\src\main.rs:80:24
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |                        ^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_huge_map`
warning: unused variable: `maybe_map`
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_map`

warning: variable does not need to be mutable
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ----^^^^^^^^^
    |             |
    |             help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: function is never used: `adjust_private_parking`
   --> importer\src\seattle.rs:116:8
    |
116 | pub fn adjust_private_parking(map: &mut Map, scenario: &Scenario) {
    |        ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: 6 warnings emitted

    Finished release [optimized] target(s) in 6m 27s
     Running `target\release\importer.exe --raw --map --city=berlin`
- Working on all berlin maps
import map data...
- Missing data/input/berlin/osm/berlin-latest.osm.pbf, so downloading http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf
- Running "curl" "--fail" "-L" "-o" "tmp_output" "http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57.2M  100 57.2M    0     0  11.4M      0  0:00:05  0:00:05 --:--:-- 11.1M
- Running "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf"
thread 'main' panicked at 'Failed to run "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf": Os { code: 2, kind: NotFound, message: "Das System kann die angegebene Datei nicht finden." }', importer\src\utils.rs:132:13
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.46\src\backtrace\mod.rs:66
   1: std::sys_common::backtrace::_print_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:78
   2: std::sys_common::backtrace::_print::{{impl}}::fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:59
   3: core::fmt::write
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libcore\fmt\mod.rs:1076
   4: std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\io\mod.rs:1537
   5: std::sys_common::backtrace::_print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:62
   6: std::sys_common::backtrace::print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:49
   7: std::panicking::default_hook::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:198
   8: std::panicking::default_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:218
   9: std::panicking::rust_panic_with_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:486
  10: std::panicking::begin_panic_handler
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:388
  11: std::panicking::begin_panic_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:342
  12: importer::utils::osmconvert
  13: importer::utils::download
  14: importer::berlin::osm_to_raw
  15: alloc::collections::btree::search::search_tree
  16: csv::reader::Reader<R>::deserialize
  17: std::rt::lang_start_internal::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:52
  18: std::panicking::try::do_call
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:297
  19: std::panicking::try
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:274
  20: std::panic::catch_unwind
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panic.rs:394
  21: std::rt::lang_start_internal
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:51
  22: main
  23: invoke_main
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  24: __scrt_common_main_seh
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  25: BaseThreadInitThunk
  26: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
import map data took 5.4766s

- import map data took 5.4766s

- import map data took 5.4766s
error: process didn't exit successfully: `target\release\importer.exe --raw --map --city=berlin` (exit code: 101)

[dabreegster]

You can ignore the warnings. It tried to execute this command: mv tmp_output data/input/berlin/osm/berlin-latest.osm.pbf. Looks like Windows calls it move, not mv.

I'll switch importer to use system calls instead of relying on Linux/Mac commands when possible. Not going to get to that for a few hours. As a temporary workaround, you could edit importer/src/utils.rs, look for all Command::new, and if you get an error about one of them, change it appropriately. I think mv becomes move, cp -> copy, and rm -> del (possibly messing with the -rfv flags).

[dabreegster]

Number of people in houses around here down from 360k to 180k, after adjusting for the planning areas that only partly overlapped the map. Does this seem generally more right to you?

[tori-d]

A couple additional data sources to throw in there:

• I found this study published in 2018 that I think actually gets at this census-level data you mentioned before, @dabreegster. Basically one of the universities here did a survey of 40,000 Berliners in which they were asked about their mobility habits (e.g. modal split, types of trips, etc.). They have detailed reports for each of Berlin's 12 districts/Bezirke. However, there is currently no open data available, just PDFs. We could try to reach out to this university however and see if they would be interested in sharing their data, perhaps they would find our project interesting: https://www.berlin.de/sen/uvk/verkehr/verkehrsdaten/zahlen-und-fakten/mobilitaet-in-staedten-srv-2018/

• For bikes, we do have open data from the automatic counting stations the city has. The table is available here: https://www.berlin.de/sen/uvk/verkehr/verkehrsplanung/radverkehr/weitere-radinfrastruktur/zaehlstellen-und-fahrradbarometer/#dauer (if you scroll there is an Excel file). It shows the hourly counts for each station over several years. Using that, we could at least get pretty accurate counts of how many bikers are going past a given point in the city on a given day / at a given time (there are 17 stations across the city). We did an analysis with the data up to 2017 a couple years ago, so we do have some already have some cleaned up versions of this data that theoretically could be used as a basis? Not sure what data specifically would be most helpful to have here though (e.g.: should the hourly data be aggregated to a certain level for each location?)

[dabreegster]

The travel surveys look really promising! Right now, our approach knows roughly how many people reside in each building, but we don't know how far they travel, or how. If we could put together a table of (origin district, destination district, percentage of bike trips, pct of driving, ...), then we could pretty easily do something more realistic. I can't understand the PDFs too easily, but if they have this quadratic table of trips between every pair of districts, 144 things isn't unreasonable to manually copy from a PDF.

Bike counts may be helpful, but I don't know how to turn the data into individual trips. This is a sample paper describing how to do it. It's way beyond my current ability to understand.

The more I read about https://activitysim.github.io/activitysim/index.html, the more I'm convinced this could be the more trustworthy way forward. Seattle's Soundcast model makes use of this. (Edit: just kidding, they use DaySim.) Glancing through the amount of detail in the different parts of their model, there's no hope of reproducing this level of rigor. I think A/B Street ought to require trip data as input. We can do extremely simple approximations that just look at OpenStreetMap data and maybe some extra census counts, but I don't know if it's worth trying to recreate what ActivitySim has done.