Build ABMs with OpenStreetMapX.jl and Agents.jl

[fbanning]

Hi,

I would like to use OSM map data for some agent-based models and was wondering if it is possible to utilise the functionality of OpenStreetMapX.jl for this. I saw that the maintainers have created a simple Zombie apocalypse model for their JuliaCon 2020 example. This has been very helpful for me to understand the general idea how to build an ABM in combination with OpenStreetMapX.jl and plotting it via folium (such a nice tool btw!).

However, I would prefer to use the Agents.jl framework for building ABMs and would like to ask whether or not it is possible to use the functionality of those packages together? I've been tinkering around with it but couldn't get the desired outcome so far.

My aim would be to integrate the MapData generated by OpenStreetMapX.jl into the space component (GraphSpace) of a model created with Agents.jl. After some agents have been added within the bounds of MapData, I would then imagine to have two separate networks - one for the agents and one for the underlying map on which the agents move. When plotting the model, both graph networks could be accessed by Agents.jl and plotted normally (respecting bounds, relative location, etc.).

Would this be at all feasible? I'm looking forward to hear your thoughts on this. Maybe it's way too complex. Maybe it's really easy and I just miss the forest for the trees. (Sidenote: I have also asked this question in the issues section of the OpenStreetMapX.jl repo because I am unsure who to ask regarding this topic.)

Best wishes

[Datseris]

Hi there,

I thought of exactly the same thing when I saw the talk, and I absolutely think we can get an interface between the two. I have already contacted @pszufe , which I hope to motivate, because I believe we will need some help on the side of OpenStreetMapX.jl (which I know nothing about).

The process to do this interfacing amounts to defining a new type of space for Agents.jl, which uses internally the mechanics of OpenStreetMapX.jl . Once we have defined this space, and fundamental methods like move_agent! and add_agent!, the rest of the interface of Agents.jl will "just work" for agents moving around in this open street map space.

Give me some time, and I will "define" the space API that Agents.jl requires (everything is already defined in the source code, but a high level developer documentation will greatly help).

[Datseris]

@fbanning can you please cross-link the issue on the other repo?

[kavir1698]

Hi,

This integration would be interesting. I have never used OSM, but just taking a look at the example you posted, I think the integration would be straight forward. The new space type in Agents can be the MapData object, which exports a directed graph that is already supported by Agents. As George said, we would need to add new methods for agent addition and movement, etc., and the rest should just work.

[fbanning]

This is the issue I've opened on the OpenStreetMapX.jl repo: https://github.com/pszufe/OpenStreetMapX.jl/issues/31

MapData object, which exports a directed graph that is already supported by Agents

Those were exactly my thoughts. I played around with a simple example but didn't get it to work properly, so I thought I might ask here. The MapData structure is pretty straight-forward to understand, even for a Julia beginner like me. Have a look at the bottom of this file.

[Libbum]

I think this would be a good opportunity to design a "Custom Space" API, which allows for extensions like this in the future to be constructed with little fanfare

Would be simple enough, just outlines what the minimal span of interfacing functions a new space must provide to correctly and completely integrate with Agents is.

[pszufe]

The field g of a MapData is actually a directed LightGraphs graph and the rest of a MapData object can be considered to be metadata for the graph g. If Agents.jl supports simulation on graph - perhaps even no code changes is required for the integration. I do not know the API of Agents.jl so still need to have a look though.

[Libbum]

Yeah, from a quick skim of your step! function in the Pandemic_Broadcasting_Sim.ipynb, it looks like a lot of that is just doing what Agents would do. Once at least one of us gets the time to sit down and understand the API of the others package, this might even be as trivial as a new integration example!

[Datseris]

If Agents.jl supports simulation on graph

Yeap, it does, directly, see for example https://juliadynamics.github.io/Agents.jl/stable/tutorial/#Agents.GraphSpace or https://juliadynamics.github.io/Agents.jl/stable/examples/sir/ for an actual implementation.

[Datseris]

Alright, I now realize that in fact, there isn't that much necessary for a "space API". Here are the necessary things one has to do, to make an "oficial" space of open street map:

• Create type OpenStreetMapSpace. While doing so, decide on a concrete version of the pos field of the agents for that space.

Then, add methods for the following functions:

• add_agent_pos!(agent::AbstractAgent, model::ABM)
• A function that gives a "random position" in the space, whatever that position may be.
• move_agent!(agent::A [, pos], model::ABM{A, <: OpenStreetMapSpace}). Notice that here we cannot just "move" agents on the graph, we have to take into account that different nodes have different physical distances between them.
• kill_agent!(agent::AbstractAgent, model::ABM)
• space_neighbors(position, model::ABM, r)

The last method is the hardest, and requires a clever design of the space structure and how it keeps track of where agents are. For DiscreteSpace we just have a vector of ids for each grid cell. For ContinuousSpace we have an SQL database. For the MapSpace I think you should have a look again at the talk of Przemysław Szufel and see how he found "agent proximity" in his simulations.

Once this space is defined, it would be nice to write a plotting function for the agents in the space in an example file, that uses this Python library.

---

p.s.: If you want to do this PR, it would be great! I will contribute as well, because I realize that in fact, we only need to define one time add_agent!(agent::AbstractAgent [, position], model::ABM) (which creates the agent) and this can be re-used verbatim for all spaces.

[pszufe]

Basically when using OpenStreetMapX agents are on graph. They can be on vertices, however in many urban applications they are moving along edges and can be in any continuous point on that edge. This situation occurs still in a discrete-event-simulation. You model time from event to event, but depending on space definition some agents on that event will not be at a vertex location. Since the main structure of the data is a LightGraphs directed graph LightGraphs.neighbors and LightGraphs.all_neighbors will be our best friends, but in some applications it could not be sufficient - there are typical issues where modelling a transportation system (vehicle length, route throughoutput, multiple lanes etc.) Perhaps we could discuss this on Skype?

BTW do you have any video tutorial of your library showing a typical application. While Agents.jl docs are quite good a more strategic view could help to decide on the best integration pattern.

[Datseris]

and can be in any continuous point on that edge.

Yeap, that is the core difficulty, and the reason we cannot simply re-use the existing GraphSpace.

Perhaps we could discuss this on Skype?

That is fine with me, but we should first find someone that is willing to implement this (because I don't think I will have the time).

BTW do you have any video tutorial of your library showing a typical application. While Agents.jl docs are quite good a more strategic view could help to decide on the best integration pattern.

No not yet, but I think what you're searching for here is some kind of developer's documentation, which actually Agents.jl doesn't have any of... I will be making videos for all JuliaDynamics packages in the near future, but they will be much more so for the user perspective.

[pszufe]

@Datseris perhaps I could implement it or simply have a PhD student implement it. However, I need to understand strategic ideas behind Agents.jl so it feels like it matches to whatever is already in library. I used to use ABM libraries in other languages (e.g. MASON in Java), however Julia's syntax is so expressive that I found it simpler just to write 2-3 lines of a scheduler (the only element I needed) myself. However, I strongly feel that Agents.jl is needed in Julia's ecosystem but I order to contribute I need to understand the direction you are heading to.

[Datseris]

We can do a quick call and I'll walk you through the internals if you want, send me a mail with potential date/time that suits you! In the meantime you can have a look at the tutorial and take note of which things are not clear: https://juliadynamics.github.io/Agents.jl/dev/tutorial/

But the core loop is in fact very simple, shown here: https://github.com/JuliaDynamics/Agents.jl/blob/master/src/simulations/step.jl . Roughly speaking, all other API functions just make the creation of agent_step! and model_step! easier.

[Libbum]

@pszufe, we have a scheduled meeting on Zoom for Tuesday at 16:00 (UTC +2) between the three maintainers. If that time works for you, then perhaps it would be the best time to get aquatinted with each others' packages and hash out a plan.

[pszufe]

Sounds great - chip me in :-)

[Datseris]

Here is an idea regarding movement and the representation of agent position. In reality, agents can be on the street, i.e. on an edge, inbetween nodes.

For example, let's say we have an agent at a starting node 0. And we say "move towards node n for 200 meters". If node n is further than 200 meters, then the agent should stay in between the two nodes, i.e. on the edge.

I think therefore that the most sensible "position" type for the agents, is in fact not a node id, but a tuple: (edge id, portion along edge). This allows us to also get distances. Let's say we are inbetween node 1 and node 2 and we ask "what is the the distance to node 3". Well, it is the distance from node 2 to node 3 plus the "portion" variable times the distance of node 1 to node 2.

What do you think about this idea?

[fbanning]

Wouldn't this introduce the problem that you cannot know the direction in which the agent has travelled before? A Tuple of (edge_id, portion_travelled) could be interpreted two ways, namely portion_travelled between node A and node B or between node B and node A. I think it might make more sense to make it a Tuple of (start_node, end_node, portion_travelled). Whenever an agent is not moving between nodes, then start_node == end_node. Or am I misunderstanding something crucial here?

[Datseris]

No you are right. Initially I was thinking that the graph is directed (so the edge also has this information) but this might not be the case.

An alternative tuple would be (edge, direction, portion), but only an expert of the LightGraphs.jl API can answer which of the two is a better way in the long run.

[fbanning]

the main structure of the data is a LightGraphs directed graph

Maybe you're right after all. If so, your solution would definitely be more elegant.

[pszufe]

The OSMX graph is directed, moreover the edges are defined by a pair of nodes. Hence, in fact, agent's position should perhaps be defined as a triple: (node1, node2, percentage_travelled) which would be equivalent to a tuple (edge_id, percentage_travelled) - each edge id points to an exact pair of nodes (see the code below).

julia> g=LightGraphs.SimpleDiGraph(3);
julia> add_edge!(g, 1,2);add_edge!(g, 2,1);
julia> collect(edges(g))
2-element Array{LightGraphs.SimpleGraphs.SimpleEdge{Int64},1}:
 Edge 1 => 2
 Edge 2 => 1

[Datseris]

Agents.jl now has a clear-cut definition of an API one has to implement for a new type of space. Specifically, one has to define a single structure and extend 5 methods ONLY! This is defined here:

https://github.com/JuliaDynamics/Agents.jl/blob/master/src/core/space_interaction_API.jl#L21

Soon we will be releasing Agents.jl 4.0, which will have reworked spaces that are much more performant. We are also about to submit a paper for peer review regarding Agents.jl, and if we could get this open street map space in there, it would be great for advertisement purposes.

[pszufe]

OK will try to do it over the weekend :-)

[Datseris]

New Developer docs page illustrates how to implement a new space on a step-by-step basis : https://github.com/JuliaDynamics/Agents.jl/blob/master/docs/src/devdocs.md

I think we have already concluded here that the position type should be a tuple {Edge, Float64} where first entry is the edge the agent is currently one and Float64 is a real number from 0 to 1 which specifies how much of the edge has been traversed by the agent (so that the Open street space is continuous space, not just discrete)

[pszufe]

Hi. I am trying to layout the code nicely and one thing remains for me unclear. Where do I keep the simulation state that can not be assigned to a particular <:AbstractAgent and is not logically part of the <: AbstractSpace? ABM is not abstract so there is no space there (as far as I understand its properties field is for simulation parameters). An example of such data could be a dictionary for faster searching specific groups of agents. Another example could be a second space (consider agents walking around a road network but also being a part of a social network). What is the pattern in such situations? I can of course always store it in the properties dedicated struct but perhaps this is rather something that you want to build a GUI around this in the future? I have man ideas to do that but do not know what look nice from your perspective.

I am thinking in terms of a good design pattern for Agents.jl like there are patterns for other libs. Consider this one: https://images.app.goo.gl/98kSxr1E3xRVGPLD9 I feel I am missing in Agents.jl an equivalent of SimState from that picture.

[Datseris]

Hi, thanks for the effort! If you try to put the code in a PR it will much easier for us to help you make "nice" code w.r.t. Agents.jl.

Where do I keep the simulation state that can not be assigned to a particular <:AbstractAgent and is not logically part of the <: AbstractSpace? .... ..... properties field is for simulation parameters

Hm, I have trouble imagining anything related to an agent based simulation that does not logically belong to one of these three options: agent fields, space fields or model properties. For example,

An example of such data could be a dictionary for faster searching specific groups of agents

for me this both logically but also programmatically should be part of the space structure. In fact, our new GridSpace explicitly creates structures that allow for faster neighbor searches and stores them in a particular field called hoods, see here: https://github.com/JuliaDynamics/Agents.jl/blob/master/src/spaces/grid.jl#L137-L144

If you can come up with an example that truly does not belong to one of these three options, I'll address it, but for now I'd say that what you describe is part of the space structure.

Another example could be a second space (consider agents walking around a road network but also being a part of a social network).

This is definitely something very advanced. At the moment this would be done by having two models, one with a GraphSpace and one with a OpenStreetSpace, where you would use the same ID in both spaces to keep track and mean the same agent. We don't have yet any infrastructure for mixed space models, but perhaps this is a good idea for the future.

Are you aware of some other ABM software that allows "dual spaces" or "multiple spaces in parallel"?

I am thinking in terms of a good design pattern for Agents.jl like there are patterns for other libs

We are now almost done performing a comparison of various ABM software with ours, and the paper will be made public in a couple of weeks (but happy to share privately until then). The main outcome of our comparison is that Agents.jl has a massively simpler interface while clearly matching the functionality of the competitors, including MASON, all the while excelling in performance 5-fold or 10-fold times over. Especially MASON is by far the one hardest ABM softwares to learn to use, with high development complexity, while we showcase how Agents.jl leads to an overall shallower learning curve due to design decisions. It is unlikely that we will alter the design of Agents.jl to match more that of MASON after this comparison. In fact, even running basic benchmarks of basic models like e.g. forest fire, wolf grass sheep and flocking was a very difficult task in MASON.

This leads me personally to the conclusion that instead of us looking into other software for design inspiration, they should be looking at us and simplify their approach and interfaces. (personal opinion!)

I feel I am missing in Agents.jl an equivalent of SimState from that picture.

From what I see in the graph you cite, SimState is in fact our AgentBasedModel. The current "simulation state" is fully equivalent with the current state of an AgentBasedModel instance. It has the agents, the space, as well as a scheduler. If you step! the model, all entries are updated dynamically and interactively. The only thing it doesn't have is an explicit RNG entity. I'll open a PR that adds this, it has been asked before.

If you see any difference with this SimState and our AgentBasedModel please feel free to point it out.

[Libbum]

Another example could be a second space (consider agents walking around a road network but also being a part of a social network).

This is definitely something very advanced. [...] Are you aware of some other ABM software that allows "dual spaces" or "multiple spaces in parallel"?

Very standard in NetLogo and MASON. NetLogo uses Link Agents that do exactly this. The only example in the MASON manual builds a school playground (on a GridSpace), then a social network (on a GraphSpace).

I initially saw this as an artefact of some form of retrofitting on already implemented spaces, but now I'm not so sure. Agents.jl was originally modelled off of Mesa, so 'what constitutes an ABM' was heavily influenced by its design choices. The discussion in #313 has seen us take up some concepts from the more mature ABM frameworks, and this may be another place we should listen.

The possibility of doing this is essentially what was being discussed in #314 in some sense too (using graphs on top of your grid/continuous space can achieve any-dimensional space without the need for the grid allowing that possibility).

My point here is that we're welcome to take a fresh look at designing space correctly here, but @pszufe's question is not a very advanced concept in other frameworks—it should definitely be possible in ours and we may end up being influenced by the legacy frameworks regardless.

[pszufe]

In MASON the SimState is an class having fields (random number generator state, schedule, asynchronous state) that is always inherited when implementing an ABM. When subclassing the user of this class adds her own fields representing the model state.

This is very different from the approach in Agents.jl where the model is presented by a concrete fixed type AgentBasedModel. While I really admire the huge work that you have done, the Julian equivalent of MASON's approach would be the following:

• define an AbstractAgentBasedModel type
• there should be functions such as get_properties, get_schedule, iterate_agents operating on that type and (to be decided) either throwing "not implemented" errror ot in some cases returning a field
• provide an implementation of SimpleAgentBasedModel, and just set AgentBasedModel=SimpleAgentBasedModel to make it clear SimpleAgentBasedModel is the base type.
• provide implementations for all methods for SimpleAgentBasedModel

The idea behind this is to use analogous approach to the one taken by LightGraphs.jl where you have an AbstractGraph type that can be extended by others. It works really nice when extending graph functionality - I went through all this integration when writing the SimpleHypergraphs.jl package. BTW HypergraphSpace could be yet another space for Agents.jl.

[Datseris]

This seems reasonable to me and straightforward to implement. But I have to ask the question: what is the benefit ?

Here is the status quo:

• Both the agent types as well as the space types are extendable and work as you've instructed via abastract types. This is identical to how LightGraphs.jl (and most other Julia packages) work. They both define an abstract type and extend some methods.
• It makes sense for the spaces and the agent types to follow this format, because each simulation will need different agents / spaces, while both agents and spaces need to be extendable.
• The current AgentBasedModel has a field properties which can contain anything the user wants. Notice also that we have made it so that when the user types model.some_property, they obtain the field some_property of the original properties field of AgentBasedModel.
• Thus, properties is functionally equivalent with defining a new AgentBasedModel subtype, with some new field some_property and accessing this some_property with model.some_property.
• We discussed in #321 whether or not a random number generator should be part of the ABM or not, but concluded that this is not a good idea. In MASON it might be, because there is only one number distribution. But Julia allows for a plethora of distributions and then everytime the user wanted to use any arbitrary distribution they would be forced to do a wonky rand(rng(model), distribution) instead of using rand directly.

Thus, although we have followed this abstract type design for our spaces, I see no immediate reason to do it for the central ABM type.

Can you please provide a specific argument why this is useful? So far you have presented what the approach should be so that we match what MASON does, but what I am missing is why we should do that.

Notice that what you describe is 10 minutes of work. I could do it pretty much instantly, but I first need to be convinced whether I should. A good reason to allow different types of ABM is to have multiple spaces in one model for example. But, given how easy it is to make ABM abstract, I only see the need to do it once we have an explicit design of how mixed spaces should be implemented. We don't have that yet...

The idea behind this is to use analogous approach to the one taken by LightGraphs.jl where you have an AbstractGraph type that can be extended by others

Agents.jl also has that, we have the AbstractSpace type and functions defined on top of it like random_position. These can also be extended by others. In fact, this is what I've been pointing out in this issue, when I said:

New Developer docs page illustrates how to implement a new space on a step-by-step basis : https://github.com/JuliaDynamics/Agents.jl/blob/master/docs/src/devdocs.md

[pszufe]

@Datseris you wrote:

If you see any difference with this SimState and our AgentBasedModel please feel free to point it out.

So I just pointed out the differences.

My original question was what is the good pattern for storing the data representing simulation state in Agents.jl model - should it be in properties or that you are rather expecting that each model has a unique space. I am just a library user so both work for me - I just want to know which one you recommend with the current library layout. To make it more specific - I will be very soon implementing a fore-mentioned model having two networks (spacial and social). I plan to place the spatial network in some OpenStreetMapSpace - but then where should I place the social network - is the properties recommended approach in such situations? Again I am only asking for your recommendation on what is the good pattern from the library perspective.

[Libbum]

Short answer, for now, would be the model.properties, yeah.

However, I think we need to consider this a little more and should have a consensus about what to do for v4.0—especially if your timeline for having a PR of OpenStreetMapSpace space (even a proof of concept) is within a week or so @pszufe.

With a decent example, it will be easier to figure out the best way forward.

[pszufe]

OK, I finally did it. What is needed are your comments on the design. There are several pieces of information that need to be stored and I have placed them either in properties or space trying to follow arguments in your discussion. I tried to do my best, however I do not feel it is a perfect Agents.jl usage example yet. I have used a discretized map version where only intersection vertices are present. This can be obviously extended to a continuous version.

So now waiting for your feedback.

using Agents, Parameters, OpenStreetMapX, StatsBase, Random, OpenStreetMapXPlot, LightGraphs, Colors, PyCall, Plots

m = get_map_data(joinpath(dirname(pathof(OpenStreetMapX)),"..","test/data/reno_east3.osm"), use_cache=false, trim_to_connected_graph=true );

@with_kw mutable struct Zombie <: AbstractAgent
    id::Int    
    pos::Int
    last_node::Int=-1
    infected::Bool = false
    infection_time::Int = -1
    total_route_len::Int = 0
end

struct OpenStreetMapSpace <: Agents.Agents.DiscreteSpace    
    m::OpenStreetMapX.MapData
    graph::SimpleDiGraph
    viz::Plots.Plot
    nodes_agents::Vector{Set{Int}}
end

OpenStreetMapSpace(m::OpenStreetMapX.MapData) =
    OpenStreetMapSpace(m, m.g, plotmap(m), [Set{Int}() for _ in nodes(m.g)] )

function nearby_ids(space::OpenStreetMapSpace, pos::Int)
    neighbors(space.m.g, pos)
end

function add_agent_pos!(agent::AbstractAgent, model::ABM)
    push!(model.space.nodes_agents[agent.pos], agent.id)
    model.agents[agent.id]=agent
end

function move_agent!(agent::A, pos, model::ABM{A, <: OpenStreetMapSpace}) where A <: AbstractAgent
    delete!(model.space.nodes_agents[agent.pos], agent.id)
    agent.pos = pos
    push!(model.space.nodes_agents[agent.pos], agent.id)
end

function move_agent!(agent::A, model::ABM{A, <: OpenStreetMapSpace}) where A <: AbstractAgent
    move_agent(agent, rand(nearby_ids(model.space, agent.pos)), model )
end

function kill_agent!(agent::AbstractAgent, model::ABM{A, <: OpenStreetMapSpace}) where A <: AbstractAgent
    delete!(model.space.nodes_agents[agent.pos], agent.id)
end
function space_neighbors(position, model::ABM{A, <: OpenStreetMapSpace}, r) where A <: AbstractAgent
    neighborhood(model.space.n.g, position, r)
end

@with_kw struct ZombieProps
    num_agents::Int = 100
    m::MapData
    nodes_infected::Vector{Set{Int}}=[Set{Int}() for _ in 1:nv(m.g)]   
    infected_agents_count::Vector{Int}=[1]    
end

function ZombieAgentBasedModel(m::MapData; properties = ZombieProps(m=m) )
    sim = AgentBasedModel(Zombie,OpenStreetMapSpace(m), properties = properties)
    for i in 1:sim.properties.num_agents
        add_agent_pos!(Zombie(id=i, pos=rand(nodes(sim.space.m.g))), sim)
    end
    sim.agents[1].infected = 1 # we start with one sick agent
    sim.agents[1].infection_time = 0    
    push!(sim.properties.nodes_infected[sim.agents[1].pos], 1) 
    sim
end

function plot(s::AgentBasedModel{Zombie,OpenStreetMapSpace,T,S} ) where {T,S}
    p = deepcopy(sim.space.viz)
    m = sim.space.m
    for aid in keys(s.agents)
        agent = s.agents[aid]
        x = OpenStreetMapX.getX(m.nodes[m.n[agent.pos]]) 
        y = OpenStreetMapX.getY(m.nodes[m.n[agent.pos]])  
        Plots.annotate!(p,x,y,Plots.text("O",agent.infected ? 11 : 10,agent.infected ? :green : :black))
    end
    p
end

function agent_step!(agent::Zombie, s::AgentBasedModel)
    pop!(s.space.nodes_agents[agent.pos], agent.id )
    agent.total_route_len += 1
    agent.infected && pop!(s.properties.nodes_infected[agent.pos], agent.id)

    ns = nearby_ids(s.space, agent.pos)

    new_node = rand(ns)

    agent.last_node = agent.pos
    agent.pos = new_node
    new_infections = 0
    if agent.infected
        for id in s.space.nodes_agents[new_node]
            if ! s.agents[id].infected
                s.agents[id].infected = true
                s.agents[id].infection_time = length(s.infected_agents_count)
                push!(s.properties.nodes_infected[new_node], id)
                new_infections += 1
            end  
        end
    else
        if length(s.properties.nodes_infected[new_node]) > 0
            agent.infected = true
            agent.infection_time = length(s.infected_agents_count)
            new_infections += 1
        end
    end
    if agent.infected
        push!(s.properties.nodes_infected[new_node], agent.id)
    end
    push!(s.space.nodes_agents[new_node], agent.id)
    push!(s.infected_agents_count,s.infected_agents_count[end]+new_infections)
end

# %%
Random.seed!(0);
sim = ZombieAgentBasedModel(m, properties = ZombieProps(m=m, num_agents=300))

frames = Plots.@animate for i in 1:50
    step!(sim, agent_step!, 1)
    plot(sim)
end

gif(frames, "anim.gif", fps = 3)

[Datseris]

That's a good start @pszufe . Here is what I think:

1. Please open a Pull Request with this code so that everyone can add comments easily directly on the code, and anyone can meaningfully change the code.
2. I'd say that it is very helpful to separate the example from the core structures. I would add the struct OpenStreetMapSpace and the related functions like nearby_ids in a different file than the example.
3. I don't think anything visualization related should be contained in the space (I'm talking about the field vis which is a Plot. This has to be something independent from both the model and the space.
4. Is there a reason that m::MapData exists in both ZombieProps and the space structure?

For me the biggest point of discussion is how the actual space is represented. What you have here is in principle fine, but of course only the intersections (verteces) are available to the agents. In some posts above I proposed that the agent position could be a tuple: (Edge, x) with x a number between 0-1. This would indicate how far along the edge has an agent moved.

The tricky part of course is that we need to associate an Edge with physical distance on the real map. Is this possible to obtain from OpenStreetMap? How?

[pszufe]

1. I was planning to do so after some initial feedback

2. OK

3. OK - this a question what is the pattern for storing visualisation data in Agents

4. Yes - because I remember from the previous discussion that it was recommended to store such additional info in properites but on the other hand the API assumes that some functions take only the space. Perhaps in this case it could still work after moving this to space. I am now thinking that maybe the OpenStretMapSpace should have its own properties field for storgage of additional information - because the API assumes that for many calls only the space is given. What do you think?

Tricky part: I would represent this as triple (node1, node2, value) so it is clear how the agent is travelling the edge and agent in a node without a direction decided can be represented than as (node, node, 0) while an agent at a node who has choosen the direction can be seen as (node1, node2, 0) All required functionality is in the OpenStreetMapX already so it is not tricky at all :-).

[pszufe]

Yet another question: should OpenStreetMapX space be the part of Agents.jl which means that Agents.jl has an OpenstreetMapX dependecy or would you like to make it as a separate package under JuliaDynamics?

[Datseris]

3. OK - this a question what is the pattern for storing visualisation data in Agents

This we will take care of in AgentsPlots.jl; for now, you can leave the plotting parts inside one of the model properties, so that the example you have created is runnable and also plottable. Then I'll move the plotting code to AgentsPlots.jl.

I would represent this as triple (node1, node2, value)

Yes, that is fine, however this requires that an internal check is done always that checks whether node1, node2 are connected in the underlying graph, right?

4.

Well, I think that m::OpenStreetMapX.MapData should be a field of the space. Any additional fields you need you can make them also parts of the space. I am not fan of having a second container inside the space. If you need an extra attribute for the space, for e.g. finding nearest neighbors or what have you, make it a proper field, no need to nest it further in some properties.

dependency

OpenStreetMap will become a dependency of agents, that's not a problem. The only thing I see has a noteworthy dependency is HTTP, but I guess we can live with that.

[Libbum]

@pszufe I've been having a look at this, and have a few questions:

• What are the distance and time units you're using in OpenStreetMapX?
• When I load some example file m = get_map_data(...), your documentation and source tells me that m.n should be a dictionary of lightgraph node numbers as keys, and the map positions as values, yet I don't see that happening:

  
  julia> m = get_map_data("test/data/torontoF.osm", use_cache=false, trim_to_connected_graph=true);

julia> m.n 384-element Array{Int64,1}: 29688408 -1154164823864377407 34936037 29604801 20979760 ...

Any idea as to why?
- What is `distance` returning? Straight line distance?
- I would assume that we'd be more interested for the Zombie example to be using `shortest_route` from one place to another, is that your take also?

[pszufe]

@Libbum Thanks for asking - those are excellent questions

• The distance units are meters and seconds (probably this could be better documented). The times are calulated on the base of travel speeds for various types of roads defined here: https://github.com/pszufe/OpenStreetMapX.jl/blob/master/src/speeds.jl
• m.n - The Dict has been replaced with Vector for faster indexing - seems the docs has not been updated
• distance return a real distance in meters when travelling the road. To be more presize OSM map has points connected by straight lines. However when parsing the map, the points are not intersections are removed from the data but the original, correct distances are preserved. Hence the value of distance between two points lying on an arc will be greater than straight line distance
• this is also possible (look at the OSMX tutorial). However, I am wondering how complex you want this model to be for the Agents.jl tutorial.

[Libbum]

Great, thanks for all that!

I think distance will be good enough for the tutorial then—yes. We can perhaps let readers know about the other tools you provide, but leave it up to them to tinker.

[Libbum]

I started to build an OpenStreetMapSpace based on the above example, and quickly came to realise that everything there was just attached to the underlying graph, then translated to coordinates in the map.

Additionally, there's a lot of additional properties there that are just not required, so I've written up an example below that does exactly the same thing, but in a more Agents.jl way. I know this is a first pass & the discussion following the example has not been taken into account here, but it's perhaps a better starting point to extend from.

Possibly, there will be no need for a space, but we can write up a decent "Ecosystem Integration" tutorial.

using Agents
using OpenStreetMapX
using OpenStreetMapXPlot
using Plots
gr()

mutable struct Zombie <: AbstractAgent
    id::Int
    pos::Int
    infected::Bool
end

function initialise()
    m = get_map_data(
        "test/data/reno_east3.osm",
        use_cache = false,
        trim_to_connected_graph = true,
    )

    model = ABM(Zombie, GraphSpace(m.g), properties = Dict(:osmmap => m))

    for _ in 1:100
        add_agent!(model, false)
    end

    patient_zero = random_agent(model)
    patient_zero.infected = true
    return model
end

function agent_step!(agent, model)
    np = nearby_positions(agent, model)
    new_pos = rand(np)
    move_agent!(agent, new_pos, model)

    if agent.infected
        map(a -> a.infected = true, nearby_agents(agent, model))
    else
        if any(a.infected for a in nearby_agents(agent, model))
            agent.infected = true
        end
    end
end

function get_coordinates(agent, model)
    getX(model.osmmap.nodes[model.osmmap.n[agent.pos]]),
    getY(model.osmmap.nodes[model.osmmap.n[agent.pos]])
end

ac(agent) = agent.infected ? :green : :black
as(agent) = agent.infected ? 6 : 5

function plotagents(model)
    # Essentially a cut down version on plotabm
    ids = model.scheduler(model)
    colors = [ac(model[i]) for i in ids]
    sizes = [as(model[i]) for i in ids]
    markers = :circle
    pos = [get_coordinates(model[i], model) for i in ids]

    scatter!(
        pos;
        markercolor = colors,
        markersize = sizes,
        markershapes = markers,
        label = "",
        markerstrokewidth = 0.5,
        markerstrokecolor = :black,
        markeralpha = 0.7,
    )
end

model = initialise()

frames = @animate for _ in 1:50
    step!(model, agent_step!, 1)
    plotmap(model.osmmap)
    plotagents(model)
end

gif(frames, "anim.gif", fps = 3)

Speaking about distances along a path etc. One way we could go about this would be to use a SimpleWeightedGraph built out of the nodes of m.g and the weights in m.w. That way we'd have an understanding of distance & travel time in the Agents representation.

[Datseris]

I think a space is necessary because I see that the discrete version makes little sense for real-world applications. You can tell from the figure that transporting just one node per step leads to a fractured simulation. On the other hand a space structure would allow using the continuous version without too much complexity...

Similar to how ContinuousSpace has a GridSpace as a subfield, this new space would have a GraphSpace as a sub-field.

[Libbum]

Sure, I can see how that may pan out. We'd need an agent type with pos::Tuple{Float64, Float64} and perhaps a speed::Float64. The position values would be obtained via something similar to get_coordinates above, represented on the GraphSpace between some set of nodes according to the 'weight' (which represents distance).

[Datseris]

I think my original suggestion of position type of (Edge, Float64) is the most suitable and solves every problem:

1. Allows continuous space.
2. Guarantees that the agent is only positioned somewhere allowed by the underlying network / street map
3. Contains starting and ending node, and thus also direction of travel.
4. Plays well with the underlying network structure becomes nodes are directly accessible instead of need of being deduced.

A simple function can translate this (Edge, Float64) to a real space position using the API of OpenStreetMapX.jl.

---

I do see one problem though: starting agents at some node without a specific directionality. Perhaps in this scenario we can simply not care which edge we choose, as long as the edge has the given starting node.

[Libbum]

Think I understand that better now. Will play around with it soon.

[Datseris]

And we can of course extend functions like nearby_pos to obtain as an input a position in real space and return node ids. This will also be useful for real world applications where you know the real-space position of e.g. the town square or the super market, and you want nearby nodes.

[Datseris]

@Libbum correction to my suggestion. @pszufe 's original suggestion of the position type being a tuple (Int, Int, AbstractFloat) is better, because we don't have to require users to know what an Edge is (and thus know LightGraphs.jl). Here the two integers are functionality equivalent with an edge, since they are start and end node.

[pszufe]

OK I have incorporated all of your comments.

The changes include

• I tried to incorporate as much as it seemed only possible code by @Libbum
• The model is continuous and agents now move 50 meters per simulation step (unless they reach an intersection - in that case they stay there till the end of their step)
  • Agents have route plans. The end of their route is marked by a block arrow while the current direction on route is marked by an empty arrow. Zombie agents have green routes so we can see where they are heading
  • If agents meet each other on intersection or I traveling the same arc from both ends the disease is transmitted
  • I was thinking it over again (Edge, Float64) is not good representation of agent for many reasons (performance, direction, representing agents on node, representing other possible agents such as drones) - hence a location of an agent is represented by (Int, Int, Float64).
  • I noticed that this agent movement framework can be made generic and hence very useful for others by using composition. It is possible to have a generic OSMXAgent with a properties field that shows it is a Zombie.

I consider this more or less complete. There are really lots of possibilities to focus on spatial properties of such environment. In particular there many nice ways for abstracting the movement of agents in a map space.

Regarding further steps it is probably good time for me to prepare a PR.

using Agents, Parameters, OpenStreetMapX, StatsBase, Random, OpenStreetMapXPlot, LightGraphs, Colors, PyCall, Plots

struct OSMXPos
    node1::Int # starting node
    node2::Int # target node
    trav::Float64 #trabelled %
end    

OSMXPos(node1::Int, node2::Int) = OSMXPos(node1, node2, 0.0) #agent decided to head to node 2
OSMXPos(node::Int) = OSMXPos(node, node) # agent arrived to node
function OSMXPos(pos::OSMXPos, delta::Float64)
    trav = pos.trav + delta
    trav >= 1.0 && return OSMXPos(pos.node2)
    trav <= 0.0 && return OSMXPos(pos.node1)
    return OSMXPos(pos.node1, pos.node2, trav)
end

@with_kw mutable struct OSMXAgent{T} <: AbstractAgent where T
    id::Int    
    pos::OSMXPos
    props::T
    path::Dict{Int,Int} = Dict{Int,Int}() #seqence of nodes
    path_distances::Dict{Int,Float64} = Dict{Int,Float64}() # distance from each node in the sequence

end

OSMXAgent(id::Int, pos::OSMXPos) = OSMXAgent{Nothing}(id,pos, nothing)

@with_kw mutable struct ZombieProps
    infected::Bool = false
end

struct OpenStreetMapSpace <: Agents.AbstractSpace    
    osmmap::OpenStreetMapX.MapData
    nodes_agents::Vector{Set{Int}} # all agents that are at a node or on route from node to another one
end

OpenStreetMapSpace(m::OpenStreetMapX.MapData) =
    OpenStreetMapSpace(m, [Set{Int}() for _ in nodes(m.g)] )

function nearby_ids(space::OpenStreetMapSpace, pos::Int)
    neighbors(space.osmmap.g, pos)
end

function add_agent_pos!(agent::OSMXAgent, model::ABM)
    push!(model.space.nodes_agents[agent.pos.node1], agent.id)
    model.agents[agent.id]=agent
end

function move_agent!(agent::OSMXAgent{P}, pos::OSMXPos, model::ABM{OSMXAgent{P}, <: OpenStreetMapSpace}) where P
    if pos.node1 !== agent.pos.node1
        delete!(model.space.nodes_agents[agent.pos.node1], agent.id)
        push!(model.space.nodes_agents[pos.node1], agent.id)
    end
    agent.pos = pos
end

function move_agent!(agent::OSMXAgent{P}, model::ABM{OSMXAgent{P}, OpenStreetMapSpace}; meters=50.0) where P    
    if agent.pos.node1 == agent.pos.node2
        #selecting new travel destination if none exists
        if ! (agent.pos.node1 in keys(agent.path))            
            nodes = Int[]
            while length(nodes) < 2
                n2 = rand(1:nv(model.space.osmmap.g))
                nodes =  getindex.(Ref(model.space.osmmap.v), 
                        shortest_route(model.space.osmmap,model.space.osmmap.n[agent.pos.node1],
                                       model.space.osmmap.n[n2])[1])
            end
            empty!(agent.path)
            setindex!.(Ref(agent.path), nodes[2:end], nodes[1:end-1])
            for i in 1:(length(nodes)-1)
                agent.path_distances[nodes[i]] = model.space.osmmap.w[nodes[i], nodes[i+1]]
            end
        end     

        node2 = agent.path[agent.pos.node1]        
        trav = meters/agent.path_distances[agent.pos.node1]
        if trav >= 1.0
            pos = OSMXPos(node2)
        else
            pos = OSMXPos(agent.pos.node1, node2, trav)
        end        
    else
        pos = OSMXPos(agent.pos, meters/agent.path_distances[agent.pos.node1])
    end 
    move_agent!(agent, pos, model)    
end

function kill_agent!(agent::OSMXAgent, model::ABM{OSMXAgent, OpenStreetMapSpace}) 
    delete!(model.space.nodes_agents[agent.pos.node1], agent.id)
    pop!(model.agents, agent.id)
end
function space_neighbors(pos::Int, model::ABM{OSMXAgent, OpenStreetMapSpace}, r)
    neighborhood(model.space.osmmap.g, position, r)
end

function initialise()
    m = get_map_data(joinpath(dirname(pathof(OpenStreetMapX)),"..","test/data/reno_east3.osm"), 
                              use_cache=false, trim_to_connected_graph=true );
    model = ABM(OSMXAgent{ZombieProps},OpenStreetMapSpace(m),properties=Dict())
    for i in 1:150
        add_agent_pos!(OSMXAgent(id=i, pos=OSMXPos(rand(nodes(model.space.osmmap.g))),props=ZombieProps()), model)
    end
    patient_zero = random_agent(model)
    patient_zero.props.infected = true
    model
end

function get_ENU(node::Int, model)
    model.space.osmmap.nodes[model.space.osmmap.n[node]]
end

function get_coordinates(agent, model)
    pos1 = get_ENU(agent.pos.node1, model)
    pos2 = get_ENU(agent.pos.node2, model)
    (getX(pos1)*(1-agent.pos.trav)+getX(pos2)*(agent.pos.trav),
     getY(pos1)*(1-agent.pos.trav)+getY(pos2)*(agent.pos.trav))
end

agentcolor(agent) = agent.props.infected ? :green : :black
agentsize(agent) = agent.props.infected ? 6 : 5

function plotagents(model, background=Plots.current())
    ids = model.scheduler(model)
    colors = [agentcolor(model[i]) for i in ids]

    for i in ids
        path = [model[i].pos.node1]
        while path[end] in keys(model[i].path)
            push!(path, model[i].path[path[end]])
        end
        length(path) < 2 && continue
        a_x, a_y = get_coordinates(model[i], model)
        enu_coords = get_ENU.(path[2:end], Ref(model))

        a_x2, a_y2 = (getX(enu_coords[1]), getY(enu_coords[1]))

        !(a_x == a_x2 && a_y == a_y2) &&  plot!(
                background, [a_x, a_x2], [a_y, a_y2], 
                color=colors[i], arrow =arrow(:open))

        length(enu_coords) > 2 &&  plot!(
                background, getX.(enu_coords),  getY.(enu_coords),
                color=colors[i], arrow =arrow(:closed))
    end

    sizes = [agentsize(model[i]) for i in ids]
    markers = :circle
    pos = [get_coordinates(model[i], model) for i in ids]
    scatter!(background,
        pos;
        markercolor = colors,
        markersize = sizes,
        markershapes = markers,
        label = "",
        markerstrokewidth = 0.5,
        markerstrokecolor = :black,
        markeralpha = 0.7,
    )

end

function agent_step!(agent, model)
    move_agent!(agent, model; meters=50.0) 
    if agent.props.infected
        for i in model.space.nodes_agents[agent.pos.node2]            
            model[i].props.infected && continue
            if agent.pos.node1 == agent.pos.node2 
                model[i].props.infected = (model[i].pos.node2 == agent.pos.node1)
            else
                model[i].props.infected = (model[i].pos.node2 == agent.pos.node1)
            end
        end
    else
        for i in model.space.nodes_agents[agent.pos.node2]            
            (! model[i].props.infected) && continue
            if agent.pos.node1 == agent.pos.node2 
                agent.props.infected = model[i].pos.node2 == agent.pos.node1
            else
                agent.props.infected = model[i].pos.node2 == agent.pos.node1
            end
        end
    end
end

Random.seed!(2);

model = initialise();
p = plotmap(model.space.osmmap);
frames = Plots.@animate for i in 1:140
    i > 1 && step!(model, agent_step!, 1)
    plotagents(model, deepcopy(p))
end

gif(frames, "anim.gif", fps = 3);

[Datseris]

Hi @pszufe that is amazing! Please put this into a PR so that you get full commit credit :) Any minor additions, like adding documentation strings, we can also help!

[Libbum]

@Datseris its a good start but still a lot of work once you drill down. Still many open questions. I'm doing some work on it offline currently.

@pszufe indeed—put up a PR and we can work from there.

[Datseris]

Implemented in #333