Network layout does not handle disconnected street networks

[emanuel030]

I tried running cea network-layout on a new project and it raised following error:

(cea) C:\Users\Assistenz>cea network-layout
City Energy Analyst version 2.9.2a
Running `cea network-layout` with the following parameters:
- general:scenario = E:\remanuel\project_case\1_Baseline
  (default: C:\reference-case-open\baseline)
- network-layout:network-type = DC
  (default: DC)
- network-layout:buildings = []
  (default: [])
- network-layout:pipe-diameter = 150
  (default: 150)
- network-layout:type-mat = T1
  (default: T1)
- network-layout:create-plant = True
  (default: True)
- network-layout:allow-looped-networks = False
  (default: False)
- thermal-network:disconnected-buildings = []
  (default: [])
Copied file: E:\remanuel\project_case\1_Baseline\databases\CH\systems\supply_systems.xls
Copied file: E:\remanuel\project_case\1_Baseline\databases\CH\lifecycle\LCA_buildings.xlsx
Copied file: E:\remanuel\project_case\1_Baseline\databases\CH\systems\thermal_networks.xls
Copied file: E:\remanuel\project_case\1_Baseline\databases\CH\benchmarks\benchmark_2000W.xls
Copied file: E:\remanuel\project_case\1_Baseline\databases\CH\uncertainty\uncertainty_distributions.xls
No handlers could be found for logger "shapely.geos"
Traceback (most recent call last):
  File "C:\ProgramData\Miniconda2\envs\cea\Scripts\cea-script.py", line 11, in <module>
    load_entry_point('cityenergyanalyst', 'console_scripts', 'cea')()
  File "c:\users\assistenz\documents\github\cityenergyanalyst\cea\interfaces\cli\cli.py", line 65, in main
    script_module.main(config)
  File "c:\users\assistenz\documents\github\cityenergyanalyst\cea\technologies\thermal_network\network_layout\main.py", line 58, in main
    network_layout(config, locator, plant_building_names)
  File "c:\users\assistenz\documents\github\cityenergyanalyst\cea\technologies\thermal_network\network_layout\main.py", line 39, in network_layout
    path_potential_network)
  File "c:\users\assistenz\documents\github\cityenergyanalyst\cea\technologies\thermal_network\network_layout\connectivity_potential.py", line 450, in
 calc_connectivity_network
    crs)
  File "c:\users\assistenz\documents\github\cityenergyanalyst\cea\technologies\thermal_network\network_layout\connectivity_potential.py", line 352, in
 one_linestring_per_intersection
    lines_merged = lines_merged.intersection(bounding_box)
  File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\shapely\geometry\base.py", line 620, in intersection
    return geom_factory(self.impl['intersection'](self, other))
  File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\shapely\topology.py", line 70, in __call__
    self._check_topology(err, this, other)
  File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\shapely\topology.py", line 39, in _check_topology
    raise err
shapely.errors.TopologicalError: This operation could not be performed. Reason: unknown

I have attached the used cea.config file as a text file. cea.txt

The reference case is working fine on my machine. The coordinate system of the streets.shp file has been checked and is the same as the zone.shp file and has format:

 {u'ellps': u'bessel',
 u'k_0': 1,
 u'lat_0': 46.95240555555556,
 u'lon_0': 7.439583333333333,
 u'no_defs': True,
 u'proj': u'somerc',
 u'units': u'm',
 u'x_0': 600000,
 u'y_0': 200000}

Help on this would be greatly appreciated!

[daren-thomas]

@JIMENOFONSECA is this the right coordinate system to use?

[martin-mosteiro]

@JIMENOFONSECA This seems to be an issue created since the move away from ArcGIS. The old network layout tool (which used BackgroundGP) was used earlier this year to generate thermal networks using the same shapefiles being used by @emanuel030. Indeed, reverting to the previous version of the layout tool (which used BackgroundGP) allows the layout to be created without issue. Could you maybe comment on what might be the problem?

[jimenofonseca]

I can take a look at it. @martin-mosteiro can you tell me which case study, files and steps you did in order to reproduce the error

[martin-mosteiro]

@JIMENOFONSECA a follow up from @emanuel030:

False hope though… the old arcgis version only creates a DC folder and edges.cpg and nodes.cpg files… it does not create shp files... this error pops up instead:

C:\ProgramData\Miniconda2\envs\cea\python.exe C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py
Path to ArcGIS workspace: c:\Users\Assistenz\AppData\Local\Temp\tmpcrftvr.gdb
Traceback (most recent call last):
File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 59, in <module>
main(cea.config.Configuration())
File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 56, in main
network_layout(config, locator)
File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 51, in network_layout
total_demand_location, create_plant, config.network_layout.allow_looped_networks)
File "C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\steiner_spanning_tree.py", line 51, in calc_steiner_spanning_tree
mst_non_directed = nx.Graph(steiner_tree(G, terminal_nodes))
File "<decorator-gen-685>", line 2, in steiner_tree
File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for
return not_implement_for_func(*args, **kwargs)
File "<decorator-gen-684>", line 2, in steiner_tree
File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for
return not_implement_for_func(*args, **kwargs)
File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\algorithms\approximation\steinertree.py", line 82, in steiner_tree
M = metric_closure(G, weight=weight)
File "<decorator-gen-681>", line 2, in metric_closure
File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for
return not_implement_for_func(*args, **kwargs)
File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\algorithms\approximation\steinertree.py", line 35, in metric_closure
raise nx.NetworkXError(msg)
networkx.exception.NetworkXError: G is not a connected graph. metric_closure is not defined.

Again I made sure that there are no loose street parts.. but no success.. are you familiar with this error?

We will meet again on Thursday afternoon and check out the issue. @emanuel030 please keep me posted if anything else in the mean time.

[jimenofonseca]

We were facing this problem a couple of weeks ago. It was solved in the current version in Master (that of a couple of weeks ago).

Let me know if you have more issues. The best, Is that you upload the Zone and streets shapefiles. I can check what is happening with that On 26 Mar 2019, 4:52 PM +0800, Martín Mosteiro-Romero notifications@github.com, wrote:

@JIMENOFONSECAhttps://github.com/JIMENOFONSECA a follow up from @emanuel030https://github.com/emanuel030:

False hope though… the old arcgis version only creates a DC folder and edges.cpg and nodes.cpg files… it does not create shp files... this error pops up instead:

C:\ProgramData\Miniconda2\envs\cea\python.exe C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py

Path to ArcGIS workspace: c:\Users\Assistenz\AppData\Local\Temp\tmpcrftvr.gdb

Traceback (most recent call last):

File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 59, in

main(cea.config.Configuration())

File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 56, in main

network_layout(config, locator)

File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 51, in network_layout

total_demand_location, create_plant, config.network_layout.allow_looped_networks)

File "C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\steiner_spanning_tree.py", line 51, in calc_steiner_spanning_tree

mst_non_directed = nx.Graph(steiner_tree(G, terminal_nodes))

File "", line 2, in steiner_tree

File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for

return not_implement_for_func(*args, **kwargs)

File "", line 2, in steiner_tree

File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for

return not_implement_for_func(*args, **kwargs)

File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\algorithms\approximation\steinertree.py", line 82, in steiner_tree

M = metric_closure(G, weight=weight)

File "", line 2, in metric_closure

File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\utils\decorators.py", line 82, in _not_implemented_for

return not_implement_for_func(*args, **kwargs)

File "C:\ProgramData\Miniconda2\envs\cea\lib\site-packages\networkx\algorithms\approximation\steinertree.py", line 35, in metric_closure

raise nx.NetworkXError(msg)

networkx.exception.NetworkXError: G is not a connected graph. metric_closure is not defined.

Again I made sure that there are no loose street parts.. but no success.. are you familiar with this error?

We will meet again on Thursday afternoon and check out the issue. @emanuel030https://github.com/emanuel030 please keep me posted if anything else in the mean time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/architecture-building-systems/CityEnergyAnalyst/issues/1836#issuecomment-476529322, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIjrgjdWAjCzQKYS1jFj_jzwTLQRqLSRks5vad-ggaJpZM4cCGWd.

[emanuel030]

@JIMENOFONSECA Thanks for looking into this issue. The used shapefiles and the config file are attached. All shapefiles are in coordinate system WGS_1984_UTM_Zone_32N like the reference case. reproduce_error.zip

[emanuel030]

Update: I drew up a new streets shape file streets.zip with which the old ArcGis version (using git reset --hard 60c3e28) does work. It produces edges and nodes shape files and just gives following warning:

C:\ProgramData\Miniconda2\envs\cea\python.exe C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py
Path to ArcGIS workspace: c:\Users\Assistenz\AppData\Local\Temp\tmpykrj65.gdb
C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\steiner_spanning_tree.py:154: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  selected_node["Name"] = "NODE" + str(new_mst_nodes.Name.count())
C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\steiner_spanning_tree.py:155: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  selected_node["Type"] = "PLANT"

Process finished with exit code 0

The shapely version still gives the same error posted in the original comment.

[martin-mosteiro]

The warnings are normal, that is a successful run. The same shapefile does not work on the current master branch, though, correct?

As discussed yesterday, that is satisfactory for the purposes of your thesis, you can generate the networks using the old branch and go back to master for simulation. We'll see about the errors in parallel.

[emanuel030]

Yes, master branch still throws the same error with the new shapefile.

[jimenofonseca]

Alright, so I found the problem. It turns out that the new open source script cannot handle shapefiles with disconnected streets. See picture.

If you remove those streets, then it runs. While I was at it, I created some warnings for the user.

@emanuel030 I attach the streets.shp without the unwanted (disconnected) streets. Would you mind testing it again using branch i1836-network-layout?

let me know how it goes.

streets.zip

[martin-mosteiro]

@JIMENOFONSECA Thanks for checking that out, unfortunately this does not seem to solve the issue for me:

C:\Users\Assistenz\Miniconda2\envs\cea\python.exe C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py
No handlers could be found for logger "shapely.geos"
Traceback (most recent call last):
  File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 62, in <module>
    main(cea.config.Configuration())
  File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 58, in main
    network_layout(config, locator, plant_building_names)
  File "C:/Users/Assistenz/Documents/GitHub/CityEnergyAnalyst/cea/technologies/thermal_network/network_layout/main.py", line 39, in network_layout
    path_potential_network)
  File "C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\connectivity_potential.py", line 450, in calc_connectivity_network
    crs)
  File "C:\Users\Assistenz\Documents\GitHub\CityEnergyAnalyst\cea\technologies\thermal_network\network_layout\connectivity_potential.py", line 352, in one_linestring_per_intersection
    lines_merged = lines_merged.intersection(bounding_box)
  File "C:\Users\Assistenz\Miniconda2\envs\cea\lib\site-packages\shapely\geometry\base.py", line 620, in intersection
    return geom_factory(self.impl['intersection'](self, other))
  File "C:\Users\Assistenz\Miniconda2\envs\cea\lib\site-packages\shapely\topology.py", line 70, in __call__
    self._check_topology(err, this, other)
  File "C:\Users\Assistenz\Miniconda2\envs\cea\lib\site-packages\shapely\topology.py", line 39, in _check_topology
    raise err
shapely.errors.TopologicalError: This operation could not be performed. Reason: unknown

Process finished with exit code 1

[emanuel030]

Same for me, the problem is not with disconnected streets in the streets shapefile. The file I have recently used was reduced to this: streets_smaller.zip and it definitely does not include disconnected streets and works with the ArcGis Version (git reset --hard 60c3e28).

[jimenofonseca]

Mmm. Are you working in the branch I mentioned? I corrected for the error you just showed me there martin.

On 28 Mar 2019, 23:01 +0800, emanuel030 notifications@github.com, wrote:

Same for me, the problem is not with disconnected streets in the streets shapefile. The file I have recently used was reduced to this: streets_smaller.ziphttps://github.com/architecture-building-systems/CityEnergyAnalyst/files/3018582/streets_smaller.zip and it definitely does not include disconnected streets and works with the ArcGis Version (git reset --hard 60c3e28https://github.com/architecture-building-systems/CityEnergyAnalyst/commit/60c3e2807e0e9978840ba73994506c85b5b7db80).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/architecture-building-systems/CityEnergyAnalyst/issues/1836#issuecomment-477633434, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIjrgsLjGgNkGCQC5qc0X8k_NxqML25Bks5vbNldgaJpZM4cCGWd.

[emanuel030]

@JIMENOFONSECA The i1836-network-layout branch does work for running cea network-layout. Thank you for looking into this!

[jimenofonseca]

@emanuel030 no problem, thanks for the feedback!, this helps us to build a better tool for you and me :-).

@martin-mosteiro would you mind checking the PR related to this issue? Thanks!

[gabriel-happle]

Dear @emanuel030 ,

In order for us to better understand the needs of our users and steer the development of the tool in the future we would like to understand why and how you are using the CityEnergyAnalyst.

Please help us by answering 10 short questions in the CEA user survey. The feedback we receive will determine the focus of the upcoming development efforts.

Kind regards, The CEA development team