Importing valves as points and splitting line question

[Lolaento]

I'm trying to import a network into QGISRED. 1Q) A bunch of the valves are positioned very closely to nodes and when importing the node don't get imported and what ends up happening is the valve creates a node that overrides the existing node. Sometimes this happens:

When it should look like:

Any ideas how I can over come this? Maybe if there is away for me to change the 1m default import valve length?

2Q) When I split lines by points in qgis, it messes up the import process to QGISRED. I've changed the pipeline ID etc to be unique, but when I import a lot of the pipelines just don't come through. Any ideas?

[neslerel]

Hi @Lolaento,

Thank you for reporting your questions and helping us to improve QGISRed!

1A) About your first question, yes, there is a default value to change that length of "1m". However, I am not sure why the end nodes of the valves are not connected to the other nodes or why your nodes are not imported. Could you send me the original files (just from this area might be enough) to test it and check what happens?

2A) Could you detail the steps to reproduce it? Are you trying to import this layer of lines in a new project or with other elements in the model? Can you try to import this layer without selecting the Id field? If it is possible to send us the files to check it would be perfect!

[Lolaento]

@neslerel Thank you for the timely reply.

I will definitely try importing again after changing the default pipeline length split value. In the meantime, I have attached the area in question:

Q1 Files.zip

Have go at importing those 3 files to QGISRED and see if you get what I'm seeing.

2A. To my knowledge (this is my first time using QGISRED), the GIS files for import needs to connected nicely with all pipelines connected to a point vector. To preface, the file I received from client was very incomplete and full of gaps. The common errors in the SHP files were:

• Some lines were not connected to any nodes
• Lines not joining up, this could be at a very small scale as well, so often it was hard to identify these disconnections
• Points were over pipelines that weren't split
• Points were not on top of the pipelines (again, often at small scales) and made it hard for me to identify where they were

Some examples:

The steps I took to remediate the proceeds are as follows:

1. In QGIS, using the "Snap points to lines" tool to include as much of the point vector objects into the model as possible
2. After all the nodes are supposedly over a pipeline, I proceeded to use the "Split lines by points" tool. Client wanted all of their assets in the model. I did this thinking that it'll help with the import and produce less "JXXXX" junctions in EPANET.

I believe this splitting of the lines did something to the importing process. Not sure? Some pipelines were not imported at all, and what got imported made the model very laggy. Every frame movement required a loading time in QGISRED, which shouldn't be the case for a smallish model like this.

Any help is appreciated!

[diogomarin]

@Lolaento Q1.Files_modified.zip

Errors detected or that may have happened.

1. Pipeline with the same identifier - When importing the SHP layer, an error may be generated. See the Netowork_test1 project that the reference layer was the original Pipeline

2. Solving the ID problem, when importing the SHP you must pay attention to spatial tolerance, because depending on the imposed value you will concatenate some nodes and automatically you will exclude some pipes Spacial Tolerance = Node Tolerance (Default Values) In the Netowork_test2 project, the ID problem was fixed and the Pipeline_modified layer was imported with the spacial tolerance equal to zero, then I used the connectivity check and everything is consistent.

3. When importing valves, attention must be doubled. First with the size that the valve will assume will be determined in the Default Values/Tolerance and the same attention to spatial tolerance that I mentioned earlier. And be careful with the values ​​of the original Valves layer attribute table layer when importing to QGISRed. Below are the default values ​​adopted and the results you will see in the Network_test3 project

As a suggestion to @neslerel , the largest of the valves are blocking (Isolation valves) are the same diameter as the pipe. Only the control valves which diameter are smaller, perhaps an automation is feasible

Anyway, I hope I helped

[neslerel]

Awesome @diogomarin!! A very good report/fix of the general issues ocurred when importing shapefiles! Regarding your suggestion, I noted it.

@Lolaento if you have some other problem, please detail it and we'll try to help you (or fix some possible bug).

[Lolaento]

Thanks guys! @diogomarin @neslerel

I'll have a go at changing my defaults.

Regarding my previous question 2A). Any insight on this? The main system I am modelling is servicing approximately 30,000 people and I'd like to automate the model build in anyway possible.

[Lolaento]

@diogomarin @neslerel

Sorry to trouble again, could one of you lovely gents please give me a brief lesson on how the "Demands manager" function works? I'm trying to assign my meter demands on to nearby junctions :)

[neslerel]

Hi @Lolaento,

About your second question: 1) The "Node tolerance" (in Default values) or "Spatial tolerance" in the import process (I noted it to unify these labels) could help you so that some nodes (the ones that do not match the end of the pipes) would be included in the model. You can increase this parameter (e.g. 0.1 m, 0.2 m). However it could generate that in case two nodes are in this tolerance area, only one was preserved. If this could be a problem, use the "Snap points to lines" tool that you mentioned is a good solution.

2) Regarding to use the "Split lines by points" tools, it also a valid step before importing this layer in a QGISRed project. Another option is not use this tool, import the "original" pipe layer and then use the "Create T connections" tool (Verifications menu). In any case, if you can send me the splitted pipe layer, I can check it to know what happens (I think it is caused by the Ids of the elements as @diogomarin suggested).

About the Demand Manager, I think @fmartine is going to answer you. However, if you detail your case, what data you have, in what format... Because, there is another tool (Load meter readings) that could be help you.

[fmartine]

Hi @Lolaento,

The Demand Manager is new in beta version 0.15 and is not completely finished, but it aims to be a platform from which the assignment of demands to nodes can be managed, based on the geolocation of consumptions. So far we have developed two cases:

• If you know the total demand of a sector, you can distribute this demand among the nodes located within the sector, with some options.

• If you know the consumption recorded at specific locations, you can assign these consumptions to the closest nodes (deploy option From punctual consumptions). I think this second case is the one that interests you. In this case, you have to select the theme of points where consumptions are located, and the field corresponding to the demand (base demand). For now, the flow units must be the same as the project. QGISRed will locate automatically the closest node (just junctions) of each consumption to allocate it. Note that many punctual consumptions can be allocated to the same node, being added.

• There is a couple of options to constrain the nodes that can be loaded. One is to consider just those nodes selected by the user using the select tool of QGIS, and the second, compatible with the previous one, is to consider just the end nodes of pipes under a certain diameter.

In the future, new options will be added to consider block demands or linear consumptions. Also, the hydraulic efficiency could be considered to account for the NRW and patterns could be assigned by sectors.

• You can overlap demands and patterns assigned directly to some nodes, with demands allocated following previous methods (they can also overlap). Finally, the total demands can be exported under some tag to configure a demand scenario. Next, different demand scenarios can be reloaded (import) by replacing previous ones o by adding them. Finally, if you are in a mess, you can remove all demands and start again with the last option of the manager.

Hope this will be enough to cover your current needs.

Just I like to highlight that QGISRed can also manage demands closely to reality, by using the Digital Twin menu. For that, you must declare first the service connections, and then associate them with meter readings. But this is out of this issue.

[neslerel]

Hi @diogomarin,

We already implemented that when valves are imported (as points) without diameter info, the default diameter is setted with the diameter of the pipe where they are inserted (as line). It will be available in next version (0.16) or non official beta version (0.15.1). In the future, the user will be able to configure this default value with the option commented or a default value (as it was implemented until now).

[diogomarin]

@neslerel Very good, it will help a lot in structuring the model. Congratulations on developing the tool.

I made a video demonstrating the demand import process, maybe it will help @Lolaento , follow the link: https://youtu.be/-2YO593FOM0

In the video I show some process:

1. Importing demand from polygons associated with consumption - OK!
2. Importing demand from the hydrometer georeferencing associated with consumption - OK!
3. Generating the "services connections" automatically - Algorithm that I'm using takes a long time but it all worked out and importantly it is possible to identify areas without networks.
4. Adding services connections to QGISRed - OK!
5. Association of consumption of georeferenced hydrometers to services connections - Pending!

I was just curious to know which algorithm was used in both processes. Since I usually do the demand distribution process using some QGIS algorithms, for example: "Join attributes by nearest" "Join distance to nearest hub (line to hub)"

My question is: In the demand management window, is the methodology close to the services connections? From the teacher's explanation @fmartine , I understand that no.

In the video, I ended up showing the python algorithm that runs in QGIS and automatically generates the services connections, but then I have to associate the georeferenced hydrometers with the junctions that generated are generated by the conversion of the services connections.

I'm imagining that the automatic process of entering the consumptions using the service connections will be through the "load meter readings" window. Also because so far there is no structuring of the base demand to insert in the nodes.

Going a little further, I hope I'm not going too far. But, I want to leave as a reflection the process of "conversion of services connections", I understand that it is the appropriate methodology to represent reality in the mathematical model, but the model ends up getting very dense with the addition of pipes and/or junctions. Is it not possible to leave the services connections "hidden" and transfer the sum of consumption automatically to the connected pipe and then 50% to the upstream and downstream nodes of each pipe, or leave the option after refining the model to convert?

Follow the code used: code.txt

[fmartine]

Thank you very much Diogo for checking the new tools developed in v0.15 for demand allocations, and for your interesting video.

Using service connections to allocate demands instead of georeferenced punctual consumptions is in fact our proposal in order to build professional hydraulic models (Digital Twins). In your last step, you have created the service connections from hydrometers using a python program, which takes a long time. In the next version 0.16, we will offer this option to do it faster either as a Tool or directly while importing connections from a point theme. Currently, connections are imported as a line theme but in our road map, it was considered also to import them from a point theme and create the connections automatically as a perpendicular segment to the closest pipe or just to the closest node (for areas without pipe networks). In such a case the ID of the points (hydrometer) will be assigned to the created connections, and demands will be associated with connections through the ID instead of launching a spatial query, so you can load different consumptions (i.e. for different periods) much faster, whithout to add or create a new point theme each time. Moreover, service connections don't need to be converted into pipes, because their demands will be allocated automatically to the closest nodes.

Regardless of future developments, you have tried to create the service connections by yourself and next, you have imported them as a line theme creating new nodes at both ends of each connection. You can now finish your work by allocating again demands from hydrometers (point theme) to the closest nodes, which will be the free end nodes of the connections, i.e. themselves because distances will be 0. Try to do it to finish your last trial. Your proposal could be an option but we never have thought of following this procedure.

Finally, I will like to ask you for sharing your network data (the GISRed model, plus the hydrometer theme) to check our algorithms to create connections automatically. By the way, we are not using any tool provided by QGIS, GCAL or SAGA libraries to launch spatial queries. All of them are coded from scratch in order to keep our software free from third parties, and sometimes to be faster and address the queries that we really need.

Thank you again for your support.

[diogomarin]

Understood, good to know that the demands allocated in the service connectios will be transferred to the next node.

As I said, I had imagined that it would be mandatory to convert the services connections into a pipe or a node.

I will forward the demonstrated model to your email - fmartine@hma.upv.es

OK?

I will add some verification data from the modeling, data obtained from the field and others from the SCADA system - To help in the development of verification and calibration tools for the hydraulic model.

As much as I can help, I'm here. Thank you for your attention, @fmartine

[fmartine]

The idea of QGISRed to avoid increasing the size of the hydraulic model with thousands of small components such as service connections or isolation valves is to treat them as separate elements but provide them with the same functionalities as if they were part of the model.

Therefore the service connections do not need to be converted into pipes or the isolation valves into regulating valves to deal with them. However, there will always be the option of integrating them into the model or extracting them.

Thank you @diogomarin for sharing your data and offering us SCADA field data, which we will need when addressing the QGISRed calibration section. You can use my e-mail for that.

[fmartine]

Hi Diogo, We have already programmed the automatic creation of service connection from a point theme as perpendicular segments to pipes. We would like to have your data to test the algorithm. Can you send them confidentially to my particular e-mail? We will make just a private use of them.

[diogomarin]

Hi @fmartine I'll meet you, forgive the delay.