search

OpenWaterAnalytics / epanet-msx

multi-species extension to epanet
MIT License
13 stars 18 forks source link

Source code change #6

Closed fengshang1972 closed 3 years ago

fengshang1972 commented 3 years ago

Source code change to have EPANET2.2 routing method, include mass balance and OPENMP.

fengshang1972 commented 3 years ago

Source codes of MSX are updated.

  1. EPANET2.2 topological routing method is adopted.
  2. MASS balance check is added for RATE species
  3. OPENMP is added to solve reactions parallelly.

Make files were not done. The build folder needs to be restructured.

samhatchett commented 3 years ago

Thanks @fengshang1972 - this is a very large PR. A few initial thoughts and questions.

It would be great to have a better sense of what might change in the build tooling - whether OpenMP is a hard requirement, how to enable/disable it as a compile- (or run-)time option. Also how you think the build folder needs to be restructured. I see there are platform subdirectories, so I imagine you are thinking to simplify that. I would suggest looking at CMake since that is a widely used standard build tool.

Is this code buildable as-is, and on what platforms?

Do you have a sense of the practical speedup of parallel execution, or benchmarks to share?

Looks like you added some solver structs - are these new/different solvers, or just making the existing ones easier to manage?

Can you link us to information on the new topo routing method? I know that's buried somewhere in the EPANET project threads, or maybe there's a good writeup in the EPANET docs...

And finally, I think there are some code readability issues (inheriting from the original codebase, like single-line if statements) that might benefit from adopting a code formatting convention. Formatting perhaps should be a separate issue though.

I'm excited to see what other people notice in the PR, and where the conversation will go!

fengshang1972 commented 3 years ago

Sam

As we have discussed, this PR is for development branch. I just wanted to reactivate the development of MSX.

Last night, Lew provided the make files for MSX, for WINDOWS. He is always so fast. Kind of the same approach for EPANET 2.2. I noticed Jim had a make file for Linux. We will try to see if that still work.

I tested for the chloramine example, 40% computational time is cut. with OPENMP. My computer is really low end. We know OPENMP will not have a huge reduction though. GPU is probably the future. But much more work is needed.

EPANET 2.2's WQ routing is different from 2.0. The routing is topological and mass balance is improved. The new MSX code adopted this approach and mass balance is reported as in EPANET 2.2. The new routing algorithm is described in 12.2 of the 2.2 manual.

The solver struct change is for better management and openmp. They are not new.

Best regards

Feng

From: Sam Hatchett notifications@github.com Sent: Thursday, February 25, 2021 10:37 AM To: OpenWaterAnalytics/epanet-msx epanet-msx@noreply.github.com Cc: Shang, Feng shang.feng@epa.gov; Mention mention@noreply.github.com Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

Thanks @fengshang1972https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffengshang1972&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548366713%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YPO4u%2F6wcuQ2wM0bhLm5wREiktziNNCvqOnNMqUM67Y%3D&reserved=0 - this is a very large PR. A few initial thoughts and questions.

It would be great to have a better sense of what might change in the build tooling - whether OpenMP is a hard requirement, how to enable/disable it as a compile- (or run-)time option. Also how you think the build folder needs to be restructured. I see there are platform subdirectories, so I imagine you are thinking to simplify that. I would suggest looking at CMake since that is a widely used standard build tool.

Is this code buildable as-is, and on what platforms?

Do you have a sense of the practical speedup of parallel execution, or benchmarks to share?

Looks like you added some solver structs - are these new/different solvers, or just making the existing ones easier to manage?

Can you link us to information on the new topo routing method? I know that's buried somewhere in the EPANET project threads, or maybe there's a good writeup in the EPANET docs...

And finally, I think there are some code readability issues (inheriting from the original codebase, like single-line if statements) that might benefit from adopting a code formatting convention. Formatting perhaps should be a separate issue though.

I'm excited to see what other people notice in the PR, and where the conversation will go!

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenWaterAnalytics%2Fepanet-msx%2Fpull%2F6%23issuecomment-785991767&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548366713%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWpLFZ7tFDGVCa%2BePtbZE32QXvT1%2B2C3vdouv%2FsZfmk%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMKPK5Q6VDUJO2IWTA5NIILTAZVAVANCNFSM4YDHZ5GA&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548376666%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KtyhTLuCXnBJ1rG1Q2ANyURcdqWmxfewGXdIz3s8NhY%3D&reserved=0.

jamesuber commented 3 years ago

Hey Feng,

First off, a big thanks from me also for re-starting development of this code base. After 15 years, I project that it is just now starting to come into its prime and will be increasingly used.

I haven't looked at your PR. I gather from the discussion that the ingestion of it may be slower, just because of its size, and perhaps also because there wasn't extensive discussion about what was going to be done, within the issues -- I may be wrong about that; sorry in advance if I am. But if there's a lesson for the future, it's that collaborative programming is enabled by communication, facilitated through the repo issues tracking, and made smoother by smaller PRs done in chunks (wherever possible). I note that this is not how Lew has done it. But Lew is not doing collaborative code development. One has to choose what model to follow.

Having said that, I want to say again, thanks for re-starting development of this code base ;-).

Finally, as some smaller particular ideas,

Thanks!

-Jim

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: fengshang1972 notifications@github.com Sent: Thursday, February 25, 2021 11:05 AM To: OpenWaterAnalytics/epanet-msx Cc: Subscribed Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

Sam

As we have discussed, this PR is for development branch. I just wanted to reactivate the development of MSX.

Last night, Lew provided the make files for MSX, for WINDOWS. He is always so fast. Kind of the same approach for EPANET 2.2. I noticed Jim had a make file for Linux. We will try to see if that still work.

I tested for the chloramine example, 40% computational time is cut. But my computer is really low end. But OPENMP will not have a huge reduction. GPU is the future. But much more work is needed.

EPANET 2.2's WQ routing is different from 2.0. The routing is topological and mass balance is improved. The new MSX code adopted this approach and mass balance is reported as in EPANET 2.2.

The solver struct change is for better management and openmp. They are not new.

Best regards

Feng

From: Sam Hatchett notifications@github.com Sent: Thursday, February 25, 2021 10:37 AM To: OpenWaterAnalytics/epanet-msx epanet-msx@noreply.github.com Cc: Shang, Feng shang.feng@epa.gov; Mention mention@noreply.github.com Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

Thanks @fengshang1972https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffengshang1972&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548366713%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=YPO4u%2F6wcuQ2wM0bhLm5wREiktziNNCvqOnNMqUM67Y%3D&reserved=0 - this is a very large PR. A few initial thoughts and questions.

It would be great to have a better sense of what might change in the build tooling - whether OpenMP is a hard requirement, how to enable/disable it as a compile- (or run-)time option. Also how you think the build folder needs to be restructured. I see there are platform subdirectories, so I imagine you are thinking to simplify that. I would suggest looking at CMake since that is a widely used standard build tool.

Is this code buildable as-is, and on what platforms?

Do you have a sense of the practical speedup of parallel execution, or benchmarks to share?

Looks like you added some solver structs - are these new/different solvers, or just making the existing ones easier to manage?

Can you link us to information on the new topo routing method? I know that's buried somewhere in the EPANET project threads, or maybe there's a good writeup in the EPANET docs...

And finally, I think there are some code readability issues (inheriting from the original codebase, like single-line if statements) that might benefit from adopting a code formatting convention. Formatting perhaps should be a separate issue though.

I'm excited to see what other people notice in the PR, and where the conversation will go!

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenWaterAnalytics%2Fepanet-msx%2Fpull%2F6%23issuecomment-785991767&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548366713%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UWpLFZ7tFDGVCa%2BePtbZE32QXvT1%2B2C3vdouv%2FsZfmk%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMKPK5Q6VDUJO2IWTA5NIILTAZVAVANCNFSM4YDHZ5GA&data=04%7C01%7Cshang.feng%40epa.gov%7C08163769ff4a425850d808d8d9a344ce%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637498642548376666%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KtyhTLuCXnBJ1rG1Q2ANyURcdqWmxfewGXdIz3s8NhY%3D&reserved=0.

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786012451, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACLJ2G42LBHKEJHA55LTAZYKZANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

michaeltryby commented 3 years ago

Hey @fengshang1972. Congratulation on your PR. Nice work! It's good to see MSX moving forward. :smile:

fengshang1972 commented 3 years ago

@jamesuber

It is kind of hard for me to divide the implementation of topological routing into small steps. I have never done coding collaboration and it will take some time for me to figure the most efficient way. Maybe we have to strike some balance between the productivity and the rules. I am sure corporate people (like you, haha) have these kind of struggle.

Performance is big issue for sure. Lew's real time compiling improved the speed significantly. In terms of parallelization, I have little knowledge and basic OPENMP is all I know. I think I run into some bugs in my previous job. Maybe they are the same as the ones you mentioned. Anyway, we surely will continuously fix the bugs.

fengshang1972 commented 3 years ago

@michaeltryby Soon we will need QA/QC for MSX.

michaeltryby commented 3 years ago

@fengshang1972 happy to help out however I can.

jamesuber commented 3 years ago

?I understand Feng. Possibly the main takeaway may be less in form/volume of a PR, and more the form/volume of prior discussions related to the objectives of the PR.

I say this as someone who loves being by himself so much, that pandemic isolation has been a dream come true. It's hard. For all of us.

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: fengshang1972 notifications@github.com Sent: Thursday, February 25, 2021 12:26 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

@jamesuberhttps://github.com/jamesuber

It is kind of hard for me to divide the implementation of topological routing into small steps. I have never done coding collaboration and it will take some time for me to figure the most efficient way. Maybe we have to strike some balance between the productivity and the rules. I am sure corporate people (like you, haha) have these kind of struggle.

Performance is big issue for sure. Lew's real time compiling improved the speed significantly. In terms of parallelization, I have little knowledge and basic OPENMP is all I know. I think I run into some bugs in my previous job. Maybe they are the same as the ones you mentioned. Anyway, we surely will continuously fix the bugs.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786069872, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACLIVN2YVVOAXBLHTZ3TA2B3FANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

LRossman commented 3 years ago

I just wanted to clarify some things mentioned in the previous comments:

  1. In my view Feng's PR had a very clear and simple goal which was to update MSX to make it consistent with the updated WQ routine in EPANET 2.2 that improves mass conservation.
  2. I supplied Feng with Cmake scripts, derived from those used for Epanet2.2, that should work for all platforms.
  3. I used the scripts to successfully build an executable for Feng's code on Windows (VS-2017)
  4. I tested the executable on the net3 msx files that are part of the current EPA distribution and they all ran successfully.
  5. I can't see where OpenMP is included in Feng's update (I searched for a reference to "omp.h" without success). Maybe he used it in another test version of the code. Regardless, OpenMP seems to have it's speed-up peter out after about 6 or so cores are used, but it's better than nothing.
  6. @jamesuber I guess you weren't paying attention to my (ongoing) collaborative efforts on OWA/EPANET.

I feel that Feng's PR should be accepted into dev, and we can start to build more project infrastructure from there (the Cmake scripts, automated testing, etc.).

ucchejbb commented 3 years ago

@LRossman @jamesuber I agree with Lew's comment about multiprocessing having some finite speed-up potential. EPANET/MSX is going to run into communication or race conditions at some point because the problems just aren't broken up to really solve across processors. Molecular Dynamics software uses 2 timesteps to solve two different sized problems across processors, but I have never thought of a way to allow EPANET to do something similar (or if it would even produce useful results if we did). But If it offers the potential to get any speed up, it is probably worth thinking about (at the user's choosing).

fengshang1972 commented 3 years ago

@LRossman Never realized I did not inculde omp.h! Not sure how that can still work, I turned the openmp flag on in VS project properties if I want openmp, and used #ifdef _OPENMP tin the codes.

jamesuber commented 3 years ago

?I'm not sure why these race conditions would occur? Maybe the code has been changed since the inception, but the previous algorithm boiled down to a step where you just had a bucket of separable initial value problems to integrate, one for each segment. I assume that's what the OpenMP is being used to do, but since I don't actually know how OpenMP works... Anyway, I'd think that a good parallel strategy to execute that loop should be able to use near to 100% of each thread. That's all that I had meant.

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Jonathan Burkhardt notifications@github.com Sent: Thursday, February 25, 2021 3:09 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

@LRossmanhttps://github.com/LRossman @jamesuberhttps://github.com/jamesuber I agree with Lew's comment about multiprocessing having some finite speed-up potential. EPANET/MSX is going to run into communication or race conditions at some point because the problems just aren't broken up to really solve across processors. Molecular Dynamics software uses 2 timesteps to solve two different sized problems across processors, but I have never thought of a way to allow EPANET to do something similar (or if it would even produce useful results if we did). But If it offers the potential to get any speed up, it is probably worth thinking about (at the user's choosing).

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786168854, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACNKUOLUGAHTLFDO2O3TA2VATANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

jamesuber commented 3 years ago

A Zoom call is probably better than this. Maybe the leadership should call one?

@LRossman no disrespect meant; I hope you know how highly I regard the contributions you make (and actually I read every one of your OWA messages).

Nobody here should feel bad or unappreciated, cause you all are making choices about your time, to make valuable contributions to the community. IMO code writing is single-threaded, and the collaborative part that's hard involves deciding what to do and roughly how to go about it. That's all I meant. ?

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Lew Rossman notifications@github.com Sent: Thursday, February 25, 2021 2:53 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

I just wanted to clarify some things mentioned in the previous comments:

  1. In my view Feng's PR had a very clear and simple goal which was to update MSX to make it consistent with the updated WQ routine in EPANET 2.2 that improves mass conservation.
  2. I supplied Feng with Cmake scripts, derived from those used for Epanet2.2, that should work for all platforms.
  3. I used the scripts to successfully build an executable for Feng's code on Windows (VS-2017)
  4. I tested the executable on the net3 msx files that are part of the current EPA distribution and they all ran successfully.
  5. I can't see where OpenMP is included in Feng's update (I searched for a reference to "omp.h" without success). Maybe he used it in another test version of the code. Regardless, OpenMP seems to have it's speed-up peter out after about 6 or so cores are used, but it's better than nothing.
  6. @jamesuberhttps://github.com/jamesuber I guess you weren't paying attention to my (ongoing) collaborative efforts on OWA/EPANET.

I feel that Feng's PR should be accepted into dev, and we can start to build more project infrastructure from there (the Cmake scripts, automated testing, etc.).

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786159866, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACISDI5AM7TNNMZHBULTA2TFDANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

ucchejbb commented 3 years ago

@jamesuber Maybe race condition was the wrong term. I was just thinking that if you had two chemical species and solved for one on one core, and the other on a separate core, the slower of the two would always limit your solution time. Splitting the problem could still gain some speed up, but you're limited to some gain. If you split the network up, things get trickier because pipes that cross sub-network boundaries need to talk to each other. You have more potential gains in speed, because you could break the problem up more, but then you need to reconcile all the sub-networks at each step. Even that solution will still be limited by the worst performing subnetwork, as everything else has to wait for it to be solved. And you'd still have to wait for that reconciliation step, which could erase any gain you got by splitting the problem.

ucchejbb commented 3 years ago

@jamesuber We could solve hydraulics on one core, and solve WQ on a separate core (or cores) because hydraulics should always be faster than WQ, and thereby gain the speed-up by effectively eliminating hydraulics.

LRossman commented 3 years ago

More clarification on how OpenMP is implemented in MSX. It's used when chemical reactions are being evaluated in each pipe, since all computed variables are local to a particular pipe. Thus if you had 100 pipes and four processors OpenMP by default assigns 25 pipes (in order) to each processor to update their WQ at each time step. The issue that Jonathan was alluding to is when the allocation of the work to each processor becomes unbalanced, like if the pipes assigned to processor 1 take much longer to compute their reactions than those assigned to processor 2, resulting in processor 2 sitting idle while the system waits for processor 1 to finish. OpenMP has some options to avoid this condition plus we might be able to implement some type of intelligent processor assignment in code. Note that we can't parallelize on WQ species since they don't behave independently.

jamesuber commented 3 years ago

Why not do the parallel processing at the segment level? Isn't there a big bucket of segments somewhere? If you randomly assigned segments to processors, I'd think the thread work would be very well balanced. That's what I thought was happening, and so we'd be able to achieve near 100% utilization.

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Lew Rossman notifications@github.com Sent: Thursday, February 25, 2021 4:37 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

More clarification on how OpenMP is implemented in MSX. It's used when chemical reactions are being evaluated in each pipe, since all computed variables are local to a particular pipe. Thus if you had 100 pipes and four processors OpenMP by default assigns 25 pipes (in order) to each processor to update their WQ at each time step. The issue that Jonathan was alluding to is when the allocation of the work to each processor becomes unbalanced, like if the pipes assigned to processor 1 take much longer to compute their reactions than those assigned to processor 2, resulting in processor 2 sitting idle while the system waits for processor 1 to finish. OpenMP has some options to avoid this condition plus we might be able to implement some type of intelligent processor assignment in code. Note that we can't parallelize on WQ species since they don't behave independently.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786244534, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACM5HULTI62YI2KLE3LTA27JPANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

LRossman commented 3 years ago

Yes, that should help. But we'd have to retool the data structures used for segments. There's no big bucket of segments -- each pipe has its own linked list of segments which simplifies the transport step of species along the pipe (after reaction occurs). If we put all segments in one big array we'd have to find a way to access them in the correct order (along the length of their pipe), plus account for them being both added and removed, so that transport could be implemented efficiently.

jamesuber commented 3 years ago

?I see. But maybe it's not too bad? Just leave the existing pipe linked list as is, and build a throw-away list of pointers each time that can be executed in parallel? Perhaps that is so inelegant that I shouldn't have said it!

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Lew Rossman notifications@github.com Sent: Thursday, February 25, 2021 5:18 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

Yes, that should help. But we'd have to retool the data structures used for segments. There's no big bucket of segments -- each pipe has its own linked list of segments which simplifies the transport step of species along the pipe (after reaction occurs). If we put all segments in one big array we'd have to find a way to access them in the correct order (along the length of their pipe), plus account for them being both added and removed, so that transport could be implemented efficiently.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786265308, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACP3QNHYDK7ZTU4SLPDTA3EB3ANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

samhatchett commented 3 years ago

Just catching up with this thread - hi everyone! Great to have such engagement.

There seems to be several things this PR is attempting to address.

In a perfect world, each of these might have been dealt with in separate issues, and then separate PRs, as each item on the list is likely to generate opinions and possibly uncover great ideas. Dealing with this as a single PR/ single commit is tricky as we can see by the diverse multithreaded conversation above. This sort of thing catches people off-guard.

Take Jim's "bucket of segments" idea - a decent idea - which just shows that the ultimate parallel execution model that best serves the project is perhaps non-obvious. Accepting a PR that implements parallel execution into the community project without first discussing the approach with that community might tend to discourage these kinds of ideas from being shared, and prevent innovative solutions from being discovered in the first place.

To move this forward, I think the less efficient way would be to start the process over, break the concerns apart into separate discussions, and have those debates prior to merging anything. The (possibly) more efficient way is to bulk-import Feng's changes and debate what to keep and what to change after the fact - which might set a bad precedent for project governance and development flow. I truly hope that folks on this thread can understand the conundrum.

Those are my thoughts for now. I would really like to hear from you all as to the preferred way forward.

fengshang1972 commented 3 years ago

@jamesuber @LRossman @samhatchett

I did not call any functions defined in omp.h. So it is OK not including the header filer. The parallelization is done through #pragma omp parallel.

I put an example of openmp. For net3-bio.msx. Time reported in the report files are 33 seconds (with openmp) vs 70 seconds (without openmp). See the results at the end of the message.

I am a practical engineer, not a perfectionist. Having something that work and our peer researchers and engineers can start to test and comment is better than no activities for a decade. I am not a very good programmer and open to any criticisms and suggestions to modify and improve the codes. And I hope we can have more people get interested in the water quality modeling in water distribution system and start to contribute, from both research and software point of view. But I do not like getting stuck with some bureaucratic rules and unnecessary arguments. Commercial software can have endless versions and patches. Here we are just talking about a development branch. I find it kind of unbelievable. I would rather spend the time reading and writing research papers and developing codes. Anyway now I am a researcher, not a software engineer anymore. I do not rush for anything.

No intention to offend anybody. I see you as my mentors and friends.

Best regards

With OPENMP

Analysis begun Thu Feb 25 20:10:22 2021

Processing MSX input file net3-bio.msx

Page 1 EPANET-MSX 1.1

Without OPENMP

Analysis begun Thu Feb 25 20:12:59 2021

Processing MSX input file net3-bio.msx

Page 1 EPANET-MSX 1.1

ucchejbb commented 3 years ago

@LRossman @jamesuber Just to clarify, I was just proposing some possible challenges and reasons why there might be a limited improvement in parallelization, not trying to say that was how the current implementation handled things. Sorry for any confusion that may have caused. Given Lew's response related to how it works, we'd get speed for each step for segment step, but still potentially suffer as those processes have to reconcile at the nodes across cores.

We can definitely test out some scenarios, with and without parallelization and against single species EPANET to better quantify speed-up. That would be worth the effort at some point if parallelization was pursued. As I said initially, any speed up might be worth it. Testing on a variety of different sized networks, and different complexity problems might help identify where a bottleneck existed.

@samhatchett -- As to the general state of the code. I think 2 or 3 of your bullets listed were really the same thing of "just bringing the code up-to-date to function with 2.2". The second side of this discussion is that I think the people that may have actually looked at this codebase in the last 8 years are all currently on this chain. Any new person is going to have to dig pretty deep into this code to make sense of it with or without a logical version control history, and probably wouldn't be in a place to comment on its functionality in the near term anyways. For that reason, I think it makes sense to accept this version and work from there as a dev branch, making sure to address the issues that you highlighted initially before it ultimately gets merged into the main branch. As Feng/Lew discussed, this is a working branch with potential improvements in mass balance vs the original MSX. The community needed a version that worked with 2.2, and this version does that.

jamesuber commented 3 years ago

Hey Jonathan - On the parallel processing, I don't believe we were thinking you were trying to explain the current code or modifications; at least I was not, so no worries from my end on that.

I think the first approach (and likely what Feng did) was to focus only on parallelizing the solution of the ODE systems for all the segments. This was the main computational drawback of MSX, cause you're likely to have thousands of such segments that require integration during each WQ time step, and depending on the solver and the problem characteristics, that load can take some time. It was terrible I remember before Lew added the capability to dynamically compile and link the code for evaluating the ODEs, as otherwise those expressions needed to be interpreted by a stack processor.

Before we'd get too far we should obviously profile the code (with the compilation option) to confirm, but I'm guessing there are people here who have done that.

In the above scenario, the requirement to "reconcile at the nodes across cores" is outside of the segment parallel processing - it comes after and is done serially (it could be parallelized...). So really, the question is "how fast can you perform N integrations of identical ODE systems, with different initial conditions, across M processors?" (A detail is that the tank ODEs are different.) That will depend on the variation in computational characteristics of the ODE systems, but since the only difference is initial conditions and this is a physical system, there might not be that much change across the segments. And if there were, a randomization of segment assignment to processors and the law of large numbers, might help smooth things out.

Thanks for the engagement. -Jim

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Jonathan Burkhardt notifications@github.com Sent: Friday, February 26, 2021 8:25 AM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

@LRossmanhttps://github.com/LRossman @jamesuberhttps://github.com/jamesuber Just to clarify, I was just proposing some possible challenges and reasons why there might be a limited improvement in parallelization, not trying to say that was how the current implementation handled things. Sorry for any confusion that may have caused. Given Lew's response related to how it works, we'd get speed for each step for segment step, but still potentially suffer as those processes have to reconcile at the nodes across cores.

We can definitely test out some scenarios, with and without parallelization and against single species EPANET to better quantify speed-up. That would be worth the effort at some point if parallelization was pursued. As I said initially, any speed up might be worth it. Testing on a variety of different sized networks, and different complexity problems might help identify where a bottleneck existed.

@samhatchetthttps://github.com/samhatchett -- As to the general state of the code. I think 2 or 3 of your bullets listed were really the same thing of "just bringing the code up-to-date to function with 2.2". The second side of this discussion is that I think the people that may have actually looked at this codebase in the last 8 years are all currently on this chain. Any new person is going to have to dig pretty deep into this code to make sense of it with or without a logical version control history, and probably wouldn't be in a place to comment on its functionality in the near term anyways. For that reason, I think it makes sense to accept this version and work from there as a dev branch, making sure to address the issues that you highlighted initially before it ultimately gets merged into the main branch. As Feng/Lew discussed, this is a working branch with potential improvements in mass balance vs the original MSX. The community needed a version that worked with 2.2, and this version does that.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786646201, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACJAWWLPKAYWIQJEKVDTA6OLZANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

ucchejbb commented 3 years ago

Jim - Just wanted to clarify, but I'm glad we're on the same page regarding where I was coming from.

I appreciate the insight into the ODE aspect of this, and I think we're thinking the same thing from two different sides. As you say, some level of profiling is likely in order if we want to invest effort seeking additional performance gains. My guess is that the serialized step of reconciling updates after segments are calculated will be the major bottleneck that might not have an easy solution. We'd need to ensure that two links weren't trying to add mass to a node at the exact same time if trying to do that step in parallel. There may be parallel programming approaches to do that that I'm not aware of, but that type of problem would cause some issues that might not manifest the same each time or at different parallelization levels.

That said, I'm just putting those thoughts out there for a longer term discussion about performance improvements, not necessarily for this specific code.

jamesuber commented 3 years ago

?OK Jonathan - I'll just say though, that I do not think the node mass balance step is going to be the computationally demanding part. It's already done serially after the ODE updates, and I doubt it is a bottleneck in terms of CPU usage.

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions james.uber@xyleminc.com M: 513.368.6303

From: Jonathan Burkhardt notifications@github.com Sent: Friday, February 26, 2021 9:05 AM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

Jim - Just wanted to clarify, but I'm glad we're on the same page regarding where I was coming from.

I appreciate the insight into the ODE aspect of this, and I think we're thinking the same thing from two different sides. As you say, some level of profiling is likely in order if we want to invest effort seeking additional performance gains. My guess is that the serialized step of reconciling updates after segments are calculated will be the major bottleneck that might not have an easy solution. We'd need to ensure that two links weren't trying to add mass to a node at the exact same time if trying to do that step in parallel. There may be parallel programming approaches to do that that I'm not aware of, but that type of problem would cause some issues that might not manifest the same each time or at different parallelization levels.

That said, I'm just putting those thoughts out there for a longer term discussion about performance improvements, not necessarily for this specific code.

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786666962, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACPV4FKABCRLMMAYWVTTA6TBXANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

samhatchett commented 3 years ago

@fengshang1972 - would you comment on why this was closed, and what you intend by this action? thanks!

fengshang1972 commented 3 years ago

MSX update is in EPA's STRAP research plan and we have deadlines to meet. At this stage, I think I should focus on the task and deliver what we have promised in the research plan. We will ask for the community for beta testing and feedback.

From: Sam Hatchett notifications@github.com Sent: Monday, March 1, 2021 1:58 PM To: OpenWaterAnalytics/epanet-msx epanet-msx@noreply.github.com Cc: Shang, Feng shang.feng@epa.gov; Mention mention@noreply.github.com Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

@fengshang1972https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ffengshang1972&data=04%7C01%7Cshang.feng%40epa.gov%7C1e6b3a2397a9448414fa08d8dce3e718%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637502218706946418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2FC7d8l8SHOOxyvInsXA3WHLzaT1R9a2%2FOkbzSFjuEU0%3D&reserved=0 - would you comment on why this was closed, and what you intend by this action? thanks!

- You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FOpenWaterAnalytics%2Fepanet-msx%2Fpull%2F6%23issuecomment-788188647&data=04%7C01%7Cshang.feng%40epa.gov%7C1e6b3a2397a9448414fa08d8dce3e718%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637502218706946418%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4YjPcoLrwNqggWdCAJuBs%2Bq7jS1fps6vTdngqM05H58%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAMKPK5TG75ZYSJ6S5VDV5P3TBPPSRANCNFSM4YDHZ5GA&data=04%7C01%7Cshang.feng%40epa.gov%7C1e6b3a2397a9448414fa08d8dce3e718%7C88b378b367484867acf976aacbeca6a7%7C0%7C0%7C637502218706956389%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uowyCRS35PXZp98LkFPOVxT7IdLB5YxT0%2BdVQNfZb8A%3D&reserved=0.

jamesuber commented 3 years ago

I wanted to say that the way that all of this has transpired is extremely unfortunate, and I believe unnecessary. I would like to call on the community for a reset here.

First, in response to @samhatchett asking for opinions on a way forward. At this point, even though it violates standard policies of interaction within the community, my opinion is that the MSX modifications that are the result of Feng's hard work should be accepted. I have this opinion only because: 1) Feng is the original author of MSX, and thus a long-standing member of the distribution system community and someone who is well known for his capabilities; 2) Feng is relatively new to collaborative code development policies (I think by his own admission), and so it can be said that policies could be bent, in this case, and finally; 3) as mentioned by Feng, the MSX code base has been dead for some time, and so in this case it is unlikely that there will be any immediate side effects even if the code were simply accepted without review.

Having said that, I do have a couple of other things to say about our future interactions. I was extremely disappointed to hear that vigorous community debate about code design, development ideas, and priorities was being described as "bureaucratic rules and unnecessary arguments." What we're trying to foster on OWA is the equivalent of stopping by somebody's cubicle for a technical chat, battering ideas back and forth on a whiteboard, or a vigorous debate or even argument about strategies over lunch. In the "old days" that was what we referred to as collegiality. Someone who regularly disregarded or dismissed their colleagues input, or who failed to acknowledge that rules are necessary for building literally anything together as human beings, would be shunned. The world of Github has not changed this basic form of engagement, or the polite protocols that allow society to smoothly function.

I'd like everyone to think of the current OWA leadership team and what they have to do. There are people from all over the world coming here and participating. This is not some extension of a Cincinnati-based distribution system clique, where everybody completes everyone else's thoughts without effort. Our code maintainers have a responsibility to treat everyone equally, whether you're a world-known rock star or someone's beginning graduate student. So, think about that, and arrange your interactions in a way that lets them to do their volunteer​ job without any extra headaches.

Come on folks. Let's start acting like the good colleagues that we are, trustful and respectful of others knowledge, yet not afraid to ever challenge ideas in pursuit of a better feature set and code base. Behave more like you're developing code with the person around the corner, and less like Github is just another code dump. This is important stuff that we are doing, so I'd hope that we can all be collegial and learn to work together for the common good.

Thanks, -Jim

Jim Uber, Ph.D. Director, Drinking Water Network Optimization Xylem - Digital Solutions @.*** M: 513.368.6303

From: fengshang1972 @.***> Sent: Thursday, February 25, 2021 9:15 PM To: OpenWaterAnalytics/epanet-msx Cc: Uber, James - Xylem; Mention Subject: Re: [OpenWaterAnalytics/epanet-msx] Source code change (#6)

@jamesuberhttps://github.com/jamesuber @LRossmanhttps://github.com/LRossman @samhatchetthttps://github.com/samhatchett

I did not call any functions defined in omp.h. So it is OK not including the header filer. The parallelization is done through #pragma omp parallel.

I put an example of openmp. For net3-bio.msx. Time reported in the report files are 33 seconds (with openmp) vs 70 seconds (without openmp). See the results at the end of the message.

I am a practical engineer, not a perfectionist. Having something that work and our peer researchers and engineers can start to test and comment is better than no activities for a decade. I am not a very good programmer and open to any criticisms and suggestions to modify and improve the codes. And I hope we can have more people get interested in the water quality modeling in water distribution system and start to contribute, from both research and software point of view. But I do not like getting stuck with some bureaucratic rules and unnecessary arguments. Commercial software can have endless versions and patches. Here we are just talking about a development branch. I find it kind of unbelievable. I would rather spend the time reading and writing research papers and developing codes. Anyway now I am a researcher, not a software engineer anymore. I do not rush for anything.

No intention to offend anybody. I see you as my mentors and friends.

Best regards

With OPENMP

Analysis begun Thu Feb 25 20:10:22 2021

Processing MSX input file net3-bio.msx

Page 1 EPANET-MSX 1.1

*

             E P A N E T  -  M S X                     *

*

          Multi-Species Water Quality                  *

*

          Analysis for Pipe  Networks                  *

*

                  Version 1.1                          *

Two-Source Biofilm Model

Water Quality Mass Balance: TL (MG) Initial Mass: 0.00000e+00 Mass Inflow: 1.57592e+08 Mass Outflow: 1.55226e+08 Mass Reacted: 0.00000e+00 Final Mass: 2.36600e+06 Mass Ratio: 1.00000 Water Quality Mass Balance: CL2 (MG) Initial Mass: 0.00000e+00 Mass Inflow: 1.07712e+09 Mass Outflow: 6.34586e+08 Mass Reacted: -4.28713e+08 Final Mass: 1.38196e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: S (MG) Initial Mass: 0.00000e+00 Mass Inflow: 4.53595e+08 Mass Outflow: 4.29667e+08 Mass Reacted: -1.09884e+07 Final Mass: 1.29392e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: Xb (UG) Initial Mass: 0.00000e+00 Mass Inflow: 8.97599e+06 Mass Outflow: 6.77147e+07 Mass Reacted: 8.44616e+07 Final Mass: 2.57228e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: Xa (UG) Initial Mass: 0.00000e+00 Mass Inflow: 0.00000e+00 Mass Outflow: 0.00000e+00 Mass Reacted: 1.61812e+08 Final Mass: 1.61812e+08 Mass Ratio: 1.00000

Analysis ended Thu Feb 25 20:10:55 2021

Without OPENMP

Analysis begun Thu Feb 25 20:12:59 2021

Processing MSX input file net3-bio.msx

Page 1 EPANET-MSX 1.1

*

             E P A N E T  -  M S X                     *

*

          Multi-Species Water Quality                  *

*

          Analysis for Pipe  Networks                  *

*

                  Version 1.1                          *

Two-Source Biofilm Model

Water Quality Mass Balance: TL (MG) Initial Mass: 0.00000e+00 Mass Inflow: 1.57592e+08 Mass Outflow: 1.55226e+08 Mass Reacted: 0.00000e+00 Final Mass: 2.36600e+06 Mass Ratio: 1.00000 Water Quality Mass Balance: CL2 (MG) Initial Mass: 0.00000e+00 Mass Inflow: 1.07712e+09 Mass Outflow: 6.34586e+08 Mass Reacted: -4.28713e+08 Final Mass: 1.38196e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: S (MG) Initial Mass: 0.00000e+00 Mass Inflow: 4.53595e+08 Mass Outflow: 4.29667e+08 Mass Reacted: -1.09884e+07 Final Mass: 1.29392e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: Xb (UG) Initial Mass: 0.00000e+00 Mass Inflow: 8.97599e+06 Mass Outflow: 6.77147e+07 Mass Reacted: 8.44616e+07 Final Mass: 2.57228e+07 Mass Ratio: 1.00000 Water Quality Mass Balance: Xa (UG) Initial Mass: 0.00000e+00 Mass Inflow: 0.00000e+00 Mass Outflow: 0.00000e+00 Mass Reacted: 1.61812e+08 Final Mass: 1.61812e+08 Mass Ratio: 1.00000

Analysis ended Thu Feb 25 20:14:09 2021

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/OpenWaterAnalytics/epanet-msx/pull/6#issuecomment-786362084, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AASMACP6QIUE3N2QXHLF2FTTA375FANCNFSM4YDHZ5GA.

CONFIDENTIALITY NOTICE: This e-mail, including any attachments and/or linked documents, is intended for the sole use of the intended addressee and may contain information that is privileged, confidential, proprietary, or otherwise protected by law. Any unauthorized review, dissemination, distribution, or copying is prohibited. If you have received this communication in error, please contact the original sender immediately by reply email and destroy all copies of the original message and any attachments. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Xylem Inc.

samhatchett commented 3 years ago

Revisiting this PR, I can see that we have several good arguments for including the code as-is, into a development branch, for further review. Thanks all for your input!

I am a bit unclear as to the USEPA's involvement going forward. Almost get the impression that we are officially forked from here on out, but I'm open to other interpretations of the comments above. I think the community would welcome participation by all parties who wish to discuss the future of this codebase and I want to echo @jamesuber 's comments about collegiality and understanding that OSS development is a team sport. As such, we do need to understand that there will be many different opinions about development and we need a safe space to hammer out the details and find a path forward.