Proposal for higher level abstraction of flowsheet models, flowsheet units, and automated flowsheet construction

[avdudchenko]

There is a need for an abstraction layer that can support developing preconfigured unit models and flowsheets, that could enable automated construction of complex flowsheets and their use in GUI. Currently, unit models in IDAES/WaterTAP, etc are designed as abstract units that can be modified/used in an infinite number of ways, but many users want standard units that can be used right away for typical process simulation and analysis (example: Reverse osmosis, requires pump, RO unit and maybe ERD. This means a user needs to build a flowsheet with at least these three unit models to do a basic analysis, or adapt another flowsheet) Furthermore, there is a need to be able to easily assemble flowsheet units in any number of configurations, for example multi-stage processes or where unit order is changed. Right now this is done manually, resulting in the development of bespoke flowsheets and unit model configurations that are not portable, in many cases these flowsheets grow in complexity and become impossible to maintain or navigate. If a standard rigid structure is adopted for how flowsheet units are configured and built, it is possible to:

1. Develop flowsheet units that can be duplicated and used in different places on a flowsheet
2. Provide intuitive API for UI software or higher-level code-based API to interact with unit models
3. Provide an automated method to assemble flowsheets, initialize them, and perform analysis. I propose the development of three classes within IDAES to support this capability:
4. FlowsheetBase – provides base configuration for the flowsheet o This houses utilities/properties/functions accessible to all unit models on the flowsheet, and all unit models are built on the FlowsheetBase
5. FlowsheetUnit – This houses all information for unit desi gned to be built on a flowsheet base, unlike a normal unit model, these units have to be preconfigured for a standard rigid operation expected of the underlying sub-unit model.
6. FlowsheetBuilder - This is a tool that would manage the FlowsheetBase and FlowsheetUnits to provide automated construction, initialization, and scaling of unit models in order requested by user using simple API calls (Similar to flowsheet_builder tool – for example using strings to configure order of unit operations), this would act as code based UI. o This might not be a block constructed on a model, but a separate tool entirely

The detailed proposal that includes the functions for each class and requirements or more or less are in the attached word doc/pdf

Proposal for the higher level abstraction of flowsheet models.pdf

Proposal for higher level abstraction of flowsheet models.docx

[andrewlee94]

I'll try to get to this tomorrow if possible, but one initial comment I had was whether FlowsheetBase and FlowsheetUnit needed to be separate, or if the functionality of FlowsheetUnit could just be optional capabilities on FlowsheetBlock.

[avdudchenko]

Yes, I am open to suggestions on if we want to adopt/modify existing blocks/code or inherit them. the reality is flowsheetbase and flowsheetunit share alot of the same innards but have just minor differences. so its likely the should inherit from same base class and (probably flowsheetblock) and add functionality on top of it.

[dallan-keylogic]

My first reaction reading through the proposal PDF is that much of this can be accomplished using existing standalone flowsheets and Pyomo Config blocks. A lot of the things in the "metadata" category can be accomplished by creating sets with standard names on the FlowsheetBlock (like an inlet_ports ComponentSet, an outlet_portsComponentSet, abidirectional_ports`ComponentSet for things that can't be categorized as an inlet or outlet, design_variables and control_variables ComponentSets, etc).

[andrewlee94]

My initial thoughts after reading through the document is that this is really a proposal for a few different things (but all important) and that it would probably pay to look at each of the individually. As I see it, the different aspects are:

1. Formalizing how we define modular sub-flowsheets and any necessary API to do this. In many ways, the infrastructure to do this already exists (either through FlowhseetBlock or UnitModelBase), and we just need to create some standards on how to do this and examples to show it. The main thing I think we are missing right now is metadata on external connectivity and APIs to access it.
2. Separate to this, there is a proposal for a way to define meta-methods to assemble flowsheets from existing modules (flowsheet builder), which would rely on the metadata defined in 1.
3. There is also the idea of having pre-configured options for modular components (flowsheets or unit models); again the infrastructure to do this exists via ConfigBlocks and I think what we really need here is just some standards and examples of how to do it.
4. More options for reporting flowsheet results. Again, I think the user level API exists for this throguh the existing report method. What I think might be missing here (from early comments and discussion) is a) a way to select from key stream to display (rather than all streams), b) a way to include other key outputs in a flowsheet report (e.g. key unit or flowsheet variables, costing data, etc.), and c) documentation to make it clear that model developers can overload the report method to do whatever they want.
5. Overarching all of this, there is a need for a way to define arbitrary metadata on ProcessModels as a whole, as well as specific expected metadata for different types of models (flowsheet, unit, property, reaction, surrogate, etc.). Addressing this I think will guide us to what is needed for the other four aspects, so I would suggest we start here.

The other thing I see is that we have a lot of stakeholders in with an interest in this, and we should involve all of them in the planning stage to make sure we don't inadvertently exclude anyone. My suggestion is to start by asking the stakeholders involved (modular flowsheet builders, UI team, Project Ahuora, anyone else who is interested) to compile a list of use-cases of interest to them and the associated metadata/APIs/utilities they wish they had to support this. We can then use this to start building out a more concrete proposal for what metadata is needed and how to use it.

Some more specific questions/clarifications about the proposal:

1. Is the proposed FlowsheetBase here the same as the existing FlowsheetBlock (probably with some extensions to support some new capabilities/metadata)?
2. How does FlowsheetUnit differ from FlowsheetBase? Are these just specific realizations (sub-classes) of FlowsheetBase?
3. Regarding Initialization and Scaling, IDAES is generally moving to using the new class based tools for all of this. This makes a lot of sense for generic/ general purpose models as you often need different tools/options available for different use cases. However, for flowsheets (especially the more pre-packaged variety) a method based approach makes a lot of sense as these tend to be a lot more case-specific. I suggest we use a class-based approach for standardization, but we could have a method based API to access these for cases where it makes sense.

[avdudchenko]

@dallan-keylogic yes- i think alot of basic structure/tooling exists but just needs to be standarized/packaged - as right now I don't actually see many flowsheets being built with these

@andrewlee94 : To all your points yes, I think the key thing is standardization and ensuring all the tools can work with "control API" (e.g. Ahuora, UI, etc), specifically providing access to standard expected methods and data. Much of it really is just a metadata management issue.

For specific questions: 1) FlowsheetBase would probably inherit FlowsheetBlock to add needed standard features - (such as initialization routines for starting/terminal points and props/packages. or we could add features to FlowsheetBlock - not sure what would be less of a headache. 2) FlowsheetUnit would really just be a metadata difference for control API to differentiate by.... in my thought FlowsheetBase/Unit would inherit from Flowsheet Block and add features to it needed by Control API's and standardized methods for controlling the Base/Unit. They have slight differences but can share a lot of functionality, which would be added onto FlowsheetBlock

The key practical difference between FlowsheetBase and FlowsheetUnit is the functions necessary for managing the flowsheet by a Control API. For example, a key difference is initialization routines. The FlowsheetBase needs to have methods to initialize/scale starting points and termination points separately, so its two separate functions. Meanwhile, FlowsheetUnit should only have a single initialize function that initializes everything. This is because the control API would need to inject the initialization routine of all units in between starting and terminal point initialization. So your initialization order would go to FlowsheetBase Starting points (and props/etc) -> All flowsheetUnits -> FlowsheetBase terminal points (and props/etc, for example, costing prop package). This can be done through a config option and single initialize function as well - in fact we could have a single FlowsheetBlock class, that supports many configurations - this feels like it will get messy and is a case of an overgeneralization, where I think explicit structure might be better for this.

There might also be other use-specific functionalities that are needed for FlowsheetUnits vs FlowsheetBase, but they would have to be identified by stakeholders.

3) Yes, we should use the new tooling for scaling - I need to look into it. I think a key problem right now with iscale is how difficult it is to rescale models after the fact - in many cases, we don't allow rescaling if a model is already scaled in a given unit model (e..g we check if a scaling factor exists, and only calc scales if it does not... so calling iscale.calc_sclaing_factors a second time does not help.

I also agree on the suggestion to get stakeholders to provide input on what's needed - and how they would want to interact with an API. The proposal lacks details on API and implementation as I want to get everyone's perspective.

I believe current stakeholders are: UI team Project Ahuora Members of IDAES/WaterTAP (Prommis?)