DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.
[X] I had searched in the issues and found no similar feature requirement.
Problem Description
The current DSS development center service includes 8 processes, and the production center service includes two processes. The number of processes is too many and the operation and maintenance is difficult, so Microservices need to be merged.
Description
Merge the existing dss-apiservice-server, dss-datapipe-server, dss-scriptis-server, dss-guide-server four independent app services into one dss-apps-server service. Merge dss-framework-orchestrator-server, dss-flow-entrance, and dss-workflow-server into dss-framework-orchestrator-server. Modify the corresponding start and stop scripts and cicd process simultaneously.
Use case
No response
solutions
1. Overall design
Merge the lib directory of the existing dss-apiservice-server, dss-datapipe-server, dss-scriptis-server, and dss-guide-server independent app services into the new dss-apps-server Services. Merge dss-framework-project-server, dss-framework-orchestrator-server, dss-flow-entrance, and dss-workflow-server into dss-server. The remote rpc call between services needs to be changed to the local rpc call.
合并前
project-server
orchestrator-dev
workflow-server-dev
flow-entrance
apiservice-server
datapipe-server
scriptis-server
guide-server
orchestrator-prod
workflow-server-prod
合并后
dss-server-dev
dss-apps-server
dss-server-prod
Modify the dss interface matching service logic of the DSS-gateway-support module. The interfaces of dss/framework/project, dss/framework/orchestrator, dss/workflow and dss/flow/entrance forward to dss-server service interface.
Develop a new CICD process for the new service structure. Keep all the old processes and create a new set of processes.
1.1 Technical Architecture
DataSphereStudio | category | lectotype | version
-- | -- | -- | --
Microservices module | Microservices governance | Spring Cloud | Finchley.RELEASE
Service Registration and Discovery | Nacos | Not involved yet
Unified Configuration Center | Managis | 1.3.5
Gateway Routing | Spring Cloud Gateway | 2.0.1.RELEASE
service call | OpenFeign | 2.0.0.RELEASE
Service Security Certification | UC | Under planning
Interface Document Engine | GITBOOK(Swagger) | Not involved yet
Service application monitoring | Spring Cloud Admin | Not involved yet
Service Link Tracking | Skywalking | Under planning
The service is degraded, fused, or restricted | Organize and compare Sentinel/Hystrix | Under planning
Load balancing between services | Spring Cloud ribbon | 2.0.0.RELEASE
Basic universal module | database | Mysql | 5.1.34(Driver version)
Data access persistence | Mybatis | 3.4.6
MVC | Spring mvc | 1.19.1
Load Balance | Nginx | 1.16.1
Project build and Management Tools | Maven | 3.0+
Distributed locks | Tentative DB implementation |
Unified Distributed cache | Research when needed | Not involved yet
Unified log collection and storage | Tentative ELK | Under planning
Message queue | Research when needed | Not involved yet
distributed transaction | Research when needed | Not involved yet
Log printing | Log4j2 + self4j | 2.17.1
Front end frame | TypeScriptis | 3.5.3
1.2 Business architecture
Component name | First-level module | Secondary module | Function point
-- | -- | -- | --
DataSphereStudio | dss-apps-server | dss-scriptis-server | Merge independent second-level service modules scriptis, user-guide, apiservice and datapipe into dss-apps-server service
dss-user-guide-server
dss-apiservice-server
dss-datapipe-server
dss-server | dss-framework-project-server | Merge project-server, orchestrator-server, workflow-server, and flow-execution into dss-server services
dss-framework-orchestrator-server
dss-workflow-server
dss-flow-execution-server
2. Module design
2.1 dss-apps-server service Merging
2.1.1 Adding Submodules
Create a dss-apps-server submodule under the dss-apps module, and create a startup class DSSAppsServerApplication, which is used to merge the lib dependent output directory of other dss-apps submodules.
2.1.2 Modifying pom Dependency
Relying on the dss-scriptis-server,dss-user-guide-server,dss-apiservice-server,dss-datapipe-server module in the pom.xml of dss-apps-server, Other submodules are relied upon to generate the final lib directory.
2.1.3 Assembly modification
Modify assembly/distribution.xml in dss-package and add a fileSet for dss-app-server to output the dss-apps-server/lib/directory to $DSS_ HOME/lib/dss-apps-server/. And retain the output directory of the original service to preserve the old service startup method.
2.1.4 Configuration File Update
Add the dss-apps-server.properties configuration file and merge the configuration file items of scriptis, guide, apiservice, and datapipe into dss-apps-server.properties. Among them, the configuration items wds. linkis.server.mybatis. and wds.linkis.server.restful. need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. And keep the old configuration files for scriptis, guide, apiservice, and datapipe to be compatible with the old service startup method. By default, the dss.properties and newly added dss-app-server.properties configurations are read. If there are any new configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.1.5 gateway Forwarding Modification
Modify the url forwarding logic of dss-gateway-support to forward interfaces of apiservice, datapipe, scriptis, and guideserver to dss-app-server.
The matching logic between the requested url and the service is as follows: The url is divided into substrings based on slashes /. If a service name contains the most substrings, the service is selected to forward the corresponding url. If the matching times are the same, the service name with the shortest length is selected. Before entering the matching rule, prefix the interfaces of these templates with a section /apps to match the dss-apps-server service. In order to be compatible with the old service startup mode, configure items to control whether to use the new forwarding logic.
2.1.6 Script Update
The corresponding scripts of scriptis, guide, apiservice, and datapipe are reserved in the sbin/ext directory, and the dss-apps-server scriptis added. Add new dss-start-all_new.sh and dss-stop-all_new.sh scripts, and add related commands for starting and stopping dss-apps-server.
The assembly/config/config. Sh script, add apps-server IP and PORT configuration items, and keep the scriptis, guide, apiservice, datapipe service corresponding to the IP and PORT configuration items, to compatible with the old way.
The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates.
A version of an old script is reserved for rollback.
2.1.7 Update the managis component template
Keep scriptis, guide, apiservice, datapipe service templates and configuration templates of DSS1.0 components, and add service templates and configuration templates of dss-apps-server.
A version of an old template is reserved for rollback.
2.1.8 Upgrading the Cluster Environment
When upgrading the cluster environment, you need to associate the services and configurations of dss-apps-server. Retain service and configuration for scriptis, guide, apiservice, datapipe. One version of the old service configuration is reserved for rollback.
2.2 Merging dss-server Services
2.2.1 Creating a Submodule
Create a new dss-server root module and create the startup class DSSServerApplication.
2.2.2 Modifying pom Dependency
It relies on dss-framework-project-server,dss-framework-orchestrator-server,dss-workflow-server,dss-flow-execut in the pom.xml of dss-server, adding dependencies to generate the final lib directory.
2.2.3 Assembly modification
Modify the assembly/distribution.xml in dss-package and add the fileSet of dss-server to output the dss-server/lib/directory to $DSS_ HOME/lib/dss-server/. And retain the output directory of the original service to preserve the old service startup method.
2.2.4 Configuration file update
Merge the configuration file items for project-server, orchestrator-server, workflow-server, and flow-execution into a new dss-server. properties. Among them, the configuration items wds.linkis.server.mybatis. and wds.linkis.server.restful. need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. By default, read the configuration of dss.properties and corresponding service dss-server.properties. If there are new dss-server.properties configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.2.5 rpc Call Modification
The remote rpc call is changed to the local rpc call and compatible with the old service startup method.
Implement a new dss-server-sender and add a configuration item isMergeService to control whether to forward the sender from project, orchestrator, and workflow to dss-server-sender. If you check in getProjectServerSender that if the configuration item is enabled, the ds-server-sender is returned. If the startup mode of service merger is adopted, you need to enable this configuration item to forward the sender of each service to the dss-server-sender, which is equivalent to the local rpc invocation. If the old service start mode is used separately, disable this configuration item and use the original remote rpc invocation mode. Implement the new senderService and change the factory to the new one.
rpc call from orchestrator-server to workflow-server before service merging:
rpc call from orchestrator-server to workflow-server after service merger:
2.2.6 Conflict handling of beans with the same name
Integrate all modules' BmlService classes into the dss-commons package to provide a unified bmlService utility class. Remove the BmlService classes of project-server, orchestrator-server, and workflow-server, and introduce the bmlservice of the dss-commons module into the user side. To register as a bean through the @bean annotation of springconf, you need to register your own bean in project, orchestrator, and workflow with a unique name because you want to be compatible with the two startup methods of service merging and separation.
MetaExportService and MetaImportService in orchestrator and workflow-server need to register and use their own beans in orchestrator and workflow-server, and their names must be unique.
2.2.7 rpc Compatible at release time
2.2.8 Compatible with AppconnManagerClient
2.2.9 Script Update
Add the dss-apps-server script in the sbin/ext directory. dss-start-all_new.sh and dss-stop-all_new.sh scripts were added, and related commands for starting and stopping dss-server were added. One version of the old start_all and stop_all scripts is reserved for rollback.
Add the IP and PORT configuration items for dss-server to the assembly/config/config.sh script.
The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates.
An old script version is reserved for rollback.
2.2.10 managis Component Template
Add the dss-server service configuration and template. And retain the original service templates.
zqburde
commented
1 year ago
Search before asking
Problem Description
The current DSS development center service includes 8 processes, and the production center service includes two processes. The number of processes is too many and the operation and maintenance is difficult, so Microservices need to be merged.
Description
Merge the existing dss-apiservice-server, dss-datapipe-server, dss-scriptis-server, dss-guide-server four independent app services into one dss-apps-server service. Merge dss-framework-orchestrator-server, dss-flow-entrance, and dss-workflow-server into dss-framework-orchestrator-server. Modify the corresponding start and stop scripts and cicd process simultaneously.
Use case
No response
solutions
1. Overall design
Modify the dss interface matching service logic of the DSS-gateway-support module. The interfaces of dss/framework/project, dss/framework/orchestrator, dss/workflow and dss/flow/entrance forward to dss-server service interface.
Develop a new CICD process for the new service structure. Keep all the old processes and create a new set of processes.
1.1 Technical Architecture
DataSphereStudio | category | lectotype | version -- | -- | -- | -- Microservices module | Microservices governance | Spring Cloud | Finchley.RELEASE Service Registration and Discovery | Nacos | Not involved yet Unified Configuration Center | Managis | 1.3.5 Gateway Routing | Spring Cloud Gateway | 2.0.1.RELEASE service call | OpenFeign | 2.0.0.RELEASE Service Security Certification | UC | Under planning Interface Document Engine | GITBOOK(Swagger) | Not involved yet Service application monitoring | Spring Cloud Admin | Not involved yet Service Link Tracking | Skywalking | Under planning The service is degraded, fused, or restricted | Organize and compare Sentinel/Hystrix | Under planning Load balancing between services | Spring Cloud ribbon | 2.0.0.RELEASE Basic universal module | database | Mysql | 5.1.34(Driver version) Data access persistence | Mybatis | 3.4.6 MVC | Spring mvc | 1.19.1 Load Balance | Nginx | 1.16.1 Project build and Management Tools | Maven | 3.0+ Distributed locks | Tentative DB implementation | Unified Distributed cache | Research when needed | Not involved yet Unified log collection and storage | Tentative ELK | Under planning Message queue | Research when needed | Not involved yet distributed transaction | Research when needed | Not involved yet Log printing | Log4j2 + self4j | 2.17.1 Front end frame | TypeScriptis | 3.5.31.2 Business architecture
Component name | First-level module | Secondary module | Function point -- | -- | -- | -- DataSphereStudio | dss-apps-server | dss-scriptis-server | Merge independent second-level service modules scriptis, user-guide, apiservice and datapipe into dss-apps-server service dss-user-guide-server dss-apiservice-server dss-datapipe-server dss-server | dss-framework-project-server | Merge project-server, orchestrator-server, workflow-server, and flow-execution into dss-server services dss-framework-orchestrator-server dss-workflow-server dss-flow-execution-server2. Module design
2.1 dss-apps-server service Merging
2.1.1 Adding Submodules Create a dss-apps-server submodule under the dss-apps module, and create a startup class DSSAppsServerApplication, which is used to merge the lib dependent output directory of other dss-apps submodules.
2.1.2 Modifying pom Dependency Relying on the dss-scriptis-server,dss-user-guide-server,dss-apiservice-server,dss-datapipe-server module in the pom.xml of dss-apps-server, Other submodules are relied upon to generate the final lib directory.
2.1.3 Assembly modification Modify assembly/distribution.xml in dss-package and add a fileSet for dss-app-server to output the dss-apps-server/lib/directory to $DSS_ HOME/lib/dss-apps-server/. And retain the output directory of the original service to preserve the old service startup method.
2.1.4 Configuration File Update Add the dss-apps-server.properties configuration file and merge the configuration file items of scriptis, guide, apiservice, and datapipe into dss-apps-server.properties. Among them, the configuration items wds. linkis.server.mybatis. and wds.linkis.server.restful. need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. And keep the old configuration files for scriptis, guide, apiservice, and datapipe to be compatible with the old service startup method. By default, the dss.properties and newly added dss-app-server.properties configurations are read. If there are any new configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.1.5 gateway Forwarding Modification Modify the url forwarding logic of dss-gateway-support to forward interfaces of apiservice, datapipe, scriptis, and guideserver to dss-app-server. The matching logic between the requested url and the service is as follows: The url is divided into substrings based on slashes /. If a service name contains the most substrings, the service is selected to forward the corresponding url. If the matching times are the same, the service name with the shortest length is selected. Before entering the matching rule, prefix the interfaces of these templates with a section /apps to match the dss-apps-server service. In order to be compatible with the old service startup mode, configure items to control whether to use the new forwarding logic.
2.1.6 Script Update The corresponding scripts of scriptis, guide, apiservice, and datapipe are reserved in the sbin/ext directory, and the dss-apps-server scriptis added. Add new dss-start-all_new.sh and dss-stop-all_new.sh scripts, and add related commands for starting and stopping dss-apps-server. The assembly/config/config. Sh script, add apps-server IP and PORT configuration items, and keep the scriptis, guide, apiservice, datapipe service corresponding to the IP and PORT configuration items, to compatible with the old way. The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates. A version of an old script is reserved for rollback.
2.1.7 Update the managis component template Keep scriptis, guide, apiservice, datapipe service templates and configuration templates of DSS1.0 components, and add service templates and configuration templates of dss-apps-server. A version of an old template is reserved for rollback.
2.1.8 Upgrading the Cluster Environment When upgrading the cluster environment, you need to associate the services and configurations of dss-apps-server. Retain service and configuration for scriptis, guide, apiservice, datapipe. One version of the old service configuration is reserved for rollback.
2.2 Merging dss-server Services
2.2.1 Creating a Submodule Create a new dss-server root module and create the startup class DSSServerApplication.
2.2.2 Modifying pom Dependency It relies on dss-framework-project-server,dss-framework-orchestrator-server,dss-workflow-server,dss-flow-execut in the pom.xml of dss-server, adding dependencies to generate the final lib directory.
2.2.3 Assembly modification Modify the assembly/distribution.xml in dss-package and add the fileSet of dss-server to output the dss-server/lib/directory to $DSS_ HOME/lib/dss-server/. And retain the output directory of the original service to preserve the old service startup method.
2.2.4 Configuration file update Merge the configuration file items for project-server, orchestrator-server, workflow-server, and flow-execution into a new dss-server. properties. Among them, the configuration items wds.linkis.server.mybatis. and wds.linkis.server.restful. need to merge the corresponding configuration items of each service in a comma separated manner, and the other non conflicting configuration items can be directly copied to the new configuration file. By default, read the configuration of dss.properties and corresponding service dss-server.properties. If there are new dss-server.properties configuration files, read the new configuration files first. If there are no new configuration files, load the previous configuration files.
2.2.5 rpc Call Modification The remote rpc call is changed to the local rpc call and compatible with the old service startup method. Implement a new dss-server-sender and add a configuration item isMergeService to control whether to forward the sender from project, orchestrator, and workflow to dss-server-sender. If you check in getProjectServerSender that if the configuration item is enabled, the ds-server-sender is returned. If the startup mode of service merger is adopted, you need to enable this configuration item to forward the sender of each service to the dss-server-sender, which is equivalent to the local rpc invocation. If the old service start mode is used separately, disable this configuration item and use the original remote rpc invocation mode. Implement the new senderService and change the factory to the new one. rpc call from orchestrator-server to workflow-server before service merging:
rpc call from orchestrator-server to workflow-server after service merger:
2.2.6 Conflict handling of beans with the same name Integrate all modules' BmlService classes into the dss-commons package to provide a unified bmlService utility class. Remove the BmlService classes of project-server, orchestrator-server, and workflow-server, and introduce the bmlservice of the dss-commons module into the user side. To register as a bean through the @bean annotation of springconf, you need to register your own bean in project, orchestrator, and workflow with a unique name because you want to be compatible with the two startup methods of service merging and separation. MetaExportService and MetaImportService in orchestrator and workflow-server need to register and use their own beans in orchestrator and workflow-server, and their names must be unique.
2.2.7 rpc Compatible at release time
2.2.8 Compatible with AppconnManagerClient
2.2.9 Script Update Add the dss-apps-server script in the sbin/ext directory. dss-start-all_new.sh and dss-stop-all_new.sh scripts were added, and related commands for starting and stopping dss-server were added. One version of the old start_all and stop_all scripts is reserved for rollback. Add the IP and PORT configuration items for dss-server to the assembly/config/config.sh script. The install and update scripts create two new templates based on the existing templates to adapt to the new deployment structure and retain the original templates. An old script version is reserved for rollback.
2.2.10 managis Component Template Add the dss-server service configuration and template. And retain the original service templates.
Anything else
No response
Are you willing to submit a PR?