search

OpenTouryoProject / OpenTouryo

”Open棟梁”は、長年の.NETアプリケーション開発実績にて蓄積したノウハウに基づき開発した.NET用アプリケーション フレームワークです。 (”OpenTouryo” , is an application framework for .NET which was developed using the accumulated know-how with a long track record in .NET application development.)
https://opentouryo.osscons.jp
240 stars 55 forks source link

Development of AsyncProcessingService function #58

Closed daisukenishino closed 9 years ago

daisukenishino commented 9 years ago

Requirement

Windows Service that achieve asynchronous processing by used DBMS as a queue.

Windows Service

C# - How to Create a Windows Service - Part 1-3 - YouTube http://www.youtube.com/watch?v=uM9o8GsO_u4

Specification

I have uploaded to the following URL. https://github.com/OpenTouryoProject/OpenTouryoDocuments/blob/develop/documents/1_User_Guide/en/AsyncProcessingService/AsyncProcessingService_EN.xlsx Note: Refer to the develop branch.

User

This function will be used in the Copy tool of Azure Storage. The following URL is Copy tool of Azure Storage of specifications. https://github.com/OpenTouryoProject/CloudOS/blob/develop/documents/AzureStorageBlobDataCopyWebsite/Specification_EN.xlsx

daisukenishino commented 9 years ago

Fixed

I have fixed little about the following URL. https://github.com/OpenTouryoProject/OpenTouryoDocuments/blob/develop/documents/1_User_Guide/en/AsyncProcessingService/AsyncProcessingService_EN.xlsx Note: Refer to the develop branch.

Please check.

daisukenishino commented 9 years ago

Fixed

I have fixed little about the following URL again. https://github.com/OpenTouryoProject/OpenTouryoDocuments/blob/develop/documents/1_User_Guide/en/AsyncProcessingService/AsyncProcessingService_EN.xlsx Note: Refer to the develop branch.

When using the "AsyncProcessingService" from "AzureStorageBlobDataCopyWebsite", then I found a new requirement. Therefor, I added this requirement to specification. Please check.

Reference https://github.com/OpenTouryoProject/CloudOS/blob/develop/documents/AzureStorageBlobDataCopyWebsite/Specification_EN.xlsx

daisukenishino commented 9 years ago

Supplement (1)

Supplementary on the implementation of "asynchronous processing service infrastructure".

daisukenishino commented 9 years ago

Procedure that we do the development and review of the prototype

(1) You create a new "feature branch" on GitHub. (2) You clone this "feature branch" to local Git. (3) You add (or modify) prototype to "feature branch" on local Git. (4) You commit a added (or modified) prototype to "feature branch" on local Git. (5) You push this "feature branch" to GitHub. (6) You send pullrequest for review to "nishino branch". (7) We review the pullrequest on GitHub. (8) The following alternative

(9) If a low-level indications are many present, "commit log" is increase. In this case, create a feature branch again, and repeat the same procedure.

daisukenishino commented 9 years ago

Supplement (2)

Question

HODC is requesting you to confirm the following specification of Asynchronous Processing Service.

  1. We will implement LayerB and LayerD in a dll.
  2. We will call this user code dll using communication control in-process call .
  3. We will implement communication control call asynchronously in windows service.

Answer

  1. We will implement LayerB and LayerD in a dll.
    ---> correct.
  2. We will call this user code dll using communication control in-process call .
    ---> correct.
  3. We will implement communication control call asynchronously in windows service.
    ---> correct.

Improvement

Previously, I pointed out that have not been implemented as of specification. You did not implement this specifications in this time's modification.

Specifications as I wrote in the document previously. However, the contents of the essential process is differs greatly from specification.

Although saying from previous, it is necessary to quickly check specification when it is felt that the understanding of the specification there is a problem. And, there is a need to improve the understanding ability of the specification.

daisukenishino commented 9 years ago

Development of sample

There is a need to develop a sample as a test program.

daisukenishino commented 9 years ago

Problem of first delivery

Has the following problems.

root/

---> NG. There is a problem with the placement position.

root/files/resource/Sql/sqlserver/AsyncProcessingService/

---> OK. All query files will be placed here.

root/programs/C#/Frameworks/Infrastructure/Framework/AsyncProcessingService/

---> NG. In here, you will place only C# file.

root/programs/C#/Frameworks/Infrastructure/Framework/ServiceInterface/

---> NG. There is a problem with the placement position. Delete all. Maybe forget Delete.

root/programs/C#/Frameworks/Infrastructure/ServiceInterface/AsyncProcessingService/

---> OK. In here, you will place the Body of AsyncProcessingService.

root/programs/C#/Samples/AsyncSvc_sample/

---> NG. In here, you will place the Body of test program that to register the asynchronous task.

root/programs/C#/Samples/WinServiceProject/

---> NG. Delete all. Or, you move these to the "root/programs/C#/Frameworks/Infrastructure/ServiceInterface/AsyncProcessingService/".

daisukenishino commented 9 years ago

In development

Following pullrequest is accepted in during development.

daisukenishino2 commented 9 years ago

In response to the problem.

In order to close the gap of our understanding, I have added a "prototype model" to the following URL. https://github.com/OpenTouryoProject/SampleProgram/tree/master/PrototypeModel/AsyncProcessingService

How to Use

It is used to understand the following technical aspects of multi-threaded programming in an asynchronous processing services.

Note

Because it is a prototype model, it does not use the following technologies and functions.

daisukenishino commented 9 years ago

How to wait

To the method of waiting, there are two method.

Spin method

In Spin method, wait for a change of state by the following way. Check the such as flag at a loop inside, and to select the next action from below.

Signal method

Use the kernel object called "Thread synchronization tool kit", such as event, mutex, semaphore, waitable timer objects.

In the asynchronous processing service.

As implemented in the "prototype model", can get the workerThreads variable as follows.

ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);

Therefore, to monitor the status change of this variable by selecting the spin method.

daisukenishino commented 9 years ago

About Log (1)

In order to debug the state of the Thread of asynchronous processing service, and place the following info log output to the required site.

"LogIF.InfoLog("ASYNC-SERVICE", ex.Message);".

If an exception occurs, and outputs the following log in the catch block of try-catch.

"LogIF.ErrorLog("ASYNC-SERVICE", ex.Message);".

Please create a new config file.

SampleLogConfAsyncProcessingService.xml

Please copy and rename the following files. https://github.com/OpenTouryoProject/OpenTouryo/blob/develop/root/files/resource/Log/SampleLogConfWebService.xml

Please define as follows.

And, please be replaced by the following define the scope of the <appender name="SERVICE-IF" type="log4net.Appender.RollingFileAppender">・・・</appender>.

Set the MaxSizeRollBackups parameters as number of log files does not increase.

<!-- Rolling log file output for appender -->
<appender name="ASYNC-SERVICE" type="log4net.Appender.RollingFileAppender">
  <param name="File" value="・・・\ASYNC-SERVICE" />
  <!-- setting of rolling -->
  <param name="StaticLogFileName" value="true" />
  <param name="RollingStyle" value="size" />
  <param name="MaximumFileSize" value="10MB" />
  <param name="MaxSizeRollBackups" value="2" />
  <param name="CountDirection" value="-1" />
  <!-- Writing settings (when adding or overwritten, output encoding) -->
  <param name="AppendToFile" value="true" />
  <encoding value="utf-8" />
  <!-- Message Format -->
  <layout type="log4net.Layout.PatternLayout">
    <param name="ConversionPattern" value="- (omitted) -" />
  </layout>
  <!-- Setting of filters (range) -->
  <filter type="log4net.Filter.LevelRangeFilter">
    <levelMin value="DEBUG" />
    <levelMax value="FATAL" />
  </filter>
</appender>

For more information,

See "1.3　log4net ---> 1.3.4 Option Settings" section of the following Document. https://github.com/OpenTouryoProject/OpenTouryoDocuments/blob/master/documents/1_User_Guide/en/2_User_Guide(Leaders)_EN.doc

daisukenishino commented 9 years ago

About Log (2)

I think that the following three of the try-catch block is required.

asynchronous processing

try-catch block will catch the exception that occurred on the asynchronous processing. The following Region is enclosed in try-catch.

asyncReturnValue = (AsyncProcessingServiceReturnValue)myBusiness.DoBusinessLogic((AsyncProcessingServiceParameterValue)asyncParameterValue, iso);

And, update the status and progress rate of the state management table.

Main thread and worker thread.

main thread

Catch the exception that occurred on the main thread. And, the problem is output to the error log. Do not stop the main thread of the infinite loop even if an exception occurs.

worker thread

Catch the exception that occurred in the excepting part of the asynchronous processing on the worker thread. And, the problem is output to the error log.

daisukenishino commented 9 years ago

Change of location of the [ *.cs ] file

Target files

Change the location of the above files.

Location

Change

The reason

Circular reference occurs.

daisukenishino commented 9 years ago

Changes about SQL file's location.

About asynchronous processing service own SQL files.

SQL file location

"root/files/resource/Sql/sqlserver/AsyncProcessingService/"

How to add SQL file

How to load the SQL

Business.dll that is output from the Business project is not the entry assembly. Therefore, you can load the SQL using the EmbeddedResourceLoader.LoadAsString method, And to set the SQL using the Dao.SetSqlByCommand method.

daisukenishino commented 9 years ago

CR (Change request)

Exception handling.

Monitor the exception information in the state management table.

Write exception information to the management table.

Please check the following. specification of asynchronous processing service.

For the failure analysis.

Association of exception information of the management table and Log for the failure analysis.

Add the following modifications.

Add a "user ID" and "asynchronous processing task ID" to the user information in the argument. "asynchronous processing task ID" is primary key of the asynchronous processing tasks record. This will display the UserID and taskID to log. This is because the information of MyUserInfo is used in the log output. https://github.com/OpenTouryoProject/OpenTouryo/blob/develop/root/programs/C%23/Frameworks/Infrastructure/ServiceInterface/AsyncProcessingService/AsyncProcessingService.cs#L326

  AsyncProcessingServiceParameterValue asyncParameterValue
    = new AsyncProcessingServiceParameterValue(
            "AsyncProcessingService", "StartCopyFromBlob", "StartCopyFromBlob", "SQL", 
            new MyUserInfo(selectedAsyncTask.UserID, selectedAsyncTask.TaskId));

Service stop processing of AsyncProcessingService will not complete.

Steps to Reproduce

I do not know why not complete in the code of the current OnStop events. Please debugging the processing of OnStop event. I also knew that Onstop event is not end before all WorkerThreads in ThreadPool is completed.

・・・

ThreadPool exists only one at the single process, it is shared by other tasks, such as .NET runtime. Therefore, a method of monitoring the int workerThreads variables acquired by GetAvailableThreads method may not be appropriate.

Please check the following. Sample program of asynchronous processing service

OnStop method had a problem.

Implementation of service stop in the current OnStop method has a problem. Therefore, I want to solve this problem in the following way.

(1) OnStop method will stop the main thread at the beginning. (2) OnStop method send a stop command to the asynchronous task in the running state. (3) Asynchronous task receives this command. and, throw the "BusinessApplicationException". (4) Therefor, as a result, the worker Thread will end quickly. (5) Main thread will not perform dispatch of new tasks, this is because Main thread is stopping at this point. Therefor, OnStop method can stop asynchronous services.

Maybe last issue

The state will remain the "Register".

The state will remain the "Register" at the abort command immediately after registration. This is because main thread does not pick up the task that stop command is sent. Therefore, this task will not be dispatched to a worker thread.

if (selectedAsyncTask.CommandId == "Abort")
{
  throw new BusinessSystemException("APSAbortCommand", ....................));
}

OnShutdown method had a problem.

While restarting or shutdown OS, there is a chance that the Asynchronous service may not always call the OnShutdown method. Therefore the worker threads are killed by the OS, when we do restart or shutdown the OS. Hence the OnShutdown method may terminate inappropriately.

To resolve the above issue, we can use technique called “Enabling and Handling pre-shutdown notification”. To Implement the same in this project.

In addition, We has informed that after this fix also there is no guarantee that it will work as per our expectation. Because OS can forcefully kill any threads at any time, when you do restart or shutdown OS.