[Main] Progress Report For Google Summer of Code Program

[malikrafsan]

Description

This issue will contain and track about the progress of this project (Polypheny .NET Driver)

[malikrafsan]

2 May - 12 May

1. Explore about GRPC project using .NET
   • Create a GRPC server and a GRPC client driver that uses the same proto files. The proto files are shared on those two projects so that to simulate the shared proto interface. The GRPC client driver is then published in NuGet and installed on different projects that use the client driver package. By this, we can simulate the end goal of Polypheny .NET driver -> creating a client driver for a client project
   • link: https://github.com/malikrafsan/Grpc-Dotnet-Server-Client/

2. Create a custom package using .NET and publish the package to NuGet
   • Create a simple math package using .NET that performs several features, such as arithmetic, statistic, exponential and logarithmic, and trigonometric. This package is then published on NuGet. By this, I can learn how to publish the .NET package. This package is then tested locally by installing it on a different project
   • Repo link: https://github.com/malikrafsan/Custom-Dotnet-Math-Package
   • NuGet link: https://www.nuget.org/packages/MathPackageMalikRafsan

[malikrafsan]

12 May - 19 May

1. Propose project structure and unit test
   • Add a PR that contains the proposed project structure, which contains two major directories: PolyphenyDotNetDriver and PolyphenyDotNetDriver.Tests. We split the project into the source code for the package and the test code for the package. By splitting, in the test code, we can access the source code as a client. This is done so that we can have separation of concern and (AFAIK) the client doesn't need to install the test code
   • PR Link: https://github.com/polypheny/Polypheny-DotNet-Driver/pull/1

2. Connect the project with Polypheny Prism Protobuf definition
   • This PR adds a git-submodule definition that refers to Polypheny Prism Protobuf
   • We can generate the C# files from the protobuf contract running shell file generate_pb.sh
   • The generated C# files are then added to the main program to test whether the generated C# files can be used
   • The test code is then added a simple new unit test to check whether we can access the generated C# successfully through the main program
   • PR Link: https://github.com/polypheny/Polypheny-DotNet-Driver/pull/2

[malikrafsan]

21 May - 29 May

Research possible alternatives to structure the code

From what I understand, .NET provides the interface and abstract class for the database connector as well -> IDbDataAdapter, IDbConnection, IDbCommand, IDbDataParameter, IDbTransaction

I think there are 3 possible approaches that we can take

• Implement interface provided .NET
• Inherit abstract class provided by .NET
• Implement our own interface (based on the previous driver, especially Go Driver)

We discussed this and decided to pick the first and second options

Build Polypheny server locally

Previously I used release results. However, this is incompatible with the current code. The solution is that we need to build the Polypheny server locally. I build it using Polypheny Control

Able to run the Golang and Python tests successfully using the built Polypheny server

Add Polypheny Connection and its test that has capabilities to open and close connection with the server, and ping the database server

The PR: https://github.com/polypheny/Polypheny-DotNet-Driver/pull/3

• This PR adds PolyphenyConnection class
• This class inherits from the DBConnection class from .NET
• This class is capable of opening a connection with the database server using the specified protobuf message
• This class is also capable to close the connection
• We also add the ping method to check connection with the server
• The test code is then added a test to check whether the open, close, and ping methods work fine

[malikrafsan]

30 May - 5 June

Optimize The Code Performance By Converting to Async Function

After learning directly from MySQL Connector codes, it would be better to change the function to be a wrapper of the async function. For example, in the Open method, we implement it by calling the OpenAsync method and await the result then return it. Here I try to do it by using the async variants of Read and Write methods from TCP NetworkStream. The implication is that Send and Receive methods using protobuf are async methods as well. By this approach, we convert these methods into wrappers of async methods:

• Open
• Close
• Ping
• ExecuteNonQuery
• Prepare

Add Execute Non Query Functionalities And Its Test

Class DbCommand from .NET which we inherit, uses two interfaces to send SQL commands to the database, which are ExecuteNonQuery and ExecuteDbDataReader. ExecuteNonQuery is used for data manipulation and schema definition, whereas ExecuteDbDataReader is used for query.

In this progress, we have implemented the ExecuteNonQuery method successfully with its test as well.

Add Prepared Functionality for Execute Non Query

In this progress, we also have implemented the prepared functionality to include parameters in the SQL command.

Here is the screenshot of the test. In the test, we insert a row by passing parameterized values using ? mark

Migrate From Prism Import to Nuget Package

In this progress, we also migrate from importing Prism Protobuf manually to importing from Nuget Package. By this approach, we centralized the management of the Prism Protobuf to the Nuget package and do not manage it directly in the driver code

[malikrafsan]

6 June - 19 June

Add Execute Query Functionality

As mentioned previously, Class DbCommand from .NET which we inherit, uses two interfaces to send SQL commands to the database, which are ExecuteNonQuery and ExecuteDbDataReader. ExecuteNonQuery is used for data manipulation and schema definition, whereas ExecuteDbDataReader is used for query.

In this progress, we have implemented the ExecuteDbDataReader and its related methods. Those methods will return PolyphenyDataReader which inherits from DbDataReader. This class allows the user to read the query result from the database

Add Test Case For Execute Query Functionality

In this progress, we also add the test case for the Execute Query Functionality. The test case will test whether the driver is able to query the database. We add two test cases.

1. Check whether the driver is able to query predefined table
2. Check whether the driver is able to query recently inserted data from a recently created table

The left image shows the first case whereas the middle image shows the second case. The test result is shown in the right image

[malikrafsan]

20 June - 26 June

Add Execute Query Mongo

Polypheny also supports document-based databases. In this implementation, we add an interface for executing Mongo-like queries for document-based databases. This interface will return an array of dictionaries, representing document data

Add Test Case For Execute Query Mongo

In this progress, we also add the test case for the Execute Query Mongo. The test case will test whether the driver is able to query the document-based database. We add two test cases.

1. Check whether the driver is able to query the document-based data

Minor Addition for Unimplemented Methods and Code Readability

In this progress, we also change and implement minor additions, such as

• Add Polypheny Config
• Change the access modifier of methods and attributes
• Removing console output
• Implement some minor methods

[malikrafsan]

27 June - 3 July

Add PolyphenyTransaction

As Polypheny supports SQL databases, one of the main features of SQL databases is ACID properties. To ensure ACID properties, a transaction feature is needed to be implemented. This week, we implemented PolyphenyTransaction. PolyphenyTransaction is a class for managing transaction features. We also made adjustments needed for this feature to the other classes as well. By implementing this feature, we can have multiple queries that work as one unit (one transaction).

This class has two methods, which are Commit and Rollback. Commit is used to make the changes in that transaction permanent. Whereas Rollback is used to cancel all of the changes in that transaction.

Add Test Cases For Transaction Feature

In this week, we also add test cases for the transaction feature. These test cases are used to test the Commit and Rollback interfaces. Commit is tested with a scenario in which we select a table that is inserted and committed in a transaction. Rollback is tested with a scenario in which we select a table that is inserted and rollbacked.

[malikrafsan]

4 July - 17 July

Research and Add The Mechanism for Calculating and Visualizing Code Coverage

To increase the reliability of the driver, we need to add more tests to ensure that the code works properly and can be maintained easily. To quantify the testing, we can add code coverage to the driver. We need to visualize the code coverage properly to better view the code coverage result. We use the NUnit package to calculate the code coverage and output the report in cobertura format. We visualize the report by using reportgenerator library, which outputs HTML and related files. Here is the visualization of the current report

Add Test Cases for PolyphenyDataReader and ProtoHelper

We also add the test cases for the PolyphenyDataReader and ProtoHelper classes. This is still ongoing progress as there are still a lot of classes and methods that need to be tested

[malikrafsan]

18 July - 24 July

Refactor Code using Interface and Dependency Injection Pattern

In order to increase the reliability of the driver, we need to increase the test for the driver. One metric that is useful for this case is code coverage. Previously, we have set up the mechanism to calculate and visualize the code coverage. However, some codes are still low in code coverage because it is harder to test.

In order to make it easier to test and in turn increase the code coverage, we refactor the code using interface pattern and dependency injection pattern. With this pattern, we can easily make unit tests using mock interfaces that work as intended without being dependent on other parts of the code. Here is an example

Increase Code Coverage

As previously mentioned, we intend to increase the code coverage. After doing the refactor, it is much easier to add new unit tests to cover other parts of the code that are harder to test using end-to-end testing. We have successfully increase the code coverage to be 90% for line coverage and 78% for branch coverage

Add Demo Project

To make it easier for the developers to use this driver, we added a demo project that contains a lot of code examples to use this driver. By doing this, the developers who want to use this driver can take a look at the code examples and customize them by their needs. The demo project can be access at here

Add Project Documentation

In addition to the demo project, we also added more explanation in the README file of the project. This README file also can make it easier for the developers who want to use this driver. The README file covers several stuff for the project, ranging from how to setup, how to use, code examples, and how to test the driver.