Rationalise the fabric networks

[davidkel]

A summary of the fabric networks that exist in this repo

+---fabric (we need to decide on this folder completely, some of these are tied to the benchmark reports )
    +---ansible-playbooks 
    |   +---v1
    |   \---v2
    +---config_kafka      (kafka is deprecated and I doubt anyone uses it now)
    +---config_raft       (3 raft orderer configuration)       
    +---config_solo       (solo orderer)
    +---config_solo_raft  (single raft configuration - change it's name)
    +---docker-compose 
    |   +---2org1peercouchdb
    |   +---2org1peercouchdb_solo_raft (keep and rename)
    |   +---2org1peergoleveldb
    |   +---2org1peergoleveldb_kafka (kafka is deprecated and I doubt anyone uses it now)
    |   +---2org1peergoleveldb_raft (keep)
    |   \---2org1peergoleveldb_solo_raft
    +---v1
    |   +---v1.0 (should go)
    |   +---v1.1 (should go) 
    |   +---v1.2 (should go) 
    |   +---v1.3 (should go)
    |   +---v1.4.0 (should go or is it tied to the reports?)
    |   +---v1.4.1 (should go)
    |   \---v1.4.4 (should go)
    \---v2
        +---v2.0.0 (should go or is it tied to the reports?)
        \---v2.1.0 (should go or is it tied to the reports?)

• There are reports for 1.4.0, 2.0.0, 2.1.0

• the config_* directories contain a configtx.yaml and crypto-config.yaml to generate genesis blocks and crypto material

• the docker-compose subdirs use docker compose to bring up a set of nodes reference config_* directories

• the v1/v2 dirs contain old format network configs which reference config_* directories and define the version of fabric images to use with the docker compose definitions

• ansible playbooks uses ibm.blockchain_platform_manager which is now end of life and only supports 1.4

In summary we have dual system (docker-compose + ansible based) here that will try to bring up various fabric configurations on a single machine using docker. There is a lot of legacy networks which are old now 1.0, 1.1, 1.2, 1.3 and 1.4 is now out of LTS support. If we wanted to keep 1.4 for comparison reasons then we should have a 1.4.12 (the final 1.4 LTS release) release. We don't have a 2.2 equivalent either.

The other problem is that there are published reports for 1.4.0, 2.0.0 and 2.1.0 which would have used this system to set them up

Question is what do we do with all of this ?

• definitely get rid of kafka, it's been deprecated for quite a while and no-one should be using it
• definitely get rid of all the v1 entries that aren't used for reports
• It's a shame about the ansible role as it was an easy way to create a docker based network, configure it, create a channel and deploy chaincode but it's end of life and we need to move to something that can be maintained so we should get rid of that too. UPDATE: In fact the ansible role is now broken as it uses fabric 1.4.6 and 1.4.7 and both of these docker images are now broken due to expired certificates
• perhaps leave the v1.4.0, v2.0.0, v2.1.0 setup as these are tied to the reports (and the perhaps the subset of docker-compose and config dirs that it used) - this should also be migrated to the new caliper network config format so that they can still work with newer versions of caliper as the old format is deprecated)

Do we want to provide specific network configurations that can be brought up for later fabric releases such as 2.2 or beyond or do we want to leave that to the users of caliper-benchmarks to provide and this repo just provides the caliper configuration for a set of reusable benchmarks ? I'm thinking that this is the best option. The problem is that hyperledger fabric doesn't provide a way to easily build different types of fabric network that can be automated. There are lots of solutions out there, some target native (test-network-nano-bash), some target docker (test-network, minifabric) others target k8s (test-network-k8s) and there are loads of others such as hyperledger cello, hyperledger bevel and a wide variety of others stored on community githubs but it hard to know which ones are actively maintained and would be suited to use here.

My preference now is that for fabric this repo contains caliper assets with chaincodes and their associated workloads to perform some standard set of benchmarking. If we ever produce further reports and the likelyhood is that we won't then we would need to describe how the network was created and may publish the assets here for the report.

One final thing, the CI currently uses this network

NETWORK=networks/fabric/v2/v2.1.0/2org1peercouchdb_raft/api/fabric-api-solo-node-prometheus.yaml

we might need to keep it or replace it with say test-network ?

[davidkel]

On further reflection I think all the networks for fabric should be removed. We should test using fabric-samples test-network as a start but also look to what would be required to use something like mini-fabric to create a more custom network. Looking at what the current fabric networks do we have

• creates a 2 org 1 peer network
• supports kafka, raft, solo as consensus providers for orderer
• supports 1 or 3 raft orderers
• supports leveldb and couchdb
• non-tls/tls/mutual-tls networks
• prometheus/grafana support
• can create a channel, install and instantiate chaincode
• doesn't have an implementation for 2.2/2.3/2.4
• is setup for multiple different fabric versions The big issue is around creating channels/install/instantiate. This mandates that it must use the non-gateway 1.4 fabric SUT binding. The 1.4 SDK is not supported anymore and we want to move caliper away from doing network configuration and use the standard tools that are available in the community. If we leave any of the networks behind we would have to provide custom code to be able to create a channel and deploy chaincode. This is not trivial and other systems do this work.

What does test network from fabric samples provide:

• creates a 2 org 1 peer network
• supports leveldb and couchdb
• option for 3 org network
• crypto material: ca or cryptogen
• single raft orderer only
• generic channel creation cmd
• generic chaincode deployment, supports Go, Java, Node
• tls only
• prometheus and grafana support (which is enabled by default, but servers need to be launched separately)
• no option for kafka (but kafka should not be considered anyway)
• the ability to use different fabric images (depending on use of the curl command for initial download or pull yourself and tag as latest)

test-network obviously doesn't cover what was provided before but I think it provides a good start. fabric also has other test networks such as test-network-k8s, test-network-nano-bash (poor name, but really this is fabric running natively locally) which we could also look to support

[aklenik]

Maybe we shouldn't provide any custom networks of our own. I'd suggest concentrating on examples that show how to put together a "Caliper project". So for every main/useful/interesting Fabric sample chaincode (and some a corresponding sample network), we could provide a project that includes the followings:

1. Node.JS project artifacts, like package.json listing Caliper as a dependency, providing a script to run the benchmark, etc.
2. A script for cloning/downloading the fabric-samples repo (I think one already exists that handles the binaries, samples repo, and docker images) and starting/initializing a specific network with a specific chaincode. Although this could be done independently, but then certain path settings would lead out of this repo's directory.
3. Network config compatible with the started network.
4. Workload module implementation(s) for the deployed chaincode('s functions).
5. Benchmark config file including the various workload modules.

I think these projects would be more useful for users and would provide a nice starting point/skeleton for custom Caliper projects.

[davidkel]

@aklenik I've actually found caliper-benchmarks to be very easy to use with test-network (once I got to grips with everything that was in it and deleted all the custom fabric networks). Admittedly it's one big project with several different benchmarks, but as an example here are the instructions for use so far

1. create a directory eg benchroot and change to it
2. install fabric using the standard fabric curl command (which installs fabric-samples into that directory)
3. clone caliper-benchmarks into benchroot
4. cd to caliper-benchmarks
5. install caliper (npm install @hyperledger/caliper)
6. bind your sut (npx caliper bind --caliper-bind-sut fabric:2.2)
7. create a network config for test network (we could easily provide this in this repo for fabric)
8. bring up test network with a channel (network.sh up createChannel)
9. choose your chaincode to deploy
10. run the appropriate benchmark for that chaincode

So for example I want to benchmark using node fabcar in test-network directory

./network.sh deployCC -ccn fabcar -ccp ../../caliper-benchmarks/src/fabric/samples/fabcar/javascript -ccl javascript

in the caliper benchmarks directory

npx caliper launch manager --caliper-workspace ./ --caliper-networkconfig testnetwork.yml --caliper-benchconfig benchmarks/samples/fabric/fabcar/config.yaml --caliper-flow-only-test --caliper-fabric-gateway-enabled

This get's it close (ie simple to get a network setup and run a benchmark) to what you are proposing as well except that all the chaincodes/benchmarks are held in a single large repo making it hard to work out exactly whats going on, but I think that the way it's set up does provide the basis for creating a simple caliper project (I can only attest to fabric here though)

The fabric team are also looking for a set of pre-canned assets which they can use to drive a workload on various fabric configuration and this repo will fit the bill via the fabric specific fixed-asset and fixed-asset-base chaincodes in this repo

No need to provide custom networks for fabric