Cannot compile ANY custom pluggable validation

[alfredomusumeci]

Description

I've been trying for a few weeks now to create a custom pluggable validation and have that instead of DefaultValidation. The code is as below:

package main

import (
    "fmt"
    "github.com/golang/protobuf/proto"
    "testing"

    validation "github.com/alfredom/fabric/core/handlers/validation/api"
    . "github.com/alfredom/fabric/core/handlers/validation/api/capabilities"
    . "github.com/alfredom/fabric/core/handlers/validation/api/identities"
    . "github.com/alfredom/fabric/core/handlers/validation/api/policies"
    . "github.com/alfredom/fabric/core/handlers/validation/api/state"
    "github.com/hyperledger/fabric-protos-go/common"
    "github.com/hyperledger/fabric/protoutil"
    "github.com/pkg/errors"
    "github.com/stretchr/testify/require"
)

// SampleValidationPlugin is an example for a validation plugin,
// and is used to exercise the dependencies that the plugin validator provides
type SampleValidationPlugin struct {
    t  *testing.T
    d  IdentityDeserializer
    c  Capabilities
    sf StateFetcher
    pe PolicyEvaluator
}

// NewSampleValidationPlugin returns an instance of a validation plugin setup
// for assertions.
func NewSampleValidationPlugin(t *testing.T) *SampleValidationPlugin {
    return &SampleValidationPlugin{t: t}
}

type MarshaledSignedData struct {
    Data      []byte `protobuf:"bytes,1,opt,name=data,proto3" json:"data,omitempty"`
    Signature []byte `protobuf:"bytes,2,opt,name=signature,proto3" json:"signature,omitempty"`
    Identity  []byte `protobuf:"bytes,2,opt,name=identity,proto3" json:"identity,omitempty"`
}

func (sd *MarshaledSignedData) Reset() {
    *sd = MarshaledSignedData{}
}

func (*MarshaledSignedData) String() string {
    panic("implement me")
}

func (*MarshaledSignedData) ProtoMessage() {
    panic("implement me")
}

func (p *SampleValidationPlugin) Validate(block *common.Block, namespace string, txPosition int, actionPosition int, contextData ...validation.ContextDatum) error {
    fmt.Printf("Validation is starting")
    txData := block.Data.Data[0]
    txn := &MarshaledSignedData{}
    err := proto.Unmarshal(txData, txn)
    require.NoError(p.t, err)

    // Check if the chaincode is instantiated
    state, err := p.sf.FetchState()
    if err != nil {
        return err
    }
    defer state.Done()

    results, err := state.GetStateMultipleKeys("lscc", []string{namespace})
    if err != nil {
        return err
    }

    _ = p.c.PrivateChannelData()

    if len(results) == 0 {
        return errors.New("not instantiated")
    }

    // Check the identity can be properly deserialized
    identity, err := p.d.DeserializeIdentity(txn.Identity)
    if err != nil {
        return err
    }

    identifier := identity.GetIdentityIdentifier()
    require.Equal(p.t, "SampleOrg", identifier.Mspid)
    require.Equal(p.t, "foo", identifier.Id)

    sd := &protoutil.SignedData{
        Signature: txn.Signature,
        Data:      txn.Data,
        Identity:  txn.Identity,
    }
    // Validate the policy
    pol := contextData[0].(SerializedPolicy).Bytes()
    err = p.pe.Evaluate(pol, []*protoutil.SignedData{sd})
    if err != nil {
        return err
    }
    fmt.Printf("Validation finished")
    fmt.Printf("Hello from Alfredo")
    return nil
}

func (p *SampleValidationPlugin) Init(dependencies ...validation.Dependency) error {
    fmt.Println("Inializating")
    for _, dep := range dependencies {
        if deserializer, isIdentityDeserializer := dep.(IdentityDeserializer); isIdentityDeserializer {
            p.d = deserializer
        }
        if capabilities, isCapabilities := dep.(Capabilities); isCapabilities {
            p.c = capabilities
        }
        if stateFetcher, isStateFetcher := dep.(StateFetcher); isStateFetcher {
            p.sf = stateFetcher
        }
        if policyEvaluator, isPolicyFetcher := dep.(PolicyEvaluator); isPolicyFetcher {
            p.pe = policyEvaluator
        }
    }
    if p.sf == nil {
        p.t.Fatal("stateFetcher not passed in init")
    }
    if p.d == nil {
        p.t.Fatal("identityDeserializer not passed in init")
    }
    if p.c == nil {
        p.t.Fatal("capabilities not passed in init")
    }
    if p.pe == nil {
        p.t.Fatal("policy fetcher not passed in init")
    }
    return nil
}

I have then added this to my core.yaml file after having compiled it as a plugin (.so):

custom:
  name: CustomValidation
   library: /etc/hyperledger/fabric/plugins/test_validation_no_flags.so

and started up my network.sh file with the command network.sh up.

Finally, I get the error:

To give a bit more context, this is my go.mod:

module main
go 1.20

replace github.com/alfredom/fabric => /home/alfredo/go/src/github.com/alfredom/fabric

replace github.com/willf/bitset v1.2.1 => github.com/bits-and-blooms/bitset v1.2.1

require (
    github.com/alfredom/fabric v0.0.0-00010101000000-000000000000
    github.com/golang/protobuf v1.5.3
    github.com/hyperledger/fabric v2.1.1+incompatible
    github.com/hyperledger/fabric-protos-go v0.3.0
    github.com/pkg/errors v0.9.1
    github.com/stretchr/testify v1.8.2
)

require (
    github.com/Knetic/govaluate v3.0.1-0.20171022003610-9aa49832a739+incompatible // indirect
    github.com/davecgh/go-spew v1.1.1 // indirect
    github.com/fsnotify/fsnotify v1.4.9 // indirect
    github.com/hashicorp/hcl v1.0.0 // indirect
    github.com/hyperledger/fabric-amcl v0.0.0-20210603140002-2670f91851c8 // indirect
    github.com/hyperledger/fabric-lib-go v1.0.0 // indirect
    github.com/magiconair/properties v1.8.1 // indirect
    github.com/miekg/pkcs11 v1.1.1 // indirect
    github.com/mitchellh/mapstructure v1.4.3 // indirect
    github.com/onsi/ginkgo v1.16.5 // indirect
    github.com/pelletier/go-toml v1.8.1 // indirect
    github.com/pmezard/go-difflib v1.0.0 // indirect
    github.com/spf13/afero v1.3.1 // indirect
    github.com/spf13/cast v1.3.1 // indirect
    github.com/spf13/jwalterweatherman v1.1.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/spf13/viper v1.7.0 // indirect
    github.com/stretchr/objx v0.5.0 // indirect
    github.com/subosito/gotenv v1.2.0 // indirect
    github.com/sykesm/zap-logfmt v0.0.4 // indirect
    go.uber.org/atomic v1.7.0 // indirect
    go.uber.org/multierr v1.6.0 // indirect
    go.uber.org/zap v1.19.0 // indirect
    golang.org/x/crypto v0.1.0 // indirect
    golang.org/x/net v0.7.0 // indirect
    golang.org/x/sys v0.5.0 // indirect
    golang.org/x/text v0.7.0 // indirect
    google.golang.org/genproto v0.0.0-20220718134204-073382fd740c // indirect
    google.golang.org/grpc v1.48.0 // indirect
    google.golang.org/protobuf v1.28.1 // indirect
    gopkg.in/ini.v1 v1.51.0 // indirect
    gopkg.in/yaml.v2 v2.4.0 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)

---

In order to ensure that I am using the same compiler options, I have cloned fabric-tools container, and made an image out of it. Then, I started my own container which I use only to build my plugin in .so (this takes care of using the right version of glibc, go, and various other options). When the file is compiled, I then move it in the folder where I have mounted my volume for the ./network.sh to locate the file. In this particular instance, my error is related to google.golang.org/protobuf/internal/detrand, however, before this was google.golang.org/protobuf, and earlier runtime/internal/atomic. In essence, a new error always pop-ups.

Things I noticed: although I am using a clone of fabric 2.5.0 repo, I still need to import "github.com/hyperledger/fabric/protoutil". If I don't do this, the file won't build as the compiler will complain that:

imports github.com/alfredom/fabric/protoutil
vendor/github.com/alfredom/fabric/protoutil/commonutils.go:17:2: use of internal package github.com/hyperledger/fabric/internal/pkg/identity not allowed

It seems like, there might be a problem with google-protobuf package as that is the one that seems to be complaining the most. However, I have made sure that my go.mod is identical to the one used in the fabric branch 2.5.0, thus I have no idea why this is happening.

Lastly, I tried building a plugin out of a hello world plugin, and that worked (it complained about missing specific interfaces, but that was expected as I didn't add PluginFactory and all that). However, I'd need to add dependencies specifically to protobuf to unpack my message and as soon as I do that everything would break.

Useful Info: I am using a clone of 2.5.0 Fabric branch, the docker images are also up to date.

Steps to reproduce

NB: this uses the fabric samples with 3 orgs.

1. Commit the existing container to a new Docker image: docker commit :

2. Run the container docker run -d --name : tail -f /dev/null

3. Install necessary packages docker exec -it sh then type su and install vim and git-build-essentials. su apt-get install -y curl git build-essential apt-get install -y curl vim

4. Copy cloned repository to the container GOPATH (NB: adjust paths accordingly) docker cp /home/alfredo/go/src/github.com/alfredom/fabric :/opt/gopath/src/github.com/fabric

5. Copy plugin to the container docker cp /home/alfredo/go/src/github.com/alfredom/fabric-samples/test-network-3-orgs/custom_validation/test_validation.go 5f2397807828:/opt/gopath/src/github.com/hyperledger/fabric/peer/test_validation.go

6. Run go mod and edit go mod init main echo 'replace github.com/alfredom/fabric => /opt/gopath/src/github.com/fabric' >> go.mod (adjust this accordingly) echo 'replace github.com/willf/bitset v1.2.1 => github.com/bits-and-blooms/bitset v1.2.1' >> go.mod

7. Build the plugin go build -buildmode=plugin -o test_validation_no_flags.so test_validation.go

8. Copy it out to the destination folder (adjust accordingly) docker cp :/opt/gopath/src/github.com/hyperledger/fabric/peer/test_validation_no_flags.so /home/alfredo/go/src/github.com/alfredom/fabric-samples/test-network-3-orgs/custom_validation/test_validation_no_flags.so

9. ./network.sh down; ./network.sh up; then show logs or alternatively change the following line in the network.sh file DOCKER_SOCK="${DOCKER_SOCK}" ${CONTAINER_CLI_COMPOSE} ${COMPOSE_FILES} up -d 2>&1 to DOCKER_SOCK="${DOCKER_SOCK}" ${CONTAINER_CLI_COMPOSE} ${COMPOSE_FILES} up.

[yacovm]

Unfortunately, the Fabric endorsement/validation plugins was introduced in Fabric v1.2. Notice the import path of the protobufs in v1.2 and the same path in the latest Fabric.

Since Go plugins only work if the plugin was compiled with the exact same version of all dependencies as the binary that loads it, you need the fabric-protos-go and fabric repositories to have the same protobuf versions.

In the beginning it was easy to ensure that, as everything was in the same project, but then it was decided to split Fabric core into several projects... which makes things harder but it should still be possible.

It seems that the google protobuf's versions are the same in both fabric-protos-go and fabric so I think that part should work.

I still need to import "github.com/hyperledger/fabric/protoutil". If I don't do this, the file won't build as the compiler will complain that:

You can try and copy the code from that package into your program. The protoutil only has helper functions.

In any case, you should also read this JIRA issue.

[alfredomusumeci]

Hello yacovm, thank you for your answer. In that case, why hasn't it been proposed to update the binary that loads the plugin to the latest version of fabric? Also, do you know the name of such binary? Perhaps, if I were to disassemble it, I could see what are the different versions of packages required, as this feels much like I am playing random with finding the correct dependencies. Moreover, the dependencies in my go.mod are the exact same as the latest branch, hence I would need to start modifying all deps inside the cloned fabric repository and that may start to break everything, so I believe it may be a better approach if I had a list of deps versions I could look at?

Also, I see that you're planning to remove the support for plugins altogether. In that case, if I wanted to modify both the Lifecycle Endorsement Policy and also the Chaincode Endorsement Policy, I have no choice but to change the underlying fabric source code and rebuild the images, is that right?

[yacovm]

In that case, why hasn't it been proposed to update the binary that loads the plugin to the latest version of fabric?

What do you mean? That binary... is the Fabric peer.

Moreover, the dependencies in my go.mod are the exact same as the latest branch, hence I would need to start modifying all deps inside the cloned fabric repository and that may start to break everything, so I believe it may be a better approach if I had a list of deps versions I could look at?

What I suggest is that you first check if your solution works with Fabric 1.2 codebase. Compile a Fabric v1.2 peer and a plugin and try to make the validation plugin work. If it doesn't work there, you have a fundamental problem unrelated to Fabric. If it works, then the problem is some dependency discrepancy.

Also, I see that you're planning to remove the support for plugins altogether. In that case, if I wanted to modify both the Lifecycle Endorsement Policy and also the Chaincode Endorsement Policy, I have no choice but to change the underlying fabric source code and rebuild the images, is that right?

I don't think there is any plan to do anything in that area, anytime soon :-)

[Chahatevil]