New bootstrap structure prevents easily building go code locally

IBM-Blockchain-Starter-Kit / chaincode-bootstrap

https://ibm-blockchain-starter-kit.github.io/

Apache License 2.0

3 stars 11 forks source link

New bootstrap structure prevents easily building go code locally #23

Open yorhodes opened 6 years ago

yorhodes commented 6 years ago

Closing #19 introduces this problem.

jt-nti commented 6 years ago

I'm not sure what the problem is here but it would probably be useful to include a reference to how to set up Go for chaincode development, e.g. https://golang.org/doc/code.html. in the chaincode bootstrap documentation. See IBM-Blockchain-Starter-Kit/IBM-Blockchain-Starter-Kit.github.io#13

Leaving this issue open for now in case there are any specific changes that could be made to the project structure that would help.

jt-nti commented 6 years ago

To build the Go chaincode locally, the bootstrap project should be under the src directory in the Go workspace. For example, based on the How to Write Go Code document,

...
src/
    github.com/golang/example/
        ...
    github.com/IBM-Blockchain-Starter-Kit/chaincode-bootstrap/
        chaincode/
            ping/
                ping.go
        ...

(Presumably fabric would need to be in the workspace as well?)

Having said that, it probably makes more sense to follow the approach described in the fabric tutorial using the fabric samples where the chaincode is actually compiled inside a development chaincode container.

Unless there's a really good reason not to, we should strive to keep the starter kit aligned with the fabric samples and tutorials.

jt-nti commented 6 years ago

https://github.com/IBM-Blockchain-Archive/learn-chaincode has a bit in the readme about setting things up to compile locally

rolivieri commented 6 years ago

Hello @jt-nti, here's a summary of my findings. The chaincode component examples I have seen online are very simple and compile just fine, even when there is no src folder (and/or without the need to update GOPATH). For instance, let's use the following folder structure as an example:

Ricardos-MacBook-Pro:ping olivieri$ pwd
/Users/olivieri/git/chaincode-bootstrap/chaincode/ping
Ricardos-MacBook-Pro:ping olivieri$ ls -la
total 23512
drwxr-xr-x  5 olivieri  staff       160 Aug 23 11:59 .
drwxr-xr-x  4 olivieri  staff       128 Aug 23 11:33 ..
-rw-r--r--  1 olivieri  staff      1952 Aug 23 11:45 ping.go
-rw-r--r--  1 olivieri  staff      1383 Aug 23 11:32 ping_test.go
Ricardos-MacBook-Pro:ping olivieri$ go build
Ricardos-MacBook-Pro:ping olivieri$ go test
2018-08-23 12:00:48.882 EDT [ContractChaincodeLog] Info -> INFO 001 ########### Contract Invoke ###########
2018-08-23 12:00:48.882 EDT [ContractChaincodeLog] Infof -> INFO 002 Chaincode is healthy.
PASS
ok      _/Users/olivieri/git/chaincode-bootstrap/chaincode/ping 0.031s

The above works just fine, no hiccups. However, let's look at a folder structure that is closer to what you will find in the real world when developing chaincode:

Ricardos-MacBook-Pro:deleteStateCC olivieri$ pwd
/Users/olivieri/git/fabric-reset-world-state/deleteStateCC
Ricardos-MacBook-Pro:deleteStateCC olivieri$ ls -la
total 23544
drwxr-xr-x  6 olivieri  staff       192 Aug 21 09:15 .
drwxr-xr-x  3 olivieri  staff        96 Aug 20 21:54 ..
-rw-r--r--  1 olivieri  staff       462 Aug 20 21:54 main.go
-rw-r--r--  1 olivieri  staff      4348 Aug 22 14:27 main_test.go
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 statemanager
Ricardos-MacBook-Pro:deleteStateCC olivieri$ go build
main.go:4:2: cannot find package "deleteStateCC/statemanager" in any of:
    /usr/local/Cellar/go/1.10.3/libexec/src/deleteStateCC/statemanager (from $GOROOT)
    /Users/olivieri/go/src/deleteStateCC/statemanager (from $GOPATH)

The above fails; it cannot find a local package (statemanager). There are multiple ways to address this. For instance, you can force the developer to move the chaincode code under the default $GOPATH directory. However, as a developer, I would prefer to have the freedom to have the chaincode anywhere on my local file system. Another possible way to address this compilation issue is to ask the developer to create a symbolic link for each chaincode repo he/she is working on under the $GOPATH folder. Though this would work, it feels like a hack to me. Also, I could see junior developers asking "What is a symbolic link?" and always having to look up online the command for creating, deleting, and updating symbolic links.

The good news is that Go already has a mechanism that allows you to compile your code no matter where you have it in your local file system. To compile the chaincode shown above, you'd need to do the following: 1) Your GOPATH includes an entry for your project, and 2) your project has a src folder, as shown below:

Ricardos-MacBook-Pro:deleteStateCC olivieri$ echo $GOPATH
/Users/olivieri/go:/Users/olivieri/git/fabric-reset-world-state
Ricardos-MacBook-Pro:deleteStateCC olivieri$ pwd
/Users/olivieri/git/fabric-reset-world-state/src/deleteStateCC
Ricardos-MacBook-Pro:deleteStateCC olivieri$ ls -la
total 23544
drwxr-xr-x  6 olivieri  staff       192 Aug 23 12:46 .
drwxr-xr-x  3 olivieri  staff        96 Aug 20 21:54 ..
-rwxr-xr-x  1 olivieri  staff  12040876 Aug 23 12:46 deleteStateCC
-rw-r--r--  1 olivieri  staff       462 Aug 20 21:54 main.go
-rw-r--r--  1 olivieri  staff      4348 Aug 22 14:27 main_test.go
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 statemanager
Ricardos-MacBook-Pro:deleteStateCC olivieri$ go build
Ricardos-MacBook-Pro:deleteStateCC olivieri$

The above now compiles just fine now that I have a src folder since the GO build process is expecting it to resolve the local dependency. Maybe there is a better way to achieve the same goal (i.e. support local compilation and testing in a way that does not feel hacky). If so, let's discuss so we can choose the best option.

Also, as a side note, I'd say that having to spawn a container for compiling and testing chaincode introduces more work for the developer. As a developer, I would prefer to compile and test my code as quickly as possible with as little overhead as possible on my local platform.

jt-nti commented 6 years ago

Hi @rolivieri, thanks for the summary. I've not seen that GOPATH trick before- it doesn't seem to be a recommended approach in the Go documentation that I've found

Given these projects are meant to define best practices, my gut feeling is that it would be better to steer people towards the standard Go setup, which does seem to be much more prescriptive about structure.

Having said that, if there are strong views that including the src directory inside the repository is the right thing to do, we would need to have that discussion at a higher level to make sure that the fabric samples and @sstone1's yo generator etc. follow the same approach.

On the subject of spawning a chaincode container- that would be overkill for simple compilation but it does seem like a good way to test and debug chaincode in a real fabric environment locally.

rolivieri commented 6 years ago

Hello @jt-nti, yes, adding multiple folders to the GOPATH variable can be quite useful as shown above. Having said that, I am not sure how often this technique is used. At the same time, I have not seen docs that state that updating the GOPATH is not a recommended approach. If you have those docs at hand, could you please share them with us? I have seen the links you shared above, but I did not find anything there discouraging doing so.

My point is not so much about the src directory having to be inside the repo. Instead, my point is more along the following lines:

As a developer, I should have the freedom to have the chaincode anywhere on my local file system.
As a developer, I should be able to compile and run test cases for my chaincode as seamless as possible, as quickly as possible, and with as little overhead as possible on my local platform.

If there is a way that we can achieve the above and does not require having the src folder and updating GOPATH, then that would work just fine. I am open to ideas to achieve this.

Let's say we do not have a src folder in the repo and we do not update the GOPATH variable as described above. What other options (besides symbolic links) are there to make the developer experience as seamless as possible? For instance, say a developer clones a chaincode repo to his/her local file system with the following folder structure:

Ricardos-MacBook-Pro:fabric-reset-world-state olivieri$ pwd
/Users/olivieri/git/fabric-reset-world-state
Ricardos-MacBook-Pro:fabric-reset-world-state olivieri$ ls -la
total 48
drwxr-xr-x   7 olivieri  staff    224 Aug 22 14:48 .
drwxr-xr-x  39 olivieri  staff   1248 Aug 23 09:49 ..
drwxr-xr-x  16 olivieri  staff    512 Aug 22 15:54 .git
-rw-r--r--   1 olivieri  staff     32 Aug 18 09:43 .gitignore
-rw-r--r--   1 olivieri  staff    885 Aug 22 15:53 .travis.yml
-rw-r--r--   1 olivieri  staff  12904 Aug 22 15:54 README.md
drwxr-xr-x   3 olivieri  staff     96 Aug 20 21:54 deleteStateCC
Ricardos-MacBook-Pro:fabric-reset-world-state olivieri$ cd deleteStateCC/
Ricardos-MacBook-Pro:deleteStateCC olivieri$ ls -la
total 23544
drwxr-xr-x  6 olivieri  staff       192 Aug 23 12:46 .
drwxr-xr-x  3 olivieri  staff        96 Aug 20 21:54 ..
-rwxr-xr-x  1 olivieri  staff  12040876 Aug 23 12:46 deleteStateCC
-rw-r--r--  1 olivieri  staff       462 Aug 20 21:54 main.go
-rw-r--r--  1 olivieri  staff      4348 Aug 22 14:27 main_test.go
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 statemanager
Ricardos-MacBook-Pro:deleteStateCC olivieri$

How can we have the developer compiling and running the test cases as quickly and as seamless as possible? If there are other mechanisms, sure, let's discuss them.

sstone1 commented 6 years ago

@rolivieri @jt-nti no open source projects written in Go check in a src directory into their source control system. All of them that I've seen just have the source code at the top level or in package folders, for example:

https://github.com/golang/tools (godoc et al)
https://github.com/moby/moby (aka Docker)
https://github.com/cloudfoundry/cli (cf)
https://github.com/hyperledger/fabric

The src directory is a Go workspace directory, not a Go project directory. A single Go workspace can hold multiple projects. Most Go development tutorials I've seen show a single system wide Go workspace being set up (usually /home/user/go), with all of the Go projects being checked out into this single system wide Go workspace (e.g. /home/user/go/src/github.com/hyperledger/fabric).

This is documented in the Go documentation here: https://golang.org/doc/code.html

A typical workspace contains many source repositories containing many packages and commands. Most Go programmers keep all their Go source code and dependencies in a single workspace.

Putting the src folder into source control is a Go anti-pattern and so I would not be in favour of doing it anywhere (starter kit, yeoman generator, etc).

rolivieri commented 6 years ago

@jt-nti @sstone1 Guys, let's put aside the solution I shared about src folder (for the time being, sure, let's say that was a horrible idea). Instead, let's focus on the problem we are trying to solve:

As a developer, I should have the freedom to have the chaincode anywhere on my local file system.
As a developer, I should be able to compile and run test cases for my chaincode as seamless as possible, as quickly as possible, and with as little overhead as possible on my local platform.

Can you suggest ideas on how to achieve this (besides symbolic links)? What other options are there to make the developer experience as seamless as possible? For instance, say a developer clones from GitHub a chaincode repository to his/her local file system with the following folder structure:

$ pwd
/Users/olivieri/git/a-chaincode-repo
$ ls -la
total 48
drwxr-xr-x   7 olivieri  staff    224 Aug 22 14:48 .
drwxr-xr-x  39 olivieri  staff   1248 Aug 23 09:49 ..
drwxr-xr-x  16 olivieri  staff    512 Aug 22 15:54 .git
-rw-r--r--   1 olivieri  staff     32 Aug 18 09:43 .gitignore
-rw-r--r--   1 olivieri  staff  12904 Aug 22 15:54 README.md
drwxr-xr-x   3 olivieri  staff     96 Aug 20 21:54 chaincode1CC
drwxr-xr-x   3 olivieri  staff     96 Aug 20 21:54 chaincode2CC
drwxr-xr-x   3 olivieri  staff     96 Aug 20 21:54 chaincode3CC
$ cd chaincode1CC/
$ ls -la
total 23544
drwxr-xr-x  6 olivieri  staff       192 Aug 23 12:46 .
drwxr-xr-x  3 olivieri  staff        96 Aug 20 21:54 ..
-rw-r--r--  1 olivieri  staff       462 Aug 20 21:54 main.go
-rw-r--r--  1 olivieri  staff      4348 Aug 22 14:27 main_test.go
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 module1
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 module2
drwxr-xr-x  3 olivieri  staff        96 Aug 22 13:51 module3
$

How can we have the developer compiling each chaincode component (note that a chaincode component may have local modules as depicted above) and running their test cases as quickly and as seamless as possible? Any suggestions, thoughts? I am looking for ideas on how to solve this problem.

sstone1 commented 6 years ago

@rolivieri see previous answer 😉

As a developer, I should have the freedom to have the chaincode anywhere on my local file system.

Go chaincode developers have to do exactly the same thing for chaincode that they do for any Go code they work on; set up a Go development environment and put it into a Go workspace as defined by the GOPATH environment variable.

If they don't like that, then perhaps they should pick another programming language for their chaincode with less restrictive rules around file system layout and structure.

As a developer, I should be able to compile and run test cases for my chaincode as seamless as possible, as quickly as possible, and with as little overhead as possible on my local platform.

As long as they check their Go chaincode projects into a Go workspace, then all the appropriate Go commands (go test, go build, etc) will just work. They will automatically rebuild all projects, including any dependencies also in the same Go workspace, before running tests.

rolivieri commented 6 years ago

@jt-nti I have removed the src folder from my forked repository (chaincode-bootstrap) and will be opening a PR soon so you can review it.

@sstone1 Thanks for moving the conversation forward. This is mainly what I wanted to discuss with folks more fluent in Go than me and not so much the src folder. I am now understanding that you are saying that the freedom to have the chaincode anywhere on my local file system is not something that we should support because that's not how Go intended things to work. If that's the case, point taken. Initially, I was expecting to have that freedom mainly because of my background with other languages. However, if this is not how Go intended things to work, sure, I won't fight the language. :)

@jt-nti Given the point discussed above with @sstone1, I have a few thoughts to discuss with you about structure in the chaincode-bootstrap repo. I believe the original intention was to tell developers they can have multiple chaincode components, each component under its own folder (under the chaincode folder). For instance, at the moment we have the ping chanincode component as an example in the repo and we are telling developers that they should be creating their own chaincode components as siblings to the ping chaincode component under the same Git repo.

$ pwd
/Users/olivieri/git/chaincode-bootstrap
$ ls -la
total 56
drwxr-xr-x   9 olivieri  staff    288 Aug 28 13:13 .
drwxr-xr-x  39 olivieri  staff   1248 Aug 27 13:09 ..
drwxr-xr-x  16 olivieri  staff    512 Aug 28 14:17 .git
-rw-r--r--   1 olivieri  staff     19 Aug 28 13:13 .gitignore
-rw-r--r--   1 olivieri  staff    748 Aug 28 13:13 .travis.yml
-rw-r--r--   1 olivieri  staff  11357 Aug 24 14:39 LICENSE
-rw-r--r--   1 olivieri  staff   1166 Aug 28 13:13 README.md
drwxr-xr-x   3 olivieri  staff     96 Aug 28 13:06 chaincode
-rw-r--r--   1 olivieri  staff    346 Aug 28 13:13 deploy_config.json
Ricardos-MacBook-Pro:chaincode-bootstrap olivieri$ cd chaincode/
$ cd chaincode/
$ ls -la
total 0
drwxr-xr-x  3 olivieri  staff   96 Aug 28 13:06 .
drwxr-xr-x  9 olivieri  staff  288 Aug 28 13:13 ..
drwxr-xr-x  5 olivieri  staff  160 Aug 28 13:13 ping
drwxr-xr-x  5 olivieri  staff  160 Aug 28 13:13 component2
drwxr-xr-x  5 olivieri  staff  160 Aug 28 13:13 component3
drwxr-xr-x  5 olivieri  staff  160 Aug 28 13:13 component4

@jt-nti Can you confirm that this is your understanding too?

jt-nti commented 6 years ago

@rolivieri that chaincode structure matches the fabric samples, and a couple of othe projects I've seen, so it makes sense to me.

rolivieri commented 6 years ago

@jt-nti Thanks for confirming your understanding of the intent for this repo as described above. I will share some additional thoughts and considerations that I think we should have given this understanding. However, before doing so, can you point me here (or on slack) to the GO chaincode projects that you refer to in your comment above? My guess is that neither of those projects are using local submodules but I would like to take a look at then so we can both have the same reference point. Thanks.

rolivieri commented 6 years ago

@jt-nti Let's then use the chaincode-bootstrap repo as our reference for the following discussion.

Let's first summarize the points we have agreed on:

Go developers do not have the freedom to have the chaincode code anywhere on their local file system. Instead, they are expected to create or have a Go workspace in their local file system. Then they must place in this Go workspace the chaincode repository they plan to work with.
We expect developers to create chaincode repos following the structure outlined in this repo. In other words, we are saying that developers can have multiple chaincode components, each component under its own folder (under the chaincode folder) as siblings to the ping chaincode component.
By chaincode component, we mean a deployable unit to Fabric. In other words, that's the code that will be installed and instantiated on a Fabric network. For example, the ping chaincode component in the chaincode-bootstrap repo is a deployable unit that can be installed and instantiated on a Fabric network.

Given the above assumptions, let's say I fork the chaincode-bootstrap as a baseline for my new project and clone it into my Go workspace. Say I then add several chaincode components to this repository. This implies, I should then have a folder structure like the one below under my Go workspace:

bin/
pkg/
src/
    github.com/golang/example/

       ...

    github.com/hyperledger/fabric

       ...

    my-forked-chaincode-bootstrap/
        chaincode/
            ping/
                ping.go
            component2/
            component3/
            component4/
        ...

Before I continue with the description of the issue I'd like to bring up, are we so far on the same page? Anything you see above that differs from your understanding or vision for this repo?

jt-nti commented 6 years ago

@rolivieri I'm new to Go but that sounds like what I was expecting.

I've made a start on https://github.com/jt-nti/fabric-devenv to try out some Go chaincode and I think the structure you describe works with the dev setup from https://hyperledger-fabric.readthedocs.io/en/release-1.1/chaincode4ade.html

rolivieri commented 6 years ago

@jt-nti Thanks for confirming we are on the same page so far. Let's then expand on the sample above. Say that within one of the chaincode components listed above, we have quite a bit of code and the developer has created sub-folders inside the chaincode component folder to organize better the structure of his/her chaincode component. For instance, say that component3 has the following structure as a way to organize the files:

src/
   my-forked-chaincode-bootstrap/
     chaincode
         ping/
           ...
         component2/
              main.go
              main_test.go
         component3/
         main.go
         main_test.go
         mychaincode.go
         folder1/
         folder2/
         folder3/
         component4/
            main.go
            main_test.go

For the go files in the main package (e.g. main.go, mychaincode.go, etc.) to import the sub-modules found in folder1, folder2, and folder3, they will need to have an import statement like the following:

import (
    "fmt"
    "component3/folder1"
    "component3/folder2"
    "component3/folder3"

    ...

    "github.com/hyperledger/fabric/core/chaincode/shim"
)

Now, say you navigate to the component3 folder and you attempt to execute go build and/or go test. Would it compile? Would you be able to run the test cases locally? Either operation would fail. To make it work, you could change the import statements from component3/folderN to my-forked-chaincode-bootstrap/chaincode/component3/folderN... but in that case, what will happen then when you deploy component3 to Fabric? It will fail to compile in that environment.

I have a suggestion to address this issue, which would require a few changes... but before getting into that, I'd prefer to pause here and make sure we are still on the same page so far. Do we agree that there is an issue or do you have a different view?

jt-nti commented 6 years ago

I'm not sure I follow what the problem is. If the chaincode compiles locally, then it should be ok when deployed to fabric. I checked with the local Go guru and he didn't think it would be a problem.

There is a potential problem with folders when deploying to IBP at the moment though, which could do with fixing. That's covered in another issue IBM-Blockchain-Starter-Kit/build-lib#67 - is that the issue you were thinking of, or is there another problem as well?

rolivieri commented 6 years ago

@jt-nti Hello, we created the following repo so we can use it as reference: https://github.com/vandangogna/my-forked-chaincode-bootstrap (this repo is a fork of https://github.com/IBM-Blockchain-Starter-Kit/chaincode-bootstrap). In this repo, you will find two branches: master and deployment.

Under the chaincode folder, you will find two chaincode components, ping and cc1. The chaincode components in the master branch both compile locally, while the cc1 in the deployment branch does not compile locally. However, if you deploy cc1 chaincode in the deployment branch to a Fabric network, it will compile and work just fine in the Fabric network... and if you deploy the cc1 chaincode in the master branch to a Fabric network, it will fail since it won't compile. The reason for this is at https://github.com/vandangogna/my-forked-chaincode-bootstrap/blob/master/chaincode/cc1/cc1.go#L5 and https://github.com/vandangogna/my-forked-chaincode-bootstrap/blob/deployment/chaincode/cc1/cc1.go#L4.

As I mentioned above, I have a suggestion to address this issue, but before doing so, do we concur that there is an issue? You can clone the repo and see what happens when you try to compile and deploy the chaincode components in both branches.

rolivieri commented 6 years ago

Just documenting the latest that has been discussed:

1) The expectation is that people wanting to write Go chaincode should be very familiar with the Go ecosystem/language and, hence, they should know the points below.

2) It is not supported to clone this repo or forks of it anywhere other than under the src folder of the GOPATH folder.

3) Modularizing the chaincode code with internal sub-folders is not supported either; instead a flat folder structure (i.e. no sub-folders) must be used for each chaincode component.

References: https://github.com/golang/go/issues/20883