search

busan15 / fabric-binaries-pi

AArch64/ARM64 binaries of Hyperledger Fabric for use with the Raspberry Pi 2/3/4.
Apache License 2.0
13 stars 3 forks source link

no matching manifest for linux/arm64/v8 in the manifest list entries #1

Open Garrocho opened 4 years ago

Garrocho commented 4 years ago

Hello @busan15 , everything jewel?

First of all, thank you very much for sharing this project. I hope you can help me solve the following problems when executing the commands:

./network.sh up docker: no matching manifest for linux/arm64/v8 in the manifest list entries.

./network.sh deployCC Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image build failed: docker build failed: Failed to pull hyperledger/fabric-ccenv:2.1: no matching manifest for linux/arm64/v8 in the manifest list entries !!!!!!!!!!!!!!! Chaincode installation on peer0.org1 has failed !!!!!!!!!!!!!!!!

ERROR !!! Deploying chaincode failed

The following file stores all logs: https://pastebin.com/raw/grt2pxag

Garrocho commented 4 years ago

I managed to correct both errors as follows:

edited the network.sh file and changed it to the fabric-tools busan repository: DOCKER_IMAGE_VERSION=$(docker run --rm busan15/fabric-tools:$IMAGETAG peer version | sed -ne 's/ Version: //p' | head -1)

For fabric-ccenv, I added a variable specifying the busan15 image in each peer in the docker-compose-test-net.yaml file: CORE_CHAINCODE_BUILDER=busan15/fabric-ccenv:$IMAGE_TAG

Despite resolving the initial issues of the issue, I was still unable to run deployCC, as the following error appears (even after running deployCC ten times): Error: failed to endorse chaincode install: rpc error: code = Unavailable desc = transport is closing

busan15 commented 4 years ago

Hey! I'm glad that our work might be useful to you, and I'm sorry that you ran into these issues.

I tested the test-network sample from the zip file before publishing it to GitHub, and it worked without issues apart from having to run ./network.sh deployCC three times. I should have tested it on a clean OS install as well, maybe there were some cached Docker images left that allowed it to work despite the references to Hyperledger's images. Since Hyperledger does not provide ARM64 images, this causes Docker to emit an error about "no matching manifest" as you reported. I'll add your fixes to a new release in a couple of days and look for other references that I might have missed.

Regarding the error about "transport is closing", that always seemed to work after trying three times. I don't really have any experience with Fabric or Hyperledger's other projects, other than a small demo that was set up as part of a university project, so I don't know what causes this problem or why it sometimes works after a while. It might occur when one of the Docker containers cannot be reached over TCP, but the network.sh script should take care of creating a Docker network and assigning the containers to it. A post on Stack Overflow suggests removing all Docker images, although ./network.sh down should have the same effect without removing non-Fabric related images.

I don't have any spare Raspberry Pis on hand any longer, and it might take a couple of weeks until I can get a hold of one and try to reproduce the issue with a fresh OS install. If you find a solution, feel free to update the readme via a pull request, or update this issue and I'll add it to the instructions.

Garrocho commented 4 years ago

Hi @busan15 ,

today, again I made a few more attempts at deployCC. And after several attempts, the following error appeared. Note that this time the image ID (busan15/fabric-ccenv or hyperledger/fabric-ccenv) did not appear:

Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build
chaincode: docker build failed: docker image build failed: no matching manifest for linux/arm64/v8 in the manifest list entries
!!!!!!!!!!!!!!! Chaincode installation on peer0.org1 has failed !!!!!!!!!!!!!!!!

ERROR !!! Deploying chaincode failed
busan15 commented 4 years ago

I looked through the test-network related files and found that the file config/core.yaml contains a chaincode building section where Docker images are pulled from $(DOCKER_NS)/fabric-\<image name>. Maybe DOCKER_NS is set to "hyperledger" somewhere and that causes the "no matching manifest for linux/arm64/v8" from your latest error message?

I would assume that DOCKER_NS is meant to simplify this procedure of rehosting the Docker images under a different repository, but it does not seem to be fully implemented as many references are still hard-coded to use images from Hyperledger.

I have added your suggested changes to a new release, and also replaced "$(DOCKER_NS)" with "busan15" in core.yaml. Could you try performing the same change (replace all four occurrences of $(DOCKER_NS) with busan15 in config/core.yaml) and see if that helps?

Garrocho commented 4 years ago

Hello @busan15 ,

only today I was able to carry out the tests. After running "network.sh deployCC" 6 times, the error message "Error: failed to endorse chaincode install: rpc error: code = Unavailable desc = transport is closing" disappeared. However, the error "Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image build failed: no matching manifest for linux / arm64 / v8 in the manifest list entries " persists.

I imagine the problem may be in this "busan15 / fabric-ccenv" image, do you agree?

Wouldn't it be interesting to try to recompile that image? If you can explain it to me, I can try to compile it here.

busan15 commented 4 years ago

I agree, it does seem like the Docker images need to be rebuilt if the same error still persists. Unfortunately, I don't remember exactly which changes had to be made to the source code, as it has been a few months since those images were built.

Using the source code available at https://github.com/hyperledger/fabric, I know that in images/<image name>/Dockerfile, "binutils-gold" had to be added to the list of packages installed with the command "RUN apk add [...]" for most of the images, as the ld command used in the compilation process would not work otherwise.

There was also a reference to an openjdk Docker image somewhere where the version tag had to be replaced with something else from this list.

That's all I can remember off the top of my head. I would like to create a fork of the Fabric repo that compiles on ARM, and I might look into that in a couple of weeks when I can try to dig up the modified source code again. Until then, I don't really know where to start changing things to make it compile.

busan15 commented 3 years ago

Hi again, and apologies for the delayed response. I have spent some time troubleshooting this issue and think I have found a workaround for it.

Starting the Docker daemon with debug logging enabled showed that the image "hyperledger/fabric-baseos:2.1" was being pulled by the ./network.sh deployCC command, which generated the "no matching manifest" error. I cannot find any reference to this name in the files from fabric-samples, but tagging the ARM image with this name seems to work as a workaround.

This can be accomplished by first retrieving it with docker pull busan15/fabric-baseos:2.1, and then tagging it with docker tag busan15/fabric-baseos:2.1 hyperledger/fabric-baseos:2.1. This ensures that "hyperledger/fabric-baseos:2.1" will resolve to the retrieved ARM image instead of the one from Hyperledger.

After resetting the test network with ./network.sh down, recreating it with ./network.sh up createChannel and running ./network.sh deployCC should work without issues.

Using a fresh install of Ubuntu 20.04.1 on a Raspberry Pi 3, I can successfully get the chaincode to deploy by:

  1. installing Docker (and golang + docker-compose, I forgot to include those in the readme)
  2. Adding the "ubuntu" user to the Docker group with sudo usermod -aG docker ubuntu followed by logging out and back in again
  3. Downloading and unpacking fabric-samples.tar.gz
  4. Running docker pull busan15/fabric-baseos:2.1
  5. Running docker tag busan15/fabric-baseos:2.1 hyperledger/fabric-baseos:2.1
  6. Starting the Fabric network with ./network.sh up createChannel
  7. Deploying the chaincode with ./network.sh deployCC repeated three times

I will update the readme with these additional steps as well. I hope this helps you in getting Fabric to run on your system, and apologies again for the delay!

jjf385385 commented 3 years ago

I agree, it does seem like the Docker images need to be rebuilt if the same error still persists. Unfortunately, I don't remember exactly which changes had to be made to the source code, as it has been a few months since those images were built.

Using the source code available at https://github.com/hyperledger/fabric, I know that in images/<image name>/Dockerfile, "binutils-gold" had to be added to the list of packages installed with the command "RUN apk add [...]" for most of the images, as the ld command used in the compilation process would not work otherwise.

There was also a reference to an openjdk Docker image somewhere where the version tag had to be replaced with something else from this list.

That's all I can remember off the top of my head. I would like to create a fork of the Fabric repo that compiles on ARM, and I might look into that in a couple of weeks when I can try to dig up the modified source code again. Until then, I don't really know where to start changing things to make it compile.

Hello @busan15 , i am a student in china and i am working on how to build fabric on raspberry. I followed your advice when compiling fabric core images, but the images fabric-tools and fabric-peer seemed to be wrong. Could you remember that what changes should be made in their dockerfile except the commond"RUN apk add binutils-gold"? I hope you can help me, thank you. When I try to run byfn.sh, some errors appear as follow:

/ | | | / \ | \ | | _ \ | | / \ | |) | | | ) | | | / \ | < | | |____/ || // _\ || _\ |_|

Build your first network (BYFN) end-to-end test

Channel name : mychannel Creating channel...

busan15 commented 3 years ago

Hello,

I found the old source with the modifications in place, and I have uploaded patch files containing these changes for fabric, fabric-baseimage, and fabric-ca to this repository. They should show the modifications that were used to make these images compile properly, and I think some of these changes were made to fix the segmentation fault errors that you are receiving. The patches likely have to be applied to the exact commit they were made for to avoid conflicts though, or you could probably open them in a text editor and work out how to apply them by hand.

The revisions that were used as a base are:

To apply these changes and also update to a later version, you could probably (for each of these three) clone the repository, checkout the commit specified above, apply the corresponding patch with git apply <patch file>, commit or stash the changes, and then update with git pull. This last step might cause conflicts, so if you are happy with using an older version of Fabric you could also stop after applying the patches and compile from there.

I hope this helps!

jjf385385 commented 3 years ago

Hello,

I found the old source with the modifications in place, and I have uploaded patch files containing these changes for fabric, fabric-baseimage, and fabric-ca to this repository. They should show the modifications that were used to make these images compile properly, and I think some of these changes were made to fix the segmentation fault errors that you are receiving. The patches likely have to be applied to the exact commit they were made for to avoid conflicts though, or you could probably open them in a text editor and work out how to apply them by hand.

The revisions that were used as a base are:

To apply these changes and also update to a later version, you could probably (for each of these three) clone the repository, checkout the commit specified above, apply the corresponding patch with git apply <patch file>, commit or stash the changes, and then update with git pull. This last step might cause conflicts, so if you are happy with using an older version of Fabric you could also stop after applying the patches and compile from there.

I hope this helps!

Thank you for the reply. However, it seemed that I didn’t perform properly as you thought. The same error comes again. Maybe the reason is that I am not familiar with the command git. I just “git clone” these repositories and then add the command “run apk add binutils-gold”. Should I execute some other commands to run byfn.sh correctly? I feel very sorry to disturb you about this question. Actually, I am studying for a master degree, and my supervisor hope that I can build fabric on raspberry with the OS rasbian (64bits beta). What’s more, we want to realize the freedom of fabric version. Therefore, I need to build these images manually. I hope you can be my hero!!!

busan15 commented 3 years ago

If you don't want to use Git, you could probably apply the patch files manually by just opening them in a text editor and implementing the changes by hand. There's no guarantee that it will work with the latest version of Fabric, but if you want to try you can start by downloading the latest version of fabric, fabric-baseimage and fabric-ca. For each of these, open the corresponding patch file from this repository (fabric, fabric-baseimage, fabric-ca). Each change starts with a section specifying which file it relates to (written as +++ b/images/ccenv/Dockerfile for example, which refers to the file images/ccenv/Dockerfile) and has some surrounding context that can be used to locate where in the file it should be applied.

For example, the following block would mean that in images/peer/Dockerfile, below the two lines starting with "ARG", you would remove the lines starting with "FROM alpine" and "RUN apk" and put "FROM ubuntu as peer-base" in their place. Below these lines, there is also another replacement where "FROM golang [...]" and "RUN apk [...]" are replaced with another "FROM golang [...]" and "RUN apt [...]". By repeating this for all changes in each of the three patch files, you would have recreated the same set of changes that we used to make the older version of Fabric compile.

diff --git a/images/peer/Dockerfile b/images/peer/Dockerfile
index 19ce91400..5e79a04d9 100644
--- a/images/peer/Dockerfile
+++ b/images/peer/Dockerfile
@@ -5,16 +5,15 @@
 ARG GO_VER
 ARG ALPINE_VER

-FROM alpine:${ALPINE_VER} as peer-base
-RUN apk add --no-cache tzdata
+FROM ubuntu as peer-base

-FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang
-RUN apk add --no-cache \
+FROM golang:${GO_VER}-buster as golang
+RUN apt install -y \

I don't have easy access to a Raspberry Pi for testing, so I can't test if this works on newer versions of Fabric. But hopefully it should at least bring you a bit closer to getting the latest version of Fabric working. Also, the Build Your First Network (BYFN) guide has been removed for newer Fabric versions, so that sample code might not work at all any more (see https://hyperledger-fabric.readthedocs.io/en/latest/build_network.html). The test-network sample code seem to have replaced it, it can be found here: https://hyperledger-fabric.readthedocs.io/en/latest/test_network.html

jjf385385 commented 3 years ago

If you don't want to use Git, you could probably apply the patch files manually by just opening them in a text editor and implementing the changes by hand. There's no guarantee that it will work with the latest version of Fabric, but if you want to try you can start by downloading the latest version of fabric, fabric-baseimage and fabric-ca. For each of these, open the corresponding patch file from this repository (fabric, fabric-baseimage, fabric-ca). Each change starts with a section specifying which file it relates to (written as +++ b/images/ccenv/Dockerfile for example, which refers to the file images/ccenv/Dockerfile) and has some surrounding context that can be used to locate where in the file it should be applied.

For example, the following block would mean that in images/peer/Dockerfile, below the two lines starting with "ARG", you would remove the lines starting with "FROM alpine" and "RUN apk" and put "FROM ubuntu as peer-base" in their place. Below these lines, there is also another replacement where "FROM golang [...]" and "RUN apk [...]" are replaced with another "FROM golang [...]" and "RUN apt [...]". By repeating this for all changes in each of the three patch files, you would have recreated the same set of changes that we used to make the older version of Fabric compile.

diff --git a/images/peer/Dockerfile b/images/peer/Dockerfile
index 19ce91400..5e79a04d9 100644
--- a/images/peer/Dockerfile
+++ b/images/peer/Dockerfile
@@ -5,16 +5,15 @@
 ARG GO_VER
 ARG ALPINE_VER

-FROM alpine:${ALPINE_VER} as peer-base
-RUN apk add --no-cache tzdata
+FROM ubuntu as peer-base

-FROM golang:${GO_VER}-alpine${ALPINE_VER} as golang
-RUN apk add --no-cache \
+FROM golang:${GO_VER}-buster as golang
+RUN apt install -y \

I don't have easy access to a Raspberry Pi for testing, so I can't test if this works on newer versions of Fabric. But hopefully it should at least bring you a bit closer to getting the latest version of Fabric working. Also, the Build Your First Network (BYFN) guide has been removed for newer Fabric versions, so that sample code might not work at all any more (see https://hyperledger-fabric.readthedocs.io/en/latest/build_network.html). The test-network sample code seem to have replaced it, it can be found here: https://hyperledger-fabric.readthedocs.io/en/latest/test_network.html

THANK YOU!!!!!!!!!!! It is all my fault that I did not notice your patch file in your repository. I made changes as you show in the patch files, and it really worked. I can now compile these fabric images manully, and these images can finally work correctly. All these success is your credit. You are really the hero in my recent life. Hope you have a nice day in all the next days!!! THANK YOU !!!!!!!!!

busan15 commented 3 years ago

Great! I'm glad you got it working! :D

Have a nice weekend you too, and good luck with your master's degree!

ghost commented 3 years ago

Hi again, and apologies for the delayed response. I have spent some time troubleshooting this issue and think I have found a workaround for it.

Starting the Docker daemon with debug logging enabled showed that the image "hyperledger/fabric-baseos:2.1" was being pulled by the ./network.sh deployCC command, which generated the "no matching manifest" error. I cannot find any reference to this name in the files from fabric-samples, but tagging the ARM image with this name seems to work as a workaround.

This can be accomplished by first retrieving it with docker pull busan15/fabric-baseos:2.1, and then tagging it with docker tag busan15/fabric-baseos:2.1 hyperledger/fabric-baseos:2.1. This ensures that "hyperledger/fabric-baseos:2.1" will resolve to the retrieved ARM image instead of the one from Hyperledger.

After resetting the test network with ./network.sh down, recreating it with ./network.sh up createChannel and running ./network.sh deployCC should work without issues.

Using a fresh install of Ubuntu 20.04.1 on a Raspberry Pi 3, I can successfully get the chaincode to deploy by:

1. installing Docker (and golang + docker-compose, I forgot to include those in the readme)

2. Adding the "ubuntu" user to the Docker group with `sudo usermod -aG docker ubuntu` followed by logging out and back in again

3. Downloading and unpacking fabric-samples.tar.gz

4. Running `docker pull busan15/fabric-baseos:2.1`

5. Running `docker tag busan15/fabric-baseos:2.1 hyperledger/fabric-baseos:2.1`

6. Starting the Fabric network with `./network.sh up createChannel`

7. Deploying the chaincode with `./network.sh deployCC` repeated three times

I will update the readme with these additional steps as well. I hope this helps you in getting Fabric to run on your system, and apologies again for the delay!

Hi Busan thank you so much for the amazing job! I was able to deploy the channel with no issues after seeing this reply from Aug 20. Just wondering... were you ever able to fix the need to call it three times? :)

Thanks! Raul Matos

busan15 commented 3 years ago

@rbrumdematos Hi!

That's great, I'm glad everything worked! I used Fabric as part of a short project at university, so I haven't really done anything with it since creating this repository. I don't know what causes the deployment command to fail twice (maybe it is a timing issue and the time it takes to run the deployCC command twice is just enough for createChannel to finish?) and I probably won't spend any time looking for the cause when I'm not going to use Fabric again for the foreseeable future. If you happen to find a better way than running the command three times, submit a pull request or notify me and I'll gladly add it to the readme. Otherwise, I'm happy that everything at least works and that these instructions were useful to you!