hieu29791
commented
5 years ago Hi, I have the same problem here, any update?
Open jazzyarchitects opened 5 years ago
hieu29791
commented
5 years ago Hi, I have the same problem here, any update?
jazzyarchitects
commented
5 years ago @hieu29791 I did some workaround for this. I had to create a kubernetes job which is run after the provisioner is created. This kubernetes job basically runs the below shell script:
mkdir -p /nfs/${id}/rca-scripts
mkdir -p /nfs/${id}/ica
mkdir -p /nfs/${id}/rca
mkdir -p /nfs/${id}/rca-data
mkdir -p /nfs/${id}/orderer
chmod a+rwx /nfs/${id}/rca-scripts
chmod a+rwx /nfs/${id}/ica
chmod a+rwx /nfs/${id}/rca
chmod a+rwx /nfs/${id}/rca-data
chmod a+rwx /nfs/${id}/orderer
rm -rf /nfs/${id}/rca-scripts/*
rm -rf /nfs/${id}/ica/*
rm -rf /nfs/${id}/rca/*
rm -rf /nfs/${id}/rca-data/*
rm -rf /nfs/${id}/orderer/*where ${id} is the org name.
Job:
apiVersion: 'batch/v1',
kind: 'Job',
metadata: {
name: 'nfs-partitions',
namespace: `${domain}`,
},
spec: {
template: {
spec: {
containers: [
{
name: 'partitioner',
image: 'alpine',
command: ['sh', '/conf/init.sh'],
volumeMounts: [
{
name: 'init',
mountPath: '/conf',
},
{
name: 'nfs',
mountPath: '/nfs',
},
],
},
],
volumes: [
{
name: 'init',
configMap: {
name: 'partition-create-script',
},
},
{
name: 'nfs',
persistentVolumeClaim: {
claimName: `raw-nfs-mount-${id}`,
},
},
],
restartPolicy: 'Never',
},
},
backoffLimit: 1,
}PVCs:
kind: 'PersistentVolumeClaim',
apiVersion: 'v1',
metadata: {
name: `raw-nfs-mount-${id}`,
namespace: `${domain}`,
},
spec: {
accessModes: ['ReadWriteMany'],
resources: {
requests: {
storage: '1Gi',
},
},
storageClassName,
volumeName: `raw-nfs-mount-${id}`,
},PV:
kind: 'PersistentVolume',
apiVersion: 'v1',
metadata: {
name: `raw-nfs-mount-${id}`,
namespace: `${domain}`,
},
spec: {
capacity: {
storage: '1Gi',
},
storageClassName,
accessModes: ['ReadWriteMany'],
persistentVolumeReclaimPolicy: 'Retain',
mountOptions: ['nfsvers=4.1', 'rsize=1048576', 'wsize=1048576', 'hard', 'timeo=600', 'retrans=2'],
nfs: {
server: efsServer,
path: '/',
},
},Configmap:
apiVersion: 'v1',
kind: 'ConfigMap',
metadata: {
name: 'partition-create-script',
labels: {
name: 'partition-create-script',
},
namespace: `${domain}`,
},
data: {
'init.sh': `
mkdir -p /nfs/${id}/rca-scripts
mkdir -p /nfs/${id}/ica
mkdir -p /nfs/${id}/rca
mkdir -p /nfs/${id}/rca-data
mkdir -p /nfs/${id}/orderer
chmod a+rwx /nfs/${id}/rca-scripts
chmod a+rwx /nfs/${id}/ica
chmod a+rwx /nfs/${id}/rca
chmod a+rwx /nfs/${id}/rca-data
chmod a+rwx /nfs/${id}/orderer
rm -rf /nfs/${id}/rca-scripts/*
rm -rf /nfs/${id}/ica/*
rm -rf /nfs/${id}/rca/*
rm -rf /nfs/${id}/rca-data/*
rm -rf /nfs/${id}/orderer/*
`,
},Hope this helps
hieu29791
commented
5 years ago @jazzyarchitects Thanks for your help. Could you explain more to me about this step? When I run start-fabric.sh I want automatically create folders on EFS. How to I migration your script and this example, I just run start-fabric.sh and this automatically do the rest.
MCLDG-zz
commented
5 years ago When you run start-fabric.sh, it calls a function makeDirs. This creates the directories on efs. Can you check the output of start-fabric.sh and confirm any errors that are seen here? You can check the dirs on efs by doing ls /opt/share on the bastion, since efs is mounted on the bastion also .
On Fri, 22 Feb 2019, 17:04 hieu29791, notifications@github.com wrote:
@jazzyarchitects https://github.com/jazzyarchitects Thanks for your help. Could you explain more to me about this step? When I run start-fabric.sh I want automatically create folders on EFS. How to I migration your script and this example, I just run start-fabric.sh and this automatically do the rest.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/aws-samples/hyperledger-on-kubernetes/issues/2#issuecomment-466325607, or mute the thread https://github.com/notifications/unsubscribe-auth/AL0HGpGQ94-NlWX8yjomu5Aq0hkX-zhZks5vP7KIgaJpZM4aKJLl .
hieu29791
commented
5 years ago When you run start-fabric.sh, it calls a function makeDirs. This creates the directories on efs. Can you check the output of start-fabric.sh and confirm any errors that are seen here? You can check the dirs on efs by doing ls /opt/share on the bastion, since efs is mounted on the bastion also . … On Fri, 22 Feb 2019, 17:04 hieu29791, @.***> wrote: @jazzyarchitects https://github.com/jazzyarchitects Thanks for your help. Could you explain more to me about this step? When I run start-fabric.sh I want automatically create folders on EFS. How to I migration your script and this example, I just run start-fabric.sh and this automatically do the rest. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AL0HGpGQ94-NlWX8yjomu5Aq0hkX-zhZks5vP7KIgaJpZM4aKJLl .
Hi, When I run start-fabric.sh, on the step Starting RCA in K8s, error display like this:
kubectl describe pod -n org0
Name: rca-org0-7c4977ccc5-4lswg
Namespace: org0
Priority: 0
PriorityClassName: <none>
Node: ip-192-168-229-54.ap-southeast-1.compute.internal/192.168.229.54
Start Time: Mon, 25 Feb 2019 04:34:42 +0000
Labels: app=hyperledger
name=rca-org0
org=org0
pod-template-hash=3705337771
role=ca
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/rca-org0-7c4977ccc5
Containers:
rca-org0:
Container ID:
Image: hyperledger/fabric-ca:x86_64-1.1.0
Image ID:
Port: 7054/TCP
Host Port: 0/TCP
Command:
sh
Args:
-c
/scripts/start-root-ca.sh 2>&1
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
FABRIC_CA_SERVER_HOME: /etc/hyperledger/fabric-ca
FABRIC_CA_SERVER_TLS_ENABLED: true
FABRIC_CA_SERVER_CSR_CN: rca-org0.org0
FABRIC_CA_SERVER_CSR_HOSTS: rca-org0.org0
FABRIC_CA_SERVER_DEBUG: true
BOOTSTRAP_USER_PASS: rca-org0-admin:rca-org0-adminpw
TARGET_CERTFILE: /data/org0-ca-cert.pem
FABRIC_ORGS: org0.org0 org1.org1 org2.org2
Mounts:
/data from rca-data (rw)
/etc/hyperledger/fabric-ca from rca (rw)
/scripts from rca-scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-v69sv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
rca:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rca-org0-pvc
ReadOnly: false
rca-scripts:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rca-scripts-org0-pvc
ReadOnly: false
rca-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: rca-data-org0-pvc
ReadOnly: false
default-token-v69sv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-v69sv
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 19m (x29 over 1h) kubelet, ip-192-168-229-54.ap-southeast-1.compute.internal Unable to mount volumes for pod "rca-org0-7c4977ccc5-4lswg_org0(abc00cc9-38b6-11e9-aef9-0a59596ba446)": timeout expired waiting for volumes to attach or mount for pod "org0"/"rca-org0-7c4977ccc5-4lswg". list of unmounted volumes=[rca rca-scripts rca-data]. list of unattached volumes=[rca rca-scripts rca-data default-token-v69sv]
Warning FailedMount 1m (x211 over 2h) kubelet, ip-192-168-229-54.ap-southeast-1.compute.internal (combined from similar events): MountVolume.SetUp failed for volume "rca-scripts-org0-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/abc00cc9-38b6-11e9-aef9-0a59596ba446/volumes/kubernetes.io~nfs/rca-scripts-org0-pv --scope -- mount -t nfs -o hard,timeo=600,retrans=2,nfsvers=4.1,rsize=1048576,wsize=1048576 fs-87913ec6.efs.ap-southeast-1.amazonaws.com:/rca-scripts /var/lib/kubelet/pods/abc00cc9-38b6-11e9-aef9-0a59596ba446/volumes/kubernetes.io~nfs/rca-scripts-org0-pv
Output: Running scope as unit run-26504.scope.
mount.nfs: mounting fs-87913ec6.efs.ap-southeast-1.amazonaws.com:/rca-scripts failed, reason given by server: No such file or directoryOn Bastion, I see folder already create on /op/share/ I don't know what wrong is it. Could you explain to me.
tree /opt/share
/opt/share
├── ica-org0
├── ica-org1
├── ica-org2
├── orderer
├── rca-data
├── rca-org0
├── rca-org1
├── rca-org2
└── rca-scripts
├── addorg-fabric-join.sh
├── addorg-fabric-setup.sh
├── create-channel.sh
├── env3orgs.sh
├── env.sh
├── fabric-delete-org.sh
├── fabric-installcc.sh
├── fabric-signconf.sh
├── fabric-updateconf.sh
├── fabric-upgradecc.sh
├── gen-channel-artifacts.sh
├── gen-channel-artifacts-template.sh
├── install-marbles-cc.sh
├── load-fabric-marbles.sh
├── load-fabric.sh
├── peer-join-channel.sh
├── register-orderer.sh
├── register-org.sh
├── register-peer.sh
├── start-intermediate-ca.sh
├── start-orderer.sh
├── start-peer.sh
├── start-root-ca.sh
├── test-fabric-abac.sh
├── test-fabric-marbles.sh
├── test-fabric-marbles-workshop.sh
└── test-marbles-cc.sh
msolefonte
commented
5 years ago Excuse me by starting this topic again. Can't be this problem related to execute the code in a local machine with kubectl and not in the ec2 instance? Executed in local, makeDir is going to create the folders in the local path, default /opt/share, which is going to allow the script continue at that point, but will show the error when data is required and not set in the EFS.
I would recommend to create the files not in local but in a job as the default way, as shown by @jazzyarchitects. This way fabric could be started from everywhere. One problem of this would be that start-fabric.shstill requires to access data from local path (assuming that it is the EFS volume), so one patch would be to create that on a folder of /tmp and deleting in after the installation. It can be completely redone, of course, but it is harder and requires more time.
I am trying to create a Hyperledger network and I get the following error for the pods when RCA is being deployed.
I can see that it is because the mount directories don't exist in the NFS server but isn't the script supposed to take care of that instead of manually mounting and creating directories?