hpe-storage / python-hpedockerplugin

HPE Native Docker Plugin
Apache License 2.0
36 stars 64 forks source link

Filesharing creation without existing cpg on array, volume status will be always "CREATING" and logs are getting into loop, not going to failed state #678

Closed c-raghav closed 5 years ago

c-raghav commented 5 years ago

Filesharing creation without existing cpg on array, volume status will be always "CREATING" and logs are getting into loop, not going to failed state

  1. create share without CPG name exists on array or on hpe.conf
  2. on plain docker create a volume with default CPG

16:02:19 dockervol.go:217: returning &dockervol.GetResponse{Volume:dockervol.DockerVolume{Name:"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040", Mountpoint:"", Status:map[string]interface {}{"fpg":interface {}(nil), "vfs":interface {}(nil), "name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040", "size":"50 GiB", "clientIPs":[]interface {}{}, "fsMode":interface {}(nil), "fsOwner":interface {}(nil), "cpg":"FC_r5", "sharePath":interface {}(nil), "protocol":"nfs", "status":"CREATING", "backend":"DEFAULT_FILE"}}, Err:""} 16:02:19 provisioner.go:333: Docker volume with name sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 found. Delete using hpe.com/hpe. 16:02:19 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040","Opts":{"manager":"k8s"}} 16:02:19 client.go:168: response: 200 OK, length=-1 16:02:19 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AVAILABLE or FAILED state and then attempt remove. 16:02:19 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040"} 16:02:19 client.go:168: response: 200 OK, length=-1 16:02:19 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AVAILABLE or FAILED state and then attempt remove. 16:02:23 provisioner.go:303: statusLogger: provision chains=0, delete chains=1, parked chains=0, ids tracked=0, connection=valid 16:02:24 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040","Opts":{"manager":"k8s"}} 16:02:24 client.go:168: response: 200 OK, length=-1 16:02:24 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AVAILABLE or FAILED state and then attempt remove. 16:02:24 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040"} 16:02:24 client.go:168: response: 200 OK, length=-1 16:02:24 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AVAILABLE or FAILED state and then attempt remove. 16:02:28 provisioner.go:303: statusLogger: provision chains=0, delete chains=1, parked chains=0, ids tracked=0, connection=valid 16:02:29 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040","Opts":{"manager":"k8s"}} 16:02:29 client.go:168: response: 200 OK, length=-1 16:02:29 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AVAILABLE or FAILED state and then attempt remove. 16:02:29 client.go:126: request: action=POST path=http://unix/VolumeDriver.Remove payload={"Name":"sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040"} 16:02:29 client.go:168: response: 200 OK, length=-1 16:02:29 dockervol.go:369: /VolumeDriver.Remove failed sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 - Share sc-file4-2a8a076d-a168-11e9-a137-f40343a6d040 is in CREATING state. Please wait for it to be in AV

2019-07-08 10:18:49,337 [ERROR] hpedockerplugin.cmd.cmd_createfpg [140283020131048] Thread-5 Create new FPG DockerFpg_4766 failed. Msg: Share backend exception: Failed to create FPG DockerFpg_4766 of size 16 using CPG FC_r5: Exception: Not found (HTTP 404) 15 - CPG does not exist 2019-07-08 10:18:49,338 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Try unlocking name DEFAULT_FILE/FC_r5/DockerFpg_4766 2019-07-08 10:18:49,338 [DEBUG] etcd.client [140283020131048] Thread-5 Deleting /fp-fpg-lock/DEFAULT_FILE/FC_r5/DockerFpg_4766 recursive=None dir=None extra args={} 2019-07-08 10:18:49,338 [DEBUG] etcd.client [140283020131048] Thread-5 Calculated params = {} 2019-07-08 10:18:49,340 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Name is unlocked : DEFAULT_FILE/FC_r5/DockerFpg_4766 2019-07-08 10:18:49,341 [ERROR] hpedockerplugin.file_manager [140283020131048] Thread-5 Unknown exception caught while creating default FPG: FPG creation failed: Create new FPG DockerFpg_4766 failed. Msg: Share backend exception: Failed to create FPG DockerFpg_4766 of size 16 using CPG FC_r5: Exception: Not found (HTTP 404) 15 - CPG does not exist 2019-07-08 10:18:49,342 [INFO] hpedockerplugin.file_manager [140283020131048] Thread-5 _generate_default_fpg_vfs_names: Generating default FPG VFS names 2019-07-08 10:18:49,342 [INFO] hpedockerplugin.cmd.cmd_generate_fpg_vfs_names [140283020131048] Thread-5 Cmd: Generating default FPG and VFS names... 2019-07-08 10:18:49,342 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Try locking name DEFAULT_FILE 2019-07-08 10:18:49,342 [DEBUG] etcd.client [140283020131048] Thread-5 Writing DEFAULT_FILE to key /fp-backend-lock/DEFAULT_FILE ttl=None dir=False append=False 2019-07-08 10:18:49,344 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Name is locked : DEFAULT_FILE 2019-07-08 10:18:49,345 [DEBUG] etcd.client [140283020131048] Thread-5 Issuing read for key /file-persona/DEFAULT_FILE.metadata with args {} 2019-07-08 10:18:49,346 [DEBUG] etcd.client [140283020131048] Thread-5 Writing {"ips_in_use": ["15.213.65.65", "15.213.65.68", "15.213.65.67"], "ips_locked_for_use": [], "counter": 4767, "default_fpgs": {"FC_r6": ["DockerFpg_2"]}} to key /file-persona/DEFAULT_FILE.metadata ttl=None dir=False append=False 2019-07-08 10:18:49,348 [INFO] hpedockerplugin.etcdutil [140283020131048] Thread-5 Write key: /file-persona/DEFAULT_FILE.metadata to ETCD, value is: {"ips_in_use": ["15.213.65.65", "15.213.65.68", "15.213.65.67"], "ips_locked_for_use": [], "counter": 4767, "default_fpgs": {"FC_r6": ["DockerFpg_2"]}} 2019-07-08 10:18:49,348 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Try unlocking name DEFAULT_FILE 2019-07-08 10:18:49,349 [DEBUG] etcd.client [140283020131048] Thread-5 Deleting /fp-backend-lock/DEFAULT_FILE recursive=None dir=None extra args={} 2019-07-08 10:18:49,349 [DEBUG] etcd.client [140283020131048] Thread-5 Calculated params = {} 2019-07-08 10:18:49,351 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Name is unlocked : DEFAULT_FILE 2019-07-08 10:18:49,351 [INFO] hpedockerplugin.file_manager [140283020131048] Thread-5 _generate_default_fpg_vfs_names: Generated: DockerFpg_4767, DockerVfs_4767 2019-07-08 10:18:49,351 [INFO] hpedockerplugin.file_manager [140283020131048] Thread-5 Names generated: FPG=DockerFpg_4767, VFS=DockerVfs_4767 2019-07-08 10:18:49,351 [INFO] hpedockerplugin.file_manager [140283020131048] Thread-5 Creating FPG DockerFpg_4767 using CPG FC_r5 2019-07-08 10:18:49,352 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Try locking name DEFAULT_FILE/FC_r5/DockerFpg_4767 2019-07-08 10:18:49,352 [DEBUG] etcd.client [140283020131048] Thread-5 Writing DEFAULT_FILE/FC_r5/DockerFpg_4767 to key /fp-fpg-lock/DEFAULT_FILE/FC_r5/DockerFpg_4767 ttl=None dir=False append=False 2019-07-08 10:18:49,354 [DEBUG] hpedockerplugin.etcdutil [140283020131048] Thread-5 Name is locked : DEFAULT_FILE/FC_r5/DockerFpg_4767 2019-07-08 10:18:49,354 [INFO] hpedockerplugin.cmd.cmd_createfpg [140283020131048] Thread-5 Creating FPG DockerFpg_4767 on the backend using CPG FC_r5 2019-07-08 10:18:49,354 [INFO] hpedockerplugin.cmd.cmd_createfpg [140283020131048] Thread-5 Default FPG size overridden to 16 2019-07-08 10:18:49,355 [DEBUG] hpe3parclient.http [140283020131048] Thread-5 REQ: curl -i https://15.213.64.15:8080/api/v1/credentials -X POST -H "User-Agent: python-3parclient" -H "Accept: application/json" -H "Content-Type: application/json"

amitk1977 commented 5 years ago

To be verified by Shashi on the final RC build of 7/10

c-raghav commented 5 years ago

''' Adding SC, PVC and POD ymls '''

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: sc-file4b provisioner: hpe.com/hpe parameters: name: TEST_SHARE size: "100" filePersona: "" cpg: FC_r10


kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcfile4b-rc spec: accessModes:

'''

Adding PV, PVC and POD '''

apiVersion: v1 kind: PersistentVolume metadata: name: pvfile-rc5 spec: capacity: storage: 25Gi accessModes:


apiVersion: v1 kind: Pod metadata: name: podfile-rc5 spec: containers:

'''

c-raghav commented 5 years ago

Verified defect , 3PARDCVLOG_678_RC_Build.txt D-678-output-verifiede.txt

'''

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: sc-file-678 provisioner: hpe.com/hpe parameters: name: TEST_SHARE size: "100" filePersona: "" cpg: FC_r10


kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcfile-rc-678 spec: accessModes:


kind: Pod apiVersion: v1 metadata: name: pod-file-rc-678 spec: containers:

attaching logs

c-raghav commented 5 years ago

Verified and closed logs , yml attached