search

hpe-storage / python-hpedockerplugin

HPE Native Docker Plugin
Apache License 2.0
36 stars 64 forks source link

File persona: share creation failed with error "too many file store, can't create file store" #656

Closed virendra-sharma closed 5 years ago

virendra-sharma commented 5 years ago

Observation: - 1) File share creation failed when trying to create bulk shares in same fpg. 2) On 3PAR only 16 fstore present but still share creation failed with fstore error.

Steps: - 1) Created share with default fpg of size 64TB. 2) Tried to create multiple share on default fpg, getting error after some share creation.

Output: - 

[root@cssosbe03-b12 ~]# docker volume inspect share35
[
    {
        "Driver": "hpe:latest",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/plugins/ef17338dae73d5d09601fbb9d1fb7e0a45b491406922eb57264686b40143f425/rootfs/opt/hpe/data/hpedocker-share35",
        "Name": "share35",
        "Options": {
            "filePersona": ""
        },
        "Scope": "global",
        "Status": {
            "backend": "DEFAULT",
            "clientIPs": [],
            "cpg": "swap_fs_cpg",
            "detailedStatus": "Share creation failed: Unknown exception occurred while creating share on new FPG. Reason: Share creation failed [share_name: share35, error: Share backend exception: Failed to create share share35: Bad request (HTTP 400) 29 - Too many file stores; can't create file store DockerFpg_3/DockerVfs_3/share35",
            "fpg": "DockerFpg_3",
            "fsMode": null,
            "fsOwner": null,
            "name": "share35",
            "protocol": "nfs",
            "sharePath": null,
            "size": "1024 GiB",
            "status": "FAILED",
            "vfs": "DockerVfs_3",
            "vfsIPs": [
                [
                    "192.168.98.49",
                    "255.255.192.0"
                ]
            ]
        }
    }
]

fstore error.txt

prablr79 commented 5 years ago

@virendra-sharma can you try directly in 3par and observe the results ? and revert ?

prablr79 commented 5 years ago

@virendra-sharma can you please confirm 3par CLI is also behaving same or not ?

virendra-sharma commented 5 years ago

@prablr79 after getting this error, i tried to create manually fstore on existing fpg and found same error. But if we allowed to create total 256 fstore on one node pair array so why are we restricting in some small number. wanted to clear this doubt.

virendra-sharma commented 5 years ago

Tried again with 30 share creation, all created without any failure. Since unable to re-produce this bug. Closing it with below share creation output.

CSIM_EOS14_1674003 cli% showfpg
                         ------(GiB)-------
FPG         -Mountpath-- --Size-- Available ActiveStates -DefaultCpg- -----VVs----- State  Version
DockerFpg_0 /DockerFpg_0 16384.00  16375.54 ACTIVATED    SHASHI_CPG   DockerFpg_0.1 normal 12.3
DockerFpg_1 /DockerFpg_1 16384.00  16375.53 ACTIVATED    SHASHI_CPG   DockerFpg_1.1 normal 12.3
DockerFpg_3 /DockerFpg_3 16384.00  16375.42 ACTIVATED    SHASHI_CPG   DockerFpg_3.1 normal 12.3
DockerFpg_4 /DockerFpg_4 16384.00  16375.53 ACTIVATED    sandesh      DockerFpg_4.1 normal 12.3
DockerFpg_6 /DockerFpg_6 16384.00  16375.53 ACTIVATED    sandesh      DockerFpg_6.1 normal 12.3
DockerFpg_7 /DockerFpg_7 16384.00  16375.53 ACTIVATED    sandesh      DockerFpg_7.1 normal 12.3
--------------------------------------------------------------------------------------------------
          6 total        98304.00  98253.08

CSIM_EOS14_1674003 cli% showfshare nfs -fpg DockerFpg_6
ShareName VFS         FileStore ------------SharePath------------ State  -Clients-
share1    DockerVfs_6 share1    /DockerFpg_6/DockerVfs_6/share1/  normal 127.0.0.1
share2    DockerVfs_6 share2    /DockerFpg_6/DockerVfs_6/share2/  normal 127.0.0.1
share3    DockerVfs_6 share3    /DockerFpg_6/DockerVfs_6/share3/  normal 127.0.0.1
share4    DockerVfs_6 share4    /DockerFpg_6/DockerVfs_6/share4/  normal 127.0.0.1
share5    DockerVfs_6 share5    /DockerFpg_6/DockerVfs_6/share5/  normal 127.0.0.1
share6    DockerVfs_6 share6    /DockerFpg_6/DockerVfs_6/share6/  normal 127.0.0.1
share7    DockerVfs_6 share7    /DockerFpg_6/DockerVfs_6/share7/  normal 127.0.0.1
share8    DockerVfs_6 share8    /DockerFpg_6/DockerVfs_6/share8/  normal 127.0.0.1
share9    DockerVfs_6 share9    /DockerFpg_6/DockerVfs_6/share9/  normal 127.0.0.1
share10   DockerVfs_6 share10   /DockerFpg_6/DockerVfs_6/share10/ normal 127.0.0.1
share11   DockerVfs_6 share11   /DockerFpg_6/DockerVfs_6/share11/ normal 127.0.0.1
share12   DockerVfs_6 share12   /DockerFpg_6/DockerVfs_6/share12/ normal 127.0.0.1
share13   DockerVfs_6 share13   /DockerFpg_6/DockerVfs_6/share13/ normal 127.0.0.1
share14   DockerVfs_6 share14   /DockerFpg_6/DockerVfs_6/share14/ normal 127.0.0.1
share15   DockerVfs_6 share15   /DockerFpg_6/DockerVfs_6/share15/ normal 127.0.0.1
----------------------------------------------------------------------------------
       15 total
CSIM_EOS14_1674003 cli% showfshare nfs -fpg DockerFpg_7
ShareName VFS         FileStore ------------SharePath------------ State  -Clients-
share16   DockerVfs_7 share16   /DockerFpg_7/DockerVfs_7/share16/ normal 127.0.0.1
share17   DockerVfs_7 share17   /DockerFpg_7/DockerVfs_7/share17/ normal 127.0.0.1
share18   DockerVfs_7 share18   /DockerFpg_7/DockerVfs_7/share18/ normal 127.0.0.1
share19   DockerVfs_7 share19   /DockerFpg_7/DockerVfs_7/share19/ normal 127.0.0.1
share20   DockerVfs_7 share20   /DockerFpg_7/DockerVfs_7/share20/ normal 127.0.0.1
share21   DockerVfs_7 share21   /DockerFpg_7/DockerVfs_7/share21/ normal 127.0.0.1
share22   DockerVfs_7 share22   /DockerFpg_7/DockerVfs_7/share22/ normal 127.0.0.1
share23   DockerVfs_7 share23   /DockerFpg_7/DockerVfs_7/share23/ normal 127.0.0.1
share24   DockerVfs_7 share24   /DockerFpg_7/DockerVfs_7/share24/ normal 127.0.0.1
share25   DockerVfs_7 share25   /DockerFpg_7/DockerVfs_7/share25/ normal 127.0.0.1
share26   DockerVfs_7 share26   /DockerFpg_7/DockerVfs_7/share26/ normal 127.0.0.1
share27   DockerVfs_7 share27   /DockerFpg_7/DockerVfs_7/share27/ normal 127.0.0.1
share28   DockerVfs_7 share28   /DockerFpg_7/DockerVfs_7/share28/ normal 127.0.0.1
share29   DockerVfs_7 share29   /DockerFpg_7/DockerVfs_7/share29/ normal 127.0.0.1
share30   DockerVfs_7 share30   /DockerFpg_7/DockerVfs_7/share30/ normal 127.0.0.1
----------------------------------------------------------------------------------
       15 total