search

nmaupu / freenas-provisioner

Kubernetes external provisioner using Freenas as backend
Apache License 2.0
132 stars 26 forks source link

message: {"error_message":"Sorry, this request could not be processed. Please try again later."}, status: 404 #28

Closed thomasdbaker closed 5 years ago

thomasdbaker commented 5 years ago

Setting up a monster Lab and would love to get this working. I have the managed-nfs-storage working but that seems to proxy an export vs interact with the API.

Normal Provisioning 4s (x2 over 19s) freenas.org/nfs_freenas-nfs-provisioner-5746757fdd-cj9kg_10e56dd8-f801-11e9-b9dd-62478df8baf0 External provisioner is provisioning volume for claim "default/freenas-test-pvc" Warning ProvisioningFailed 4s (x2 over 19s) freenas.org/nfs_freenas-nfs-provisioner-5746757fdd-cj9kg_10e56dd8-f801-11e9-b9dd-62478df8baf0 failed to provision volume with StorageClass "freenas-nfs": Error getting dataset "mnt/k8spool03" - message: {"error_message":"Sorry, this request could not be processed. Please try again later."}, status: 404 Normal ExternalProvisioning 3s (x3 over 19s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "freenas.org/nfs" or manually created by system administrator

Here is what I see when calling the API manually.

{ "avail": 101417787392, "id": 109, "mountpoint": "/mnt/k8spool03", "name": "k8spool03", "path": "k8spool03", "status": "-", "type": "dataset", "used": 421888, "used_pct": 0 }

FreeNas version: FreeNAS-11.2-U6

travisghansen commented 5 years ago

I think you may have a configuration off a little bit. Can you send over the storage class definition you used?

thomasdbaker commented 5 years ago

I think you may have a configuration off a little bit. Can you send over the storage class definition you used?

That would be great if that was the case. I tried it both ways and went back to default other than the datasetParentName and changing the mountOptions from an Object to an Array. You can see the values that I used below.

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: freenas-nfs

annotations:

storageclass.kubernetes.io/is-default-class: "true"

provisioner: freenas.org/nfs allowVolumeExpansion: true reclaimPolicy: Delete mountOptions: [] parameters:

namespace of the secret which contains FreeNAS server connection details

default: kube-system

serverSecretNamespace: "kube-system"

name of the secret which contains FreeNAS server connection details

default: freenas-nfs

serverSecretName: "freenas-nfs"

the name of the parent dataset (or simply pool) where all resources will

be created, it must exist before provisioner will work

example: tank/k8s/mycluster

default: tank

datasetParentName: "mnt/k8spool03"

whether to enforce quotas for each dataset

if enabled each newly provisioned dataset will set the appropriate quota

per the PVC

default: true

datasetEnableQuotas: "true"

whether to reserve space when the dataset is create

if enabled each newly provisioned dataset will set the appropriate value

per the PVC

default: true

datasetEnableReservation: "true"

if enabled provisioner will create parent datasets for each namespace

otherwise, all datasets will be provisioned in a flat manner

default: true

datasetEnableNamespaces: "true"

if datasetEnableNamespaces is enabled, sets a per-namespace quota

example: 5M | 10G | 1T (M, Mi, MB, MiB, G, Gi, GB, GiB, T, Ti, TB, or TiB)

default: 0 (no quota)

datasetNamespaceQuota:

if datasetEnableNamespaces is enabled, sets a per-namespace reservation

this should not be greater than the datasetNamespaceQuota

example: 5M | 10G | 1T (M, Mi, MB, MiB, G, Gi, GB, GiB, T, Ti, TB, or TiB)

default: 0

datasetNamespaceReservation:

if enabled created datasets will adhere to reliable pattern

if datasetNamespaces == true dataset pattern is: //

if datasetNamespaces == false dataset pattern is: /-

if disabled, datasets will be created with a name pvc- (the name of the provisioned PV)

default: true

datasetDeterministicNames: "true"

if enabled and datasetDeterministicNames is enabled then dataset that

already exist (pre-povisioned out of band) will be ratained by the

provisioner during deletion of the reclaim process

ignored if datasetDeterministicNames is disabled (collisions result in failure)

default: true

datasetRetainPreExisting: "true"

the following parameters determine permissions and ownership of the

dataset mount directory (on FreeNAS) immediately upon creation

default: 0777, root, wheel

datasetPermissionsMode: "0777"

datasetPermissionsUser: "root"

datasetPermissionsGroup: "wheel"

this determines what the 'server' property of the NFS share will be in

in kubernetes, it's purpose is to provide flexibility between the control

and data planes of FreeNAS

default: uses the 'host' value from the secret

shareHost: "192.168.1.11"

determines if newly created NFS shares will have the 'All Directories'

option checked

default: true

shareAlldirs: "true"

Authorized hosts/networks (space-separated) allowed to access the shares

default: ""

shareAllowedHosts:

shareAllowedNetworks: "10.0.0.0/24, 192.168.1.0/24"

Determines root mapping

cannot be used simultaneously with shareMapAll{User,Group}

default: root:wheel

shareMaprootUser: root

shareMaprootGroup: wheel

Determines user mapping for all access (not recommended)

cannot be used simultaneously with shareMaproot{User,Group}

default: ""

shareMapallUser: "root"

shareMapallGroup: "wheel"

if enabled and datasetDeterministicNames is enabled then shares that

already exist (pre-provisioned out of band) will be retained by the

provisioner during deletion of the reclaim process

ignored if datasetDeterministicNames is disabled (collisions result in failure)

default: true

shareRetainPreExisting: "true"

travisghansen commented 5 years ago

Do you have a full pool you want to dedicate to this? k8spool03?

thomasdbaker commented 5 years ago

Yes that is correct.

Do you have a full pool you want to dedicate to this? k8spool03?

travisghansen commented 5 years ago

Ok, remove the mnt/ portion from the datasetParentName parameter.

I would recommend creating at least one level of dataset however instead of using the root pool dataset. For example create k8spool03/<cluster name> and set the parameter as appropriate as well.

Also you may want to delete the pcv and cleanup any resources on FreeNAS before as well (basically start over). I don't think you'll have anything created at this point but doesn't hurt to be sure (would only be a child dataset and the nfs share).

thomasdbaker commented 5 years ago

Ok, remove the mnt/ portion from the datasetParentName parameter.

I would recommend creating at least one level of dataset however instead of using the root pool dataset. For example create k8spool03/<cluster name> and set the parameter as appropriate as well.

Also you may want to delete the pcv and cleanup any resources on FreeNAS before as well (basically start over). I don't think you'll have anything created at this point but doesn't hurt to be sure (would only be a child dataset and the nfs share).

You are a Genius! that was the Key! Thank you very much!

freenas-test-pvc Bound pvc-0a057f09-9cc5-4f0a-b108-a07dc28e4e2a 1Mi RWX freenas-nfs 68s image

travisghansen commented 5 years ago

Nice! Thanks for checking it out! The iscsi variant is available here if you have need for use cases where block is a better fit (again, I highly recommend creating a child dataset before going much further so you can have a single pool better service multiple clusters and multiple providers).

I've also been working on a new csi driver to replace both projects with 'the future' as it were. It'll support resize operations, snapshots, etc, etc. I made a bunch of progress and got derailed a bit.

For dataset naming I'd do something like: k8spool03/a/nfs and k8spool03/a/iscsi. For this project you'd set datasetParentName to the first and for the iscsi one the second. Then for cluster b replace a.