search

vmware-archive / photon-controller

Photon Controller
Other
26 stars 4 forks source link

Unable to deploy kubernetes cluster/create VM on photon controller - UnsupportedFileSystem: FS type 99 error #53

Open vasu-yadlapalli opened 7 years ago

vasu-yadlapalli commented 7 years ago

Hi,

I am facing issues on deploying the kuberntes cluster on photon controller, in fact not able to create any VM also . I have uploaded kebenetes image as below and enabled kebernetes cluster on photon controller.

esxcloud@esxcloud-management-vm [ ~ ]$ photon image list Using target 'http://100.99.49.162:28080' ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 4b46441d-af09-4f18-b3a0-053d8547e637 photon-1.0TP2.ova READY 16777216098 ON_DEMAND 17% 100% cb4565cf-3790-4722-b376-451d4288067c photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 17% 100% edfd2c6f-c7e5-4099-971c-3a42a0ed149f photon-kubernetes-vm.vmdk READY 85899345920 ON_DEMAND 17% 100%

Below is the error on photon controller agent log on ESX server: INFO [2016-10-24 08:26:33,521] [1625936:1625904:Thread-19] [image_manager.py:_move_image:292] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: _move_image: /vmfs/volumes/vvol:6090a058e019896e-6080d532b104a018/ ERROR [2016-10-24 08:26:33,522] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: Move image 027e21b7-df14-4598-93df-750031a9e179 to /vmfs/volumes/vvol:60 ERROR [2016-10-24 08:26:33,522] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: Traceback (most recent call last): ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI ERROR [2016-10-24 08:26:33,523] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: File "/var/lib/jenkins/workspace/esxcloud-python/python/create_vib.WrI ERROR [2016-10-24 08:26:33,524] [1625936:1625904:Thread-19] [init.py:exception:1193] [Request:038297ab-eb52-4b85-a5fe-ef75937f635a] host.image.image_manager: UnsupportedFileSystem: FS type 99

I am not able to deploy any VM or cluster on the photon controller. If this is not the right forum to submit the issue, please let me know the correct forum to raise same.

Below is the photon controller log snippet:

INFO [2016-10-24 08:31:21,417] com.vmware.photon.controller.common.clients.HostClient: finished create_vm target 100.99.49.77, reservation b74c7164-297b-4eac-b007-580fd84fcd2d INFO [2016-10-24 08:31:21,417] com.vmware.photon.controller.common.clients.HostClient: Checking CreateVmResponse(result:SYSTEM_ERROR, error:FS type 99) INFO [2016-10-24 08:31:21,426] com.vmware.photon.controller.common.metrics.RpcMetricInterceptor: Caught exception during createVm: {} ! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99 ! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640) ! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.access$600(HostClient.java:1415) ! at com.vmware.photon.controller.common.clients.HostClient.createVm(HostClient.java:507) ! at com.vmware.photon.controller.common.metrics.RpcMetricInterceptor.invoke(RpcMetricInterceptor.java:51) ! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.createVm(VmCreateStepCmd.java:91) ! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.execute(VmCreateStepCmd.java:75) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-10-24 08:31:21,426] com.vmware.photon.controller.common.metrics.RpcMetricInterceptor: Finished call to createVm ERROR [2016-10-24 08:31:21,426] com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd: failed creating VM 27e25bf8-f805-4f6a-8f94-99257e33c7de ! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99 ! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640) ! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.access$600(HostClient.java:1415) ! at com.vmware.photon.controller.common.clients.HostClient.createVm(HostClient.java:507) ! at com.vmware.photon.controller.common.metrics.RpcMetricInterceptor.invoke(RpcMetricInterceptor.java:51) ! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.createVm(VmCreateStepCmd.java:91) ! at com.vmware.photon.controller.api.frontend.commands.steps.VmCreateStepCmd.execute(VmCreateStepCmd.java:75) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-10-24 08:31:21,427] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={PATCH}, OperationId={12755}, Uri={http://127.0.0.1:19000/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,428] com.vmware.photon.controller.cloudstore.xenon.entity.VmService: [Req: d55f3bd6-c2d3-46bd-bd44-c3748bfeaf48] [/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de] Patching service /photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de INFO [2016-10-24 08:31:21,433] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={PATCH}, OperationId={12755}, Uri={http://127.0.0.1:19000/photon/cloudstore/vms/27e25bf8-f805-4f6a-8f94-99257e33c7de}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,433] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={POST}, OperationId={12759}, Uri={http://127.0.0.1:19000/core/node-selectors/default/forwarding?path=/core/local-query-tasks&target=ALL}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,445] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={POST}, OperationId={12759}, Uri={http://127.0.0.1:19000/core/node-selectors/default/forwarding?path=/core/local-query-tasks&target=ALL}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,446] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={GET}, OperationId={12762}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,446] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={GET}, OperationId={12762}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,448] com.vmware.photon.controller.common.xenon.XenonRestClient: send: STARTED Action={PATCH}, OperationId={12764}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-10-24 08:31:21,455] com.vmware.photon.controller.cloudstore.xenon.entity.DiskService: [Req: d55f3bd6-c2d3-46bd-bd44-c3748bfeaf48] [/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50] Patching DiskService /photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50 INFO [2016-10-24 08:31:21,463] com.vmware.photon.controller.common.xenon.XenonRestClient: send: SUCCESS Action={PATCH}, OperationId={12764}, Uri={http://127.0.0.1:19000/photon/cloudstore/disks/7c14818f-181a-4d8b-90d0-1845f29eae50}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} ERROR [2016-10-24 08:31:21,463] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception ! com.vmware.photon.controller.common.clients.exceptions.SystemErrorException: FS type 99 ! at com.vmware.photon.controller.common.clients.HostClient$ResponseValidator.checkCreateVmResponse(HostClient.java:1640)

vasu-yadlapalli commented 7 years ago

I ues below command

esxcloud@esxcloud-management-vm [ ~ ]$ photon cluster create -n k8-cluster -k KUBERNETES --dns 100.99.8.39 --gateway 100.99.48.1 --netmask 255.255.248.0 --master-ip 100.99.49.165 --container-network 172.0.0.1/16 --etcd1 100.99.49.166 -s 1 -v cluster-tiny-vm Using target 'http://100.99.49.162:28080' etcd server 2 static IP address (leave blank for none):

Creating cluster: k8-cluster (KUBERNETES) VM flavor: cluster-tiny-vm Slave count: 1

Are you sure [y/n]? y 2016/10/26 09:29:22 photon: Task 'a1ebbce8-3c9f-4392-9d4d-703d783dd0fd' is in error state: {@step=={"sequence"=>"1","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error [Task "CREATE_VM": step "RESERVE_RESOURCE" failed with error code "InternalError", message "Please contact the system administrator about request #c05056b2-6878-4228-a2c3-2482e3df5c92"]. /photon/clustermanager/vm-provision-tasks/1b1cccb8-1ab0-4ed6-9db3-97a6880f5768]', data: 'map[]' }],"warnings"=>[],"operation"=>"CREATE_KUBERNETES_CLUSTER_SETUP_ETCD","startedTime"=>"1477474156774","queuedTime"=>"1477474156747","endTime"=>"1477474161777","options"=>map[]}} API Errors: [photon: { HTTP status: '0', code: 'InternalError', message: 'Failed to rollout KubernetesEtcd. Error: MultiException[java.lang.IllegalStateException: VmProvisionTaskService failed with error [Task "CREATE_VM": step "RESERVE_RESOURCE" failed with error code "InternalError", message "Please contact the system administrator about request #c05056b2-6878-4228-a2c3-2482e3df5c92"]. /photon/clustermanager/vm-provision-tasks/1b1cccb8-1ab0-4ed6-9db3-97a6880f5768]', data: 'map[]' }]

esxcloud@esxcloud-management-vm [ ~ ]$ docker version Client: Version: 1.11.0 API version: 1.23 Go version: go1.5.4 Git commit: 4dc5990 Built: Wed Apr 13 19:36:04 2016 OS/Arch: linux/amd64

Server: Version: 1.11.0 API version: 1.23 Go version: go1.5.4 Git commit: 4dc5990 Built: Wed Apr 13 19:36:04 2016 OS/Arch: linux/amd64

mwest44 commented 7 years ago

Update: I just saw your earlier post with log info. What kind of datastore are you deploying on? Which version of Photon Controller?

General Info on this type of error: Reserve resource error means that there was no resource to place the VM. Check the Flavor you used. Make sure number of CPUs is not greater than your host (important if you are using nested hosts) and that there is enough RAM and storage. Remember that by default there is no overcommit and we calculate the available resource based on the Total available - the sum of all "Configured" resources rather than "Active". So even powered off VMs will be count as taking resource.

Also check that you have a network created. depending on the build you are using, we require you to either specify the network to place the VMs on or to set a default network. Check the Photon Network -h for the command

you can also get this error when you have used all of the resources allocated to your project through the Resource Ticket

vasu-yadlapalli commented 7 years ago

Hi,

I have used 1.0.0 version of photon controller. Its VMFS5 datastore having sufficient staorage.

Flavor info :+1 fe572a50-9f88-4872-a7bb-1202f046b67b cluster-tiny-vm vm vm 1 COUNT vm.flavor.cluster-other-vm 1 COUNT vm.cpu 1 COUNT vm.memory 1 GB vm.cost 1 COUNT ESX have 8 CPU, 16GB memory available and 300 GB free space on datastore.

I have created a Project with 10 vCPU and 10GB memory which is more than my esx have, is that a problem?

e3c78efd-f881-4758-b987-2ad68339b210 prject2 vm.cpu 10 COUNT vm.cost 2 COUNT vm.memory 10 GB vm.cpu 2 COUNT vm 30 COUNT vm.memory 2 GB ephemeral-disk.capacity 200 GB vm.flavor.cluster-other-vm 2 COUNT ephemeral-disk 10 COUNT vm 2 COUNT ephemeral-disk.capacity 160 GB ephemeral-disk.flavor.cluster-vm-disk 2 COUNT ephemeral-disk 2 COUNT ephemeral-disk.cost 2 COUNT

Also when I see the Image replication progress is not 100%, is that a issue?

esxcloud@esxcloud-management-vm [ ~ ]$ photon image show edfd2c6f-c7e5-4099-971c-3a42a0ed149f Using target 'http://100.99.49.162:28080' Image ID: edfd2c6f-c7e5-4099-971c-3a42a0ed149f Name: photon-kubernetes-vm.vmdk State: READY Size: 85899345920 Byte(s) Image Replication Type: ON_DEMAND Image Replication Progress: 17% Image Seeding Progress: 100% Settings:

vasu-yadlapalli commented 7 years ago

Network info: esxcloud@esxcloud-management-vm [ ~ ]$ photon network list Using target 'http://100.99.49.162:28080' ID Name State PortGroups Descriptions IsDefault d0c61a9a-58d0-42e2-b122-519982300b3f dev-network READY [VM Network] Dev Network for VMs true

mwest44 commented 7 years ago

I'm checking into your actual error. One note: We have always said that the Kubernetes image should be EAGER instead of ON_DEMAND. I think that is just because the copy from image datastore to the datastore where VM is placed could cause a timeout. I don't think that is your problem here though. Can you try a single VM create, you can download the photonOS image from vmware.github.io I suspect it will fail as well.

mwest44 commented 7 years ago

Are you using Vvols? Photon platform assumes no vCenter and we don't support any vCenter dependent constructs (Datastore Clusters, Distributed switch, etc.) I've never tried this with vVols, but I would expect it to fail.

vasu-yadlapalli commented 7 years ago

Hi,

I tried creating single vm using photon-1.0TP2.ova image but that also resulted same error. In fact i tried to create VM using photon-management-vm-disk1.vmdk image also , but failed with same error.

My esx box having Vvol and other SAN based datastores, but i am not using those datastores any where in photon deployment . The one I am using was local disk of ESX, no shared storage. I am bit surprised why photon controller tries to copy the image on the other data stores? I saw some log entries which tries to copy image or searching image on other datastores of ESX.

mwest44 commented 7 years ago

Unless you explicitly use the DATASTORES_ALLOWED parameter in your deployment.yml file, PC will automatically discover all datastores attached to your hosts and use them as candidates for placement.

vasu-yadlapalli commented 7 years ago

Thanks for the reply. Below is deployment file used for PC management VM

hosts:

i havent added any other param? Let me know if any extra params needed here?

mwest44 commented 7 years ago

This info is all available on our github wiki. There are sample templates on this page. You should make sure to add the ALLOWED_DATASTORES tag to every Cloud host metadata section.

https://github.com/vmware/photon-controller/wiki/deployment-template