Open alancivita opened 7 years ago
Your .196 host is defined twice. I'm surprised this deployed You also need to make sure that each host (Mgmt and cloud) is connected to at least one data store that is tagged as an image data store
Sent from my iPhone
On Dec 2, 2016, at 1:25 AM, alancivita notifications@github.com<mailto:notifications@github.com> wrote:
Hi i've started an environment from scratch: 4 esx hosts U2 P4 with no shared storage this time: each host has a local datastore nated datastore42x (with x = c d e f) they yaml file looks like this: hosts:
the deployment went fine, i've created Lighwve users and i'm able to create tenants, images flavors and so on, but i can't create vms. when i try to create a vm i always get this error:
016/12/02 09:09:34 photon: Task '478312f4542a94bff0640' is in error state: {@stephttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_step&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=yxrCtdQKD-SMogDiKPmN6FjPaiJ8QZXQBuPxPqQLvao&e==={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }],"warnings"=>[],"operation"=>"RESERVE_RESOURCE","startedTime"=>"1480669775470","queuedTime"=>"1480669775463","endTime"=>"1480669775492","options"=>map[54577808-ee79-40eb-a8d1-a91b439c34cd:vm]}} API Errors: [photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }]
the images don't replicate between the hypervisors as far as i can see, furthermore i can't uploaded images with ON_DEMAND option.
Am i doing something wrong? Thanks. Alan
- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_vmware_photon-2Dcontroller_issues_70&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=dXFPT_bECp2et9de4Wnt3rNpO5I-coIAH6JJ2Jkw2oM&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIPyrJxwuaqB3sAeghMuNtqCnsCkSK0gks5rD-2DP6gaJpZM4LCWeV&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=an5apMyurmjJ6tXYVOr-I1xWykoCRs6dcmxtq5Xwr0o&e=.
The .196 was a copy mistake sorry... address_ranges: 10.35.20.197-10.35.20.198 this is the correct one that has been used.
i tagged all 4 datastores as image data stores.
thanks for the reply!
you will need to look at /var/log/esxcloud/photon-controller-core.log in the management VM that is handling the VM create to see more detail on the error. Specifically you need to verify if it is actually the disk that is the missing resource. Could also be the network. I don't actually know how we are calculating those percentages when using only local storage. You can browse the datastores on each of the hosts to verify if the images are actually on each host.
Hi Michael, well i've check on each datastore and the image is not replicated, the datastores on the cloud hosts are empty. I've noticed something strange: each host hast also 2 shared datastores but i'm not using them. I've unmounted shared datastores and redeployed photon controller..same issues, but in the logs i've found this line: INFO [2016-12-05 09:22:22,026] com.vmware.photon.controller.common.clients.HostClient: Checking GetConfigResponse(result:OK, hostConfig:HostConfig(agent_id:2f18a5e5542d1da6a02e0, datastores:[Datastore(id:58369b27-6e82acd4-741e-d89d673046b8, name:S57ESXVMFS516, type:SHARED_VMFS, tags:[SHARED_VMFS]), Datastore(id:58369b15-35af6c2a-7d4b-d89d673046b8, name:S57ESXVMFS515, type:SHARED_VMFS, tags:[SHARED_VMFS]), Datastore(id:5840053b-429644f0-6ee3-d89d67203eb8, name:datastore42d, type:SHARED_VMFS, tags:[SHARED_VMFS])], address:ServerAddress(host:10.90.233.196, port:8835), networks:[Network(id:Public-10.90.52.0-23, types:[VM]), Network(id:Public-10.90.44.0-23, types:[VM])], management_only:true, image_datastore_ids:[5840053b-429644f0-6ee3-d89d67203eb8], memory_mb:196573, cpu_count:24, esx_version:6.0.0, deployment_id:2f18a5e5542d1da62a428))
where S57ESXVMFS515 and S57ESXVMFS516 are the unmounted datastores.
this is the error i get when i try to create a VM: ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/scheduler/placement/4e8a1d51542e60710c888] java.lang.Throwable: Place failure, constraints cannot be satisfied for request: Resource(vm:Vm(id:ba595d9b-5680-4422-870e-b4a275c72ecd, flavor:cluster-other-vm, state:STOPPED, disks:[Disk(id:67bd2f06-5008-4f97-82c2-50b1120478ff, flavor:4e8a1d51542d23246ee60, persistent:false, new_disk:true, capacity_gb:15, image:DiskImage(id:aa9c52a4-701e-47d1-9b06-2cee5984e212, clone_type:COPY_ON_WRITE), flavor_info:Flavor(name:cluster-vm-disk, cost:[QuotaLineItem(key:ephemeral-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.flavor.cluster-vm-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.cost, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.capacity, value:15.0, unit:GB)]))], flavor_info:Flavor(name:cluster-other-vm, cost:[QuotaLineItem(key:vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.flavor.cluster-other-vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.cpu, value:1.0, unit:COUNT), QuotaLineItem(key:vm.memory, value:4.0, unit:GB), QuotaLineItem(key:vm.cost, value:1.0, unit:COUNT)]), resource_constraints:[ResourceConstraint(type:NETWORK, values:[Public-10.90.52.0-23])], tenant_id:4e8a1d51542d2603093f9, project_id:4e8a1d51542d261291528)) at com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService.handleGetCandidateResult(PlacementTaskService.java:260) at com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService.lambda$handlePlaceRequest$0(PlacementTaskService.java:222) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_Success(CloudStoreConstraintChecker.java:632) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_HandleStep(CloudStoreConstraintChecker.java:272) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_AnalyzeCandidates(CloudStoreConstraintChecker.java:610) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_HandleStep(CloudStoreConstraintChecker.java:269) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.lambda$getCandidates_GetCandidates$10(CloudStoreConstraintChecker.java:574) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.lambda$startService$10(ServiceHost.java:2113) at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2607) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2562) at com.vmware.xenon.common.ServiceHost.lambda$processServiceStart$20(ServiceHost.java:2523) at com.vmware.xenon.common.Operation.lambda$nestCompletion$2(Operation.java:1248) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2538) at com.vmware.xenon.common.ServiceHost.lambda$processServiceStart$19(ServiceHost.java:2482) at com.vmware.xenon.common.Operation.lambda$nestCompletion$2(Operation.java:1248) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.services.common.QueryTaskService.handleQueryCompletion(QueryTaskService.java:598) at com.vmware.xenon.services.common.QueryTaskService.lambda$forwardQueryToDocumentIndexService$5(QueryTaskService.java:481) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.services.common.LuceneDocumentIndexService.handleQueryTaskPatch(LuceneDocumentIndexService.java:699) at com.vmware.xenon.services.common.LuceneDocumentIndexService.handleQueryRequest(LuceneDocumentIndexService.java:596) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd: reserve resource failed: NO_SUCH_RESOURCE, Place failure, constraints cannot be satisfied for request: Resource(vm:Vm(id:ba595d9b-5680-4422-870e-b4a275c72ecd, flavor:cluster-other-vm, state:STOPPED, disks:[Disk(id:67bd2f06-5008-4f97-82c2-50b1120478ff, flavor:4e8a1d51542d23246ee60, persistent:false, new_disk:true, capacity_gb:15, image:DiskImage(id:aa9c52a4-701e-47d1-9b06-2cee5984e212, clone_type:COPY_ON_WRITE), flavor_info:Flavor(name:cluster-vm-disk, cost:[QuotaLineItem(key:ephemeral-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.flavor.cluster-vm-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.cost, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.capacity, value:15.0, unit:GB)]))], flavor_info:Flavor(name:cluster-other-vm, cost:[QuotaLineItem(key:vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.flavor.cluster-other-vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.cpu, value:1.0, unit:COUNT), QuotaLineItem(key:vm.memory, value:4.0, unit:GB), QuotaLineItem(key:vm.cost, value:1.0, unit:COUNT)]), resource_constraints:[ResourceConstraint(type:NETWORK, values:[Public-10.90.52.0-23])], tenant_id:4e8a1d51542d2603093f9, project_id:4e8a1d51542d261291528)) ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception ! com.vmware.photon.controller.api.frontend.exceptions.external.NoSuchResourceException: null ! at com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd.loadReservation(ResourceReserveStepCmd.java:441) ! at com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd.execute(ResourceReserveStepCmd.java:162) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,012] com.vmware.photon.controller.cloudstore.xenon.entity.TaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/tasks/4e8a1d51542e607107298] Patching service /photon/cloudstore/tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,013] com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd: Resource reservation failed, mark entity ba595d9b-5680-4422-870e-b4a275c72ecd state as ERROR INFO [2016-12-05 09:36:52,014] com.vmware.photon.controller.cloudstore.xenon.entity.VmService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/vms/ba595d9b-5680-4422-870e-b4a275c72ecd] Patching service /photon/cloudstore/vms/ba595d9b-5680-4422-870e-b4a275c72ecd INFO [2016-12-05 09:36:52,022] com.vmware.photon.controller.cloudstore.xenon.entity.DiskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/disks/67bd2f06-5008-4f97-82c2-50b1120478ff] Patching DiskService /photon/cloudstore/disks/67bd2f06-5008-4f97-82c2-50b1120478ff INFO [2016-12-05 09:36:52,023] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Processed in 32ms ERROR [2016-12-05 09:36:52,023] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception ! com.vmware.photon.controller.api.frontend.exceptions.external.TaskNotCompletedException: Step "StepEntity{id=null, kind=step, state=ERROR, operation=RESERVE_RESOURCE, startedTime=Mon Dec 05 09:36:51 GMT 2016, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=Mon Dec 05 09:36:52 GMT 2016, sequence=0, error=StepErrorEntity{id=null, kind=step-error, code=NoSuchResource, message=null}}" did not complete. ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:118) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,024] com.vmware.photon.controller.cloudstore.xenon.entity.EntityLockService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/entity-locks/ba595d9b-5680-4422-870e-b4a275c72ecd] Handling PUT for EntityLockService /photon/cloudstore/entity-locks/ba595d9b-5680-4422-870e-b4a275c72ecd INFO [2016-12-05 09:36:52,024] com.vmware.photon.controller.api.frontend.backends.EntityLockXenonBackend: Entity Lock with taskId : 4e8a1d51542e607107298 and entityId : ba595d9b-5680-4422-870e-b4a275c72ecd has been cleared ERROR [2016-12-05 09:36:52,024] com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand: Task 4e8a1d51542e607107298 failed ! com.vmware.photon.controller.api.frontend.exceptions.external.TaskNotCompletedException: Step "StepEntity{id=null, kind=step, state=ERROR, operation=RESERVE_RESOURCE, startedTime=Mon Dec 05 09:36:51 GMT 2016, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=Mon Dec 05 09:36:52 GMT 2016, sequence=0, error=StepErrorEntity{id=null, kind=step-error, code=NoSuchResource, message=null}}" did not complete. ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:118) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,025] com.vmware.photon.controller.cloudstore.xenon.entity.TaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/tasks/4e8a1d51542e607107298] Patching service /photon/cloudstore/tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,026] com.vmware.photon.controller.api.frontend.backends.TaskXenonBackend: Task TaskEntity{id=4e8a1d51542e607107298, kind=task, entityId=ba595d9b-5680-4422-870e-b4a275c72ecd, entityKind=vm, state=QUEUED, operation=CREATE_VM, startedTime=null, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=null} has been marked as ERROR INFO [2016-12-05 09:36:52,026] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Processed in 39ms INFO [2016-12-05 09:36:52,116] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,254] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528 INFO [2016-12-05 09:36:52,388] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528/vms WARN [2016-12-05 09:36:52,390] com.vmware.photon.controller.common.xenon.XenonRestClient: send: WARN Action={POST}, StatusCode={202}, OperationId={1096113}, Uri={https://10.90.44.55:19000/core/query-tasks}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-12-05 09:36:52,627] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528/clusters WARN [2016-12-05 09:36:52,629] com.vmware.photon.controller.common.xenon.XenonRestClient: send: WARN Action={POST}, StatusCode={202}, OperationId={1096150}, Uri={https://10.90.44.55:19000/core/query-tasks}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}
I find it strange to see this: Datastore(id:5840053b-429644f0-6ee3-d89d67203eb8, name:datastore42d, type:SHARED_VMFS, tags:[SHARED_VMFS])] Since this is a local datastore.
Without access to the system its difficult to tell if this is a configuration issue or not. My suggestion is to add the ALLOWED_DATASTORES tag to your yaml and deploy again. ALLOWED DATASTORES provides a whitelist of datastores that PC can use. Without this tag, PC will discover all datastores and use them for possible VM placement.
ah ok i see, thanks, will give it a go shortly. Thanks
Hi again :) well i've redeployed everything without luck. i've used this yml file and executed the deployment using the command line:
hosts:
Not sure if the ALLOWED_DATASTORES option i've used is correct, but the deployment went fine, but still getting the same errors.
Sorry to bother you ! and thanks for your time, but just trying to figure out possible scenarios. attached the log file from the controller photon-controller.log.zip
What does photon image list
show you? I wonder if the image wasn't replicated before you tried to create the VM.
./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 9281ceb7-b873-4162-88fe-9231c7058414 photon-OS.ova READY 16777216098 EAGER 17% 25% b975421d-90c3-4f11-a975-80cc6d945b86 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%
and it is in this state for 30 mins at least
maybe related to this error: ERROR [2016-12-05 17:37:42,402] com.vmware.photon.controller.housekeeper.xenon.ImageSeederService: [Req: 604727f0-37dd-453f-81f7-da63197fb599] [/photon/housekeeper/image-seeder/b7d0828c542ecbce48429] java.lang.RuntimeException: Image seeding failed: 1 image seeding succeeded, 3 image seeding failed or cancelled at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.processAwaitCompletion(ImageSeederService.java:388) at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.handleStartedStage(ImageSeederService.java:252) at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.handlePatch(ImageSeederService.java:109) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:408) at com.vmware.xenon.common.StatefulService.lambda$loadAndLinkState$7(StatefulService.java:1175) at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.loadServiceState(ServiceHost.java:2698) at com.vmware.xenon.common.StatefulService.loadAndLinkState(StatefulService.java:1177) at com.vmware.xenon.common.StatefulService.handleRequestLoadingAndLinkingState(StatefulService.java:465) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:349) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:313) at com.vmware.xenon.common.ServiceHost.lambda$queueOrScheduleRequest$31(ServiceHost.java:3710) at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Can I assume that the photon-OS.ova is the file that you're using to create the VM? It looks like it's being replicated to all the hosts, but it's not there yet. I suspect that this is the underlying issue.
In the log I see this:
Image seeding failed: 1 image seeding succeeded, 3 image seeding failed or cancelled
Just before that, I see: ERROR [2016-12-05 17:37:38,983] com.vmware.photon.controller.housekeeper.xenon.ImageHostToHostCopyService: [/photon/ housekeeper/image-host-to-host-copiers/b7d0828c542ecbce4e5d8] com.vmware.photon.controller.common.clients.exceptions .SystemErrorException: Could not connect to 10.90.233.198:8835: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
That's the core problem.
As part of setting up Photon Controller, we install new SSL certificates on ESXi. Sometimes this makes vCenter unhappy. Are you using vCenter for the same hosts that you're using Photon Controller? I wonder if there's a conflict, or if something just went wrong in the certificate installation.
I'm not sure if the datastore list is the problem, but the deployment yaml you used will not solve the problem if it is. Basically you have said that 42c,d,e,f can be used on .195 and .196. These are your MGMT hosts and you have not tagged them for VM placement. The Cloud hosts have no ALLOWED_DATASTORES tag, so will try to use all Datastores. Add the ALLOWED_DATASTORES tag to .197 and .198 hosts
@AlainRoy Hi, i'm not using vCenter at the moment, just simple esxi hosts. @mwest44 Thanks, i see ! will try to redeploy again tomorrow morning, it's evening here. thanks Alan
I'm looking around for someone who is better at debugging issues around our SSL certificates.
To debug the SSL certificate issue, please help get:
Thanks, Long
Hi Long, please find attached the logs requested. bed time here, will back online tomorrow morning. thanks Alan
Thanks, Alan. I could not find the root cause yet, please help gather some more data:
Thanks, Long
Hi Long, 1) see attached file (i ran the command both from the photon-controller and the esx host) 2) [root@hdesx42c:~] ls -l /etc/vmware/ssl/ total 116 -rw-r--r-- 1 root root 1188 Nov 23 09:35 69af5a5a.0 -rw-r--r-- 1 root root 649 Nov 24 10:24 69af5a5a.r0 -rw-r--r-- 1 root root 1188 Nov 23 09:35 82b12678.0 -rw-r--r-- 1 root root 649 Nov 24 10:24 82b12678.r0 -rw-r--r-- 1 root root 1188 Dec 5 14:31 castore.pem -r-------- 1 root root 82667 Dec 5 14:31 rui.bak -rw-r--r-- 1 root root 1168 Dec 5 14:31 rui.crt -rw------- 1 root root 1709 Dec 5 14:31 rui.key -rw-r--r-T 1 root root 0 Nov 2 00:49 vsanvp_castore.pem -rw-r--r-- 1 root root 64 Nov 30 12:51 vsanvp_secret
3) ./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 9281ceb7-b873-4162-88fe-9231c7058414 photon-OS.ova READY 16777216098 EAGER 17% 25% b975421d-90c3-4f11-a975-80cc6d945b86 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%
4) see attached files photon-controller.log.zip openssl_output.txt.zip
Thanks Alan
The issue is that CA certificate files (/etc/vmware/ssl/.0, .r0) on ESXi are not refreshed after the host joined lightwave domain. These files have timestamps from Nov 23/24, and I expect timestamp from Dec 5.
Could you help get some more data?
Thanks, Long
HI, all 4 esx hosts have version: VMware ESXi 6.0.0 build-4600944 all 4 hosts have lwsmd running ps -c|grep lwsmd 912765 912765 grep grep lwsmd 851177 851177 sh /bin/sh /sbin/watchdog.sh -s lwsmd /usr/lib/vmware/likewise/sbin/lwsmd ++group=likewise --syslog 851189 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851190 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851191 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851192 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851193 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851194 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851195 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851196 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851197 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851198 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851208 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851235 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851361 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851362 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog
yes all 4 hosts have timestamps from 23/24 November for .0 .r0 files.. let me know if there is anything else i can do.. these are test hosts so i'm free to do whatever i want.
Thanks for your prompt response. I have forwarded the information to my colleagues who work on this component. Will get back to you soon. Thanks!
Hi Alan,
Can you please run following commands and get the output for each?
openssl x509 -in 69af5a5a.0 -noout -text and openssl x509 -in 82b12678.0 -noout -text /usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text
Also please collect the support bundle by running vm-supprt
Thanks -Suresh
Hi Suresh thanks for your reply, i ran the commands from only 1 of the esx hosts, let me know if you want me to run the command from each of them. 1) openssl x509 -in 69af5a5a.0 -noout -text [root@hdesx42e:/etc/vmware/ssl] openssl x509 -in 69af5a5a.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 2) openssl x509 -in 82b12678.0 -noout -text [root@hdesx42e:/etc/vmware/ssl] openssl x509 -in 82b12678.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 3) usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text [root@hdesx42e:/etc/vmware/ssl] /usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text Number of entries in store : 1 Alias : f4fa7b050b95f694967b6d449cefa1537e3b45e7 Entry type : Trusted Cert Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 the vm-support file is 35Mb and can't uploaded here.. i'll upload it on my gdrive and will provide a link shortly. I've generated vm-support only for 1 hosts let me know if you need for all of them. Many thanks Alan
i've split the tgx in 4 zip files. thanks esx-hdesx42e-2016-12-06--19.57-001.zip esx-hdesx42e-2016-12-06--19.57-002.zip esx-hdesx42e-2016-12-06--19.57-003.zip esx-hdesx42e-2016-12-06--19.57-004.zip
Thanks Alan,
Host certificate is signed by a root certificate from 10.90.44.54 and the root for that is not getting pulled from lightwave server. Endpoint certificate store also does not have the certificate in it's TRUSTED_ROOTS.
10.90.44.54 and 10.90.44.56 are they replicating each other?
Can you run /usr/lib/vmware/vmafd/bin/dir-cli nodes list to see what are the lightwave servers in the cluster?
Hi, that's odd because 10.90.44.54 is an old ip address from a previous deployment. photon controller has ip 10.90.44.55 and lightwave 10.90.44.56
[root@hdesx42e:~] /usr/lib/vmware/vmafd/bin/dir-cli nodes list Enter password for administrator@esxcloud: dir-cli failed, error= Could not connect to VMware Directory Service via LDAP.
the same command from the lightwave container provides: root [ /opt/vmware/bin ]# ./dir-cli nodes list Enter password for administrator@esxcloud: Node: ec-mgmt-10-90-233-196ccbef.esxcloud Type: PSC Site: Default-first-site
Node: ec-mgmt-10-90-233-1955a0d0 Type: Management
Node: hdesx42e Type: Management
Node: hdesx42f Type: Management
Node: hdesx42c Type: Management
Node: hdesx42d Type: Management
Hi there, a quick update: i was able to get the right certificates, signed by 10.90.44.56 and not 10.90.44.54 which was an old ip from a previous deployment: First i've removed the .0 and .r0 certificates from /etc/vmware/ssl/ and then removed the afd.db from /etc/vmware/vmafd/db/ directory. i've done this because this was the only place referring to 10.90.44.54 [root@hdesx42c:/etc/vmware/vmafd] grep -r "10.90.44.54" * ��5afd.d#10.90.44.54Default-first-siteesxcloudXJ��� ��/afd.desxcloudX5b� 10.90.44.54esxcloud
Then i've redeploy photon-controller from scratch and now cerficates are correctly signed by 10.90.44.56: [root@hdesx42d:/etc/vmware/ssl] openssl x509 -in 3df40f5b.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: ed:e0:29:ba:25:df:57:96 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.56 Validity Not Before: Dec 6 12:41:33 2016 GMT Not After : Dec 4 12:41:33 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.56 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c5:9b:6d:7e:4c:48:24:c9:21:23:92:d4:e5:40: 52:f5:24:05:a6:1d:32:e7:9a:60:a9:1f:90:04:fb: e5:5a:07:02:53:05:8d:ef:38:9d:6b:4e:6b:13:25: 98:bd:ed:50:92:fb:a4:68:3e:0a:d8:0c:c1:33:41: f6:ce:1a:4d:be:97:7c:68:b7:a0:7d:dd:58:0c:59: a4:39:f7:b1:97:97:0d:e2:a8:2e:11:9b:33:e4:f7: 73:f8:c4:66:67:ce:4c:9d:9e:ea:67:b0:0d:ff:2a: 31:db:87:54:ff:86:e7:aa:96:96:51:af:df:9a:0b: e0:e2:5d:03:36:cd:54:cb:01:e8:4b:6c:0a:01:ea: 44:87:38:e0:c9:4a:75:63:e0:ff:84:bc:b8:de:ef: 30:4b:71:17:18:73:cf:e4:08:16:1d:f6:1c:a7:d9: 8b:ef:46:71:7b:bf:91:3a:4a:7d:bd:7a:3b:73:f7: 44:8e:e1:cf:94:ea:8c:42:16:60:7a:50:da:2a:5e: 24:93:af:06:04:85:c6:81:7d:89:fa:a5:9d:40:a5: 96:1a:b3:49:3b:84:ed:f7:93:8d:9e:44:d9:bc:82: a1:66:59:13:d0:c7:0a:8b:21:4d:f7:19:a3:fb:dd: 36:77:2c:d7:e1:2a:46:f5:28:8c:ee:cd:5d:42:a1: b0:93 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 27:72:98:88:D8:7C:9B:F1:A9:10:47:71:75:58:74:09:96:02:45:D6 X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 8f:95:53:c3:d4:cd:a9:ff:48:e8:b4:1d:7e:8f:c6:5a:32:b8: 60:38:e2:ab:d5:94:50:56:51:99:65:cd:e9:d1:cb:9e:ef:bd: 1e:91:3e:61:26:3b:53:26:8c:ea:15:77:46:69:d1:70:03:cd: 32:fa:80:79:43:28:f0:38:c9:f3:51:e4:23:bb:7f:c9:de:da: ba:40:81:e4:b2:59:47:9d:99:ee:30:b2:4f:40:9e:f9:a2:36: c2:23:e2:b5:4a:ba:5d:ab:92:9e:16:2b:d5:4d:ec:07:06:93: 56:44:3e:85:75:87:0f:64:b2:01:ba:92:1b:dc:d0:0b:a5:68: c7:e8:51:5d:65:0d:b5:8f:aa:3d:69:0f:83:ea:72:08:60:e9: bd:a5:e3:04:78:42:b3:5f:df:7d:a7:ed:b5:3f:29:67:1b:82: db:5f:f4:7d:40:04:43:02:63:bc:f6:42:a1:de:74:df:11:77: a3:75:02:8e:e6:e0:ea:45:38:53:96:f5:58:ae:c1:a7:26:9a: 7c:90:06:45:2c:e1:91:9a:20:d9:70:10:fa:d8:7c:87:9b:a5: bc:0d:bb:af:00:25:94:e7:29:07:7a:29:3f:c6:cd:c1:96:40: b8:27:29:1f:77:98:fa:b2:e6:fa:98:01:d4:b4:4a:f9:49:68: 95:29:9c:fc
Finally, i can see images being replicated across all the hosts! and can create VMs
ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 67a5f7d8-0502-4cee-bd0c-9762e0564c60 photon-OS.ova READY 16777216098 EAGER 100% 100% 73c07279-be38-48b0-92aa-4f0c7d4cb11a photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 100% 100%
I still have a copy of afd.db and certificates.
Alan
Thanks for the update, Alan! I am glad it is working for you now.
Please send us the afd.db file and we can find out why certificates sync did not work.
Thanks, Long
i've attached the file.. just one question, shouldn't this file be deleted by photon system destroy? or overwritten by a new deployment? thanks Alan
Our design is to overwrite the certificates by new deployment. We need to investigate why it did not work. I had opened a bug.
thanks for the explanations, just a note, before deleting the adf.db I've only deleted the certificates and redeployed, but the certs were recreated with 10.90.44.54 (the old ip); After that I've destroyed the environment, deleted the certs and the afa.db...and this worked fine.
I did not find afd.db attached in your previous comment. Did I miss anything?
sorry, the original extension was unsupported :) afd.db.orig.zip
Great! thanks!
Hi i've started an environment from scratch: 4 esx hosts U2 P4 with no shared storage this time: each host has a local datastore nated datastore42x (with x = c d e f) they yaml file looks like this: hosts:
the deployment went fine, i've created Lighwve users and i'm able to create tenants, images flavors and so on, but i can't create vms. when i try to create a vm i always get this error:
016/12/02 09:09:34 photon: Task '478312f4542a94bff0640' is in error state: {@step=={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }],"warnings"=>[],"operation"=>"RESERVE_RESOURCE","startedTime"=>"1480669775470","queuedTime"=>"1480669775463","endTime"=>"1480669775492","options"=>map[54577808-ee79-40eb-a8d1-a91b439c34cd:vm]}} API Errors: [photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }]
the images don't replicate between the hypervisors as far as i can see, furthermore i can't uploaded images with ON_DEMAND option.
./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 2533b39f-918d-4b33-8e2e-080987461fe1 photonOS.ova READY 16777216098 EAGER 50% 25% 678e579a-bc0f-4fd6-8fc6-049f509090a0 kubernetes-1.4.3-pc-1.1.0-5de1cb7.ova READY 41943040096 EAGER 50% 25% f4848be3-acca-4b81-b5d4-988fb24de178 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%
Am i doing something wrong? Thanks. Alan