vmware-archive / photon-controller

Photon Controller
Other
27 stars 4 forks source link

Can't create vms with local datastores #70

Open alancivita opened 7 years ago

alancivita commented 7 years ago

Hi i've started an environment from scratch: 4 esx hosts U2 P4 with no shared storage this time: each host has a local datastore nated datastore42x (with x = c d e f) they yaml file looks like this: hosts:

the deployment went fine, i've created Lighwve users and i'm able to create tenants, images flavors and so on, but i can't create vms. when i try to create a vm i always get this error:

016/12/02 09:09:34 photon: Task '478312f4542a94bff0640' is in error state: {@step=={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }],"warnings"=>[],"operation"=>"RESERVE_RESOURCE","startedTime"=>"1480669775470","queuedTime"=>"1480669775463","endTime"=>"1480669775492","options"=>map[54577808-ee79-40eb-a8d1-a91b439c34cd:vm]}} API Errors: [photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }]

the images don't replicate between the hypervisors as far as i can see, furthermore i can't uploaded images with ON_DEMAND option.

./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 2533b39f-918d-4b33-8e2e-080987461fe1 photonOS.ova READY 16777216098 EAGER 50% 25% 678e579a-bc0f-4fd6-8fc6-049f509090a0 kubernetes-1.4.3-pc-1.1.0-5de1cb7.ova READY 41943040096 EAGER 50% 25% f4848be3-acca-4b81-b5d4-988fb24de178 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%

Am i doing something wrong? Thanks. Alan

mwest44 commented 7 years ago

Your .196 host is defined twice. I'm surprised this deployed You also need to make sure that each host (Mgmt and cloud) is connected to at least one data store that is tagged as an image data store

Sent from my iPhone

On Dec 2, 2016, at 1:25 AM, alancivita notifications@github.com<mailto:notifications@github.com> wrote:

Hi i've started an environment from scratch: 4 esx hosts U2 P4 with no shared storage this time: each host has a local datastore nated datastore42x (with x = c d e f) they yaml file looks like this: hosts:

the deployment went fine, i've created Lighwve users and i'm able to create tenants, images flavors and so on, but i can't create vms. when i try to create a vm i always get this error:

016/12/02 09:09:34 photon: Task '478312f4542a94bff0640' is in error state: {@stephttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_step&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=yxrCtdQKD-SMogDiKPmN6FjPaiJ8QZXQBuPxPqQLvao&e==={"sequence"=>"0","state"=>"ERROR","errors"=>[photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }],"warnings"=>[],"operation"=>"RESERVE_RESOURCE","startedTime"=>"1480669775470","queuedTime"=>"1480669775463","endTime"=>"1480669775492","options"=>map[54577808-ee79-40eb-a8d1-a91b439c34cd:vm]}} API Errors: [photon: { HTTP status: '0', code: 'NoSuchResource', message: '', data: 'map[]' }]

the images don't replicate between the hypervisors as far as i can see, furthermore i can't uploaded images with ON_DEMAND option.

Am i doing something wrong? Thanks. Alan

- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_vmware_photon-2Dcontroller_issues_70&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=dXFPT_bECp2et9de4Wnt3rNpO5I-coIAH6JJ2Jkw2oM&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AIPyrJxwuaqB3sAeghMuNtqCnsCkSK0gks5rD-2DP6gaJpZM4LCWeV&d=DgMCaQ&c=uilaK90D4TOVoH58JNXRgQ&r=NfogdLpANSFztq0YNdB2kA&m=jlnCJ7MPOClEd4TQOU5VGpXzvdbKwZcV_kjiIsRd6OQ&s=an5apMyurmjJ6tXYVOr-I1xWykoCRs6dcmxtq5Xwr0o&e=.

alancivita commented 7 years ago

The .196 was a copy mistake sorry... address_ranges: 10.35.20.197-10.35.20.198 this is the correct one that has been used.

i tagged all 4 datastores as image data stores.

thanks for the reply!

mwest44 commented 7 years ago

you will need to look at /var/log/esxcloud/photon-controller-core.log in the management VM that is handling the VM create to see more detail on the error. Specifically you need to verify if it is actually the disk that is the missing resource. Could also be the network. I don't actually know how we are calculating those percentages when using only local storage. You can browse the datastores on each of the hosts to verify if the images are actually on each host.

alancivita commented 7 years ago

Hi Michael, well i've check on each datastore and the image is not replicated, the datastores on the cloud hosts are empty. I've noticed something strange: each host hast also 2 shared datastores but i'm not using them. I've unmounted shared datastores and redeployed photon controller..same issues, but in the logs i've found this line: INFO [2016-12-05 09:22:22,026] com.vmware.photon.controller.common.clients.HostClient: Checking GetConfigResponse(result:OK, hostConfig:HostConfig(agent_id:2f18a5e5542d1da6a02e0, datastores:[Datastore(id:58369b27-6e82acd4-741e-d89d673046b8, name:S57ESXVMFS516, type:SHARED_VMFS, tags:[SHARED_VMFS]), Datastore(id:58369b15-35af6c2a-7d4b-d89d673046b8, name:S57ESXVMFS515, type:SHARED_VMFS, tags:[SHARED_VMFS]), Datastore(id:5840053b-429644f0-6ee3-d89d67203eb8, name:datastore42d, type:SHARED_VMFS, tags:[SHARED_VMFS])], address:ServerAddress(host:10.90.233.196, port:8835), networks:[Network(id:Public-10.90.52.0-23, types:[VM]), Network(id:Public-10.90.44.0-23, types:[VM])], management_only:true, image_datastore_ids:[5840053b-429644f0-6ee3-d89d67203eb8], memory_mb:196573, cpu_count:24, esx_version:6.0.0, deployment_id:2f18a5e5542d1da62a428))

where S57ESXVMFS515 and S57ESXVMFS516 are the unmounted datastores.

this is the error i get when i try to create a VM: ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/scheduler/placement/4e8a1d51542e60710c888] java.lang.Throwable: Place failure, constraints cannot be satisfied for request: Resource(vm:Vm(id:ba595d9b-5680-4422-870e-b4a275c72ecd, flavor:cluster-other-vm, state:STOPPED, disks:[Disk(id:67bd2f06-5008-4f97-82c2-50b1120478ff, flavor:4e8a1d51542d23246ee60, persistent:false, new_disk:true, capacity_gb:15, image:DiskImage(id:aa9c52a4-701e-47d1-9b06-2cee5984e212, clone_type:COPY_ON_WRITE), flavor_info:Flavor(name:cluster-vm-disk, cost:[QuotaLineItem(key:ephemeral-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.flavor.cluster-vm-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.cost, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.capacity, value:15.0, unit:GB)]))], flavor_info:Flavor(name:cluster-other-vm, cost:[QuotaLineItem(key:vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.flavor.cluster-other-vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.cpu, value:1.0, unit:COUNT), QuotaLineItem(key:vm.memory, value:4.0, unit:GB), QuotaLineItem(key:vm.cost, value:1.0, unit:COUNT)]), resource_constraints:[ResourceConstraint(type:NETWORK, values:[Public-10.90.52.0-23])], tenant_id:4e8a1d51542d2603093f9, project_id:4e8a1d51542d261291528)) at com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService.handleGetCandidateResult(PlacementTaskService.java:260) at com.vmware.photon.controller.rootscheduler.xenon.task.PlacementTaskService.lambda$handlePlaceRequest$0(PlacementTaskService.java:222) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_Success(CloudStoreConstraintChecker.java:632) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_HandleStep(CloudStoreConstraintChecker.java:272) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_AnalyzeCandidates(CloudStoreConstraintChecker.java:610) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.getCandidates_HandleStep(CloudStoreConstraintChecker.java:269) at com.vmware.photon.controller.rootscheduler.service.CloudStoreConstraintChecker.lambda$getCandidates_GetCandidates$10(CloudStoreConstraintChecker.java:574) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.lambda$startService$10(ServiceHost.java:2113) at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2607) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2562) at com.vmware.xenon.common.ServiceHost.lambda$processServiceStart$20(ServiceHost.java:2523) at com.vmware.xenon.common.Operation.lambda$nestCompletion$2(Operation.java:1248) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.processServiceStart(ServiceHost.java:2538) at com.vmware.xenon.common.ServiceHost.lambda$processServiceStart$19(ServiceHost.java:2482) at com.vmware.xenon.common.Operation.lambda$nestCompletion$2(Operation.java:1248) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.services.common.QueryTaskService.handleQueryCompletion(QueryTaskService.java:598) at com.vmware.xenon.services.common.QueryTaskService.lambda$forwardQueryToDocumentIndexService$5(QueryTaskService.java:481) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.services.common.LuceneDocumentIndexService.handleQueryTaskPatch(LuceneDocumentIndexService.java:699) at com.vmware.xenon.services.common.LuceneDocumentIndexService.handleQueryRequest(LuceneDocumentIndexService.java:596) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd: reserve resource failed: NO_SUCH_RESOURCE, Place failure, constraints cannot be satisfied for request: Resource(vm:Vm(id:ba595d9b-5680-4422-870e-b4a275c72ecd, flavor:cluster-other-vm, state:STOPPED, disks:[Disk(id:67bd2f06-5008-4f97-82c2-50b1120478ff, flavor:4e8a1d51542d23246ee60, persistent:false, new_disk:true, capacity_gb:15, image:DiskImage(id:aa9c52a4-701e-47d1-9b06-2cee5984e212, clone_type:COPY_ON_WRITE), flavor_info:Flavor(name:cluster-vm-disk, cost:[QuotaLineItem(key:ephemeral-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.flavor.cluster-vm-disk, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.cost, value:1.0, unit:COUNT), QuotaLineItem(key:ephemeral-disk.capacity, value:15.0, unit:GB)]))], flavor_info:Flavor(name:cluster-other-vm, cost:[QuotaLineItem(key:vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.flavor.cluster-other-vm, value:1.0, unit:COUNT), QuotaLineItem(key:vm.cpu, value:1.0, unit:COUNT), QuotaLineItem(key:vm.memory, value:4.0, unit:GB), QuotaLineItem(key:vm.cost, value:1.0, unit:COUNT)]), resource_constraints:[ResourceConstraint(type:NETWORK, values:[Public-10.90.52.0-23])], tenant_id:4e8a1d51542d2603093f9, project_id:4e8a1d51542d261291528)) ERROR [2016-12-05 09:36:52,011] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception ! com.vmware.photon.controller.api.frontend.exceptions.external.NoSuchResourceException: null ! at com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd.loadReservation(ResourceReserveStepCmd.java:441) ! at com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd.execute(ResourceReserveStepCmd.java:162) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:115) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,012] com.vmware.photon.controller.cloudstore.xenon.entity.TaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/tasks/4e8a1d51542e607107298] Patching service /photon/cloudstore/tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,013] com.vmware.photon.controller.api.frontend.commands.steps.ResourceReserveStepCmd: Resource reservation failed, mark entity ba595d9b-5680-4422-870e-b4a275c72ecd state as ERROR INFO [2016-12-05 09:36:52,014] com.vmware.photon.controller.cloudstore.xenon.entity.VmService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/vms/ba595d9b-5680-4422-870e-b4a275c72ecd] Patching service /photon/cloudstore/vms/ba595d9b-5680-4422-870e-b4a275c72ecd INFO [2016-12-05 09:36:52,022] com.vmware.photon.controller.cloudstore.xenon.entity.DiskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/disks/67bd2f06-5008-4f97-82c2-50b1120478ff] Patching DiskService /photon/cloudstore/disks/67bd2f06-5008-4f97-82c2-50b1120478ff INFO [2016-12-05 09:36:52,023] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Processed in 32ms ERROR [2016-12-05 09:36:52,023] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Command execution failed with exception ! com.vmware.photon.controller.api.frontend.exceptions.external.TaskNotCompletedException: Step "StepEntity{id=null, kind=step, state=ERROR, operation=RESERVE_RESOURCE, startedTime=Mon Dec 05 09:36:51 GMT 2016, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=Mon Dec 05 09:36:52 GMT 2016, sequence=0, error=StepErrorEntity{id=null, kind=step-error, code=NoSuchResource, message=null}}" did not complete. ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:118) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,024] com.vmware.photon.controller.cloudstore.xenon.entity.EntityLockService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/entity-locks/ba595d9b-5680-4422-870e-b4a275c72ecd] Handling PUT for EntityLockService /photon/cloudstore/entity-locks/ba595d9b-5680-4422-870e-b4a275c72ecd INFO [2016-12-05 09:36:52,024] com.vmware.photon.controller.api.frontend.backends.EntityLockXenonBackend: Entity Lock with taskId : 4e8a1d51542e607107298 and entityId : ba595d9b-5680-4422-870e-b4a275c72ecd has been cleared ERROR [2016-12-05 09:36:52,024] com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand: Task 4e8a1d51542e607107298 failed ! com.vmware.photon.controller.api.frontend.exceptions.external.TaskNotCompletedException: Step "StepEntity{id=null, kind=step, state=ERROR, operation=RESERVE_RESOURCE, startedTime=Mon Dec 05 09:36:51 GMT 2016, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=Mon Dec 05 09:36:52 GMT 2016, sequence=0, error=StepErrorEntity{id=null, kind=step-error, code=NoSuchResource, message=null}}" did not complete. ! at com.vmware.photon.controller.api.frontend.commands.tasks.TaskCommand.execute(TaskCommand.java:118) ! at com.vmware.photon.controller.api.frontend.commands.BaseCommand.run(BaseCommand.java:58) ! at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ! at java.util.concurrent.FutureTask.run(FutureTask.java:266) ! at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ! at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ! at java.lang.Thread.run(Thread.java:745) INFO [2016-12-05 09:36:52,025] com.vmware.photon.controller.cloudstore.xenon.entity.TaskService: [Req: 874de5af-9b86-4497-96af-22b9b5972e2e] [/photon/cloudstore/tasks/4e8a1d51542e607107298] Patching service /photon/cloudstore/tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,026] com.vmware.photon.controller.api.frontend.backends.TaskXenonBackend: Task TaskEntity{id=4e8a1d51542e607107298, kind=task, entityId=ba595d9b-5680-4422-870e-b4a275c72ecd, entityKind=vm, state=QUEUED, operation=CREATE_VM, startedTime=null, queuedTime=Mon Dec 05 09:36:51 GMT 2016, endTime=null} has been marked as ERROR INFO [2016-12-05 09:36:52,026] com.vmware.photon.controller.api.frontend.commands.BaseCommand: Processed in 39ms INFO [2016-12-05 09:36:52,116] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: tasks/4e8a1d51542e607107298 INFO [2016-12-05 09:36:52,254] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528 INFO [2016-12-05 09:36:52,388] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528/vms WARN [2016-12-05 09:36:52,390] com.vmware.photon.controller.common.xenon.XenonRestClient: send: WARN Action={POST}, StatusCode={202}, OperationId={1096113}, Uri={https://10.90.44.55:19000/core/query-tasks}, Referer={127.0.0.1}, jsonBody={NOT LOGGED} INFO [2016-12-05 09:36:52,627] com.vmware.photon.controller.api.frontend.auth.AuthFilter: Allow: API call: projects/4e8a1d51542d261291528/clusters WARN [2016-12-05 09:36:52,629] com.vmware.photon.controller.common.xenon.XenonRestClient: send: WARN Action={POST}, StatusCode={202}, OperationId={1096150}, Uri={https://10.90.44.55:19000/core/query-tasks}, Referer={127.0.0.1}, jsonBody={NOT LOGGED}

mwest44 commented 7 years ago

I find it strange to see this: Datastore(id:5840053b-429644f0-6ee3-d89d67203eb8, name:datastore42d, type:SHARED_VMFS, tags:[SHARED_VMFS])] Since this is a local datastore.

Without access to the system its difficult to tell if this is a configuration issue or not. My suggestion is to add the ALLOWED_DATASTORES tag to your yaml and deploy again. ALLOWED DATASTORES provides a whitelist of datastores that PC can use. Without this tag, PC will discover all datastores and use them for possible VM placement.

alancivita commented 7 years ago

ah ok i see, thanks, will give it a go shortly. Thanks

alancivita commented 7 years ago

Hi again :) well i've redeployed everything without luck. i've used this yml file and executed the deployment using the command line:

hosts:

Not sure if the ALLOWED_DATASTORES option i've used is correct, but the deployment went fine, but still getting the same errors.

Sorry to bother you ! and thanks for your time, but just trying to figure out possible scenarios. attached the log file from the controller photon-controller.log.zip

AlainRoy commented 7 years ago

What does photon image list show you? I wonder if the image wasn't replicated before you tried to create the VM.

alancivita commented 7 years ago

./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 9281ceb7-b873-4162-88fe-9231c7058414 photon-OS.ova READY 16777216098 EAGER 17% 25% b975421d-90c3-4f11-a975-80cc6d945b86 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%

and it is in this state for 30 mins at least

alancivita commented 7 years ago

maybe related to this error: ERROR [2016-12-05 17:37:42,402] com.vmware.photon.controller.housekeeper.xenon.ImageSeederService: [Req: 604727f0-37dd-453f-81f7-da63197fb599] [/photon/housekeeper/image-seeder/b7d0828c542ecbce48429] java.lang.RuntimeException: Image seeding failed: 1 image seeding succeeded, 3 image seeding failed or cancelled at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.processAwaitCompletion(ImageSeederService.java:388) at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.handleStartedStage(ImageSeederService.java:252) at com.vmware.photon.controller.housekeeper.xenon.ImageSeederService.handlePatch(ImageSeederService.java:109) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:408) at com.vmware.xenon.common.StatefulService.lambda$loadAndLinkState$7(StatefulService.java:1175) at com.vmware.xenon.common.Operation.lambda$nestCompletion$1(Operation.java:1233) at com.vmware.xenon.common.Operation.completeOrFail(Operation.java:1201) at com.vmware.xenon.common.Operation.complete(Operation.java:1105) at com.vmware.xenon.common.ServiceHost.loadServiceState(ServiceHost.java:2698) at com.vmware.xenon.common.StatefulService.loadAndLinkState(StatefulService.java:1177) at com.vmware.xenon.common.StatefulService.handleRequestLoadingAndLinkingState(StatefulService.java:465) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:349) at com.vmware.xenon.common.StatefulService.handleRequest(StatefulService.java:313) at com.vmware.xenon.common.ServiceHost.lambda$queueOrScheduleRequest$31(ServiceHost.java:3710) at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

AlainRoy commented 7 years ago

Can I assume that the photon-OS.ova is the file that you're using to create the VM? It looks like it's being replicated to all the hosts, but it's not there yet. I suspect that this is the underlying issue.

In the log I see this: Image seeding failed: 1 image seeding succeeded, 3 image seeding failed or cancelled

Just before that, I see: ERROR [2016-12-05 17:37:38,983] com.vmware.photon.controller.housekeeper.xenon.ImageHostToHostCopyService: [/photon/ housekeeper/image-host-to-host-copiers/b7d0828c542ecbce4e5d8] com.vmware.photon.controller.common.clients.exceptions .SystemErrorException: Could not connect to 10.90.233.198:8835: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

That's the core problem.

As part of setting up Photon Controller, we install new SSL certificates on ESXi. Sometimes this makes vCenter unhappy. Are you using vCenter for the same hosts that you're using Photon Controller? I wonder if there's a conflict, or if something just went wrong in the certificate installation.

mwest44 commented 7 years ago

I'm not sure if the datastore list is the problem, but the deployment yaml you used will not solve the problem if it is. Basically you have said that 42c,d,e,f can be used on .195 and .196. These are your MGMT hosts and you have not tagged them for VM placement. The Cloud hosts have no ALLOWED_DATASTORES tag, so will try to use all Datastores. Add the ALLOWED_DATASTORES tag to .197 and .198 hosts

alancivita commented 7 years ago

@AlainRoy Hi, i'm not using vCenter at the moment, just simple esxi hosts. @mwest44 Thanks, i see ! will try to redeploy again tomorrow morning, it's evening here. thanks Alan

AlainRoy commented 7 years ago

I'm looking around for someone who is better at debugging issues around our SSL certificates.

longzhou commented 7 years ago

To debug the SSL certificate issue, please help get:

  1. installer logs: in install VM, please get all files under /var/log/esxcloud/photon-controller-core, including script_logs sub-folder.
  2. host agent log: on ESXi host 10.90.233.195, please get /scratch/log/photon-controller-agent.log.

Thanks, Long

alancivita commented 7 years ago

Hi Long, please find attached the logs requested. bed time here, will back online tomorrow morning. thanks Alan

photon-controller-agent.log.zip photon-controller-core.zip

longzhou commented 7 years ago

Thanks, Alan. I could not find the root cause yet, please help gather some more data:

  1. output of "openssl s_client -showcerts -connect 10.90.233.198:443 </dev/null"
  2. ssh to 10.90.233.195, get output of "ls -l /etc/vmware/ssl".
  3. please run "photon image list" again tomorrow morning, and see if the periodic maintenance tasks can fix the image.
  4. latest photon-controller.log from 10.90.44.55.

Thanks, Long

alancivita commented 7 years ago

Hi Long, 1) see attached file (i ran the command both from the photon-controller and the esx host) 2) [root@hdesx42c:~] ls -l /etc/vmware/ssl/ total 116 -rw-r--r-- 1 root root 1188 Nov 23 09:35 69af5a5a.0 -rw-r--r-- 1 root root 649 Nov 24 10:24 69af5a5a.r0 -rw-r--r-- 1 root root 1188 Nov 23 09:35 82b12678.0 -rw-r--r-- 1 root root 649 Nov 24 10:24 82b12678.r0 -rw-r--r-- 1 root root 1188 Dec 5 14:31 castore.pem -r-------- 1 root root 82667 Dec 5 14:31 rui.bak -rw-r--r-- 1 root root 1168 Dec 5 14:31 rui.crt -rw------- 1 root root 1709 Dec 5 14:31 rui.key -rw-r--r-T 1 root root 0 Nov 2 00:49 vsanvp_castore.pem -rw-r--r-- 1 root root 64 Nov 30 12:51 vsanvp_secret

3) ./photon image list ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 9281ceb7-b873-4162-88fe-9231c7058414 photon-OS.ova READY 16777216098 EAGER 17% 25% b975421d-90c3-4f11-a975-80cc6d945b86 photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 33% 50%

4) see attached files photon-controller.log.zip openssl_output.txt.zip

Thanks Alan

longzhou commented 7 years ago

The issue is that CA certificate files (/etc/vmware/ssl/.0, .r0) on ESXi are not refreshed after the host joined lightwave domain. These files have timestamps from Nov 23/24, and I expect timestamp from Dec 5.

Could you help get some more data?

  1. what's the ESXi version? you can run "vmware -v" command.
  2. is lwsmd deamon running on ESXi? could you get output of "ps -c"?
  3. could you check other ESXi hosts to see if they have similiar problem?

Thanks, Long

alancivita commented 7 years ago

HI, all 4 esx hosts have version: VMware ESXi 6.0.0 build-4600944 all 4 hosts have lwsmd running ps -c|grep lwsmd 912765 912765 grep grep lwsmd 851177 851177 sh /bin/sh /sbin/watchdog.sh -s lwsmd /usr/lib/vmware/likewise/sbin/lwsmd ++group=likewise --syslog 851189 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851190 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851191 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851192 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851193 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851194 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851195 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851196 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851197 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851198 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851208 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851235 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851361 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog 851362 851189 lwsmd /usr/lib/vmware/likewise/sbin/lwsmd --syslog

yes all 4 hosts have timestamps from 23/24 November for .0 .r0 files.. let me know if there is anything else i can do.. these are test hosts so i'm free to do whatever i want.

longzhou commented 7 years ago

Thanks for your prompt response. I have forwarded the information to my colleagues who work on this component. Will get back to you soon. Thanks!

schellappan commented 7 years ago

Hi Alan,

Can you please run following commands and get the output for each?

openssl x509 -in 69af5a5a.0 -noout -text and openssl x509 -in 82b12678.0 -noout -text /usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text

Also please collect the support bundle by running vm-supprt

Thanks -Suresh

alancivita commented 7 years ago

Hi Suresh thanks for your reply, i ran the commands from only 1 of the esx hosts, let me know if you want me to run the command from each of them. 1) openssl x509 -in 69af5a5a.0 -noout -text [root@hdesx42e:/etc/vmware/ssl] openssl x509 -in 69af5a5a.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 2) openssl x509 -in 82b12678.0 -noout -text [root@hdesx42e:/etc/vmware/ssl] openssl x509 -in 82b12678.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 3) usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text [root@hdesx42e:/etc/vmware/ssl] /usr/lib/vmware/vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text Number of entries in store : 1 Alias : f4fa7b050b95f694967b6d449cefa1537e3b45e7 Entry type : Trusted Cert Certificate: Data: Version: 3 (0x2) Serial Number: d6:af:e9:0a:8d:02:06:43 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Validity Not Before: Nov 20 09:23:01 2016 GMT Not After : Nov 18 09:23:01 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.54 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:eb:d7:67:32:ea:f0:26:5f:d9:c0:28:cd:94: a8:10:83:7e:a6:6a:8b:30:74:3e:6e:dd:ae:95:d4: db:31:22:c7:85:69:f3:37:d9:4d:bd:26:5c:8f:96: 43:dd:16:42:f5:23:b2:5e:f8:32:70:3d:bd:ea:b9: 06:93:a9:c8:44:cd:28:0e:53:9c:99:79:20:f8:08: a2:c8:c3:46:aa:fe:f3:5a:0d:0f:7c:7c:f4:1f:d8: e0:c2:b3:e4:d8:93:fd:0c:4d:32:09:44:cd:ea:ad: c3:62:14:e5:9c:70:8b:ce:7c:2c:70:b7:fe:cf:b2: 5c:3f:f3:66:85:73:20:1e:cc:6b:be:96:82:07:25: c9:31:77:da:c1:73:87:b4:b9:da:b4:72:0f:a7:41: 11:2d:a7:52:02:cd:0f:98:b0:b4:9e:02:c1:36:d3: 1e:5f:fb:93:af:af:76:86:44:c0:83:08:31:8e:d4: 1e:27:08:08:18:78:d2:b6:69:02:cd:6f:2e:17:0d: e7:17:96:49:c2:f9:d1:4e:f9:4d:49:e4:6d:2f:eb: 9e:c7:7c:f7:35:d2:c7:63:f8:52:bc:56:b8:44:4d: fe:42:06:fd:25:35:f2:1f:63:2c:59:3a:01:48:61: 1c:77:67:b5:40:75:80:af:7b:79:de:3e:34:16:b9: b9:d1 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: DE:53:E4:4F:4D:1E:2E:2B:13:F5:8D:E2:F2:39:97:24:E6:89:C1:9E X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 80:8f:86:6e:19:8f:45:d9:f4:32:3d:11:45:7d:dd:23:94:81: f2:58:c4:3a:c0:62:03:7f:f6:04:71:23:53:6d:4e:db:3e:e5: db:79:04:b3:6d:0d:67:d1:84:37:0a:5e:cf:f0:f8:60:13:7e: ad:74:d2:2b:4f:4d:20:f7:1e:9d:d0:da:9d:56:8b:71:30:6a: ef:b7:17:12:d9:16:60:82:b6:ab:ab:18:d5:42:23:b9:9a:f3: ae:0f:6f:03:7a:79:a0:54:04:0a:0d:41:23:77:47:90:2a:d2: ee:2b:89:7a:47:0c:1b:ce:9f:59:5c:65:62:e3:9c:a7:cd:a7: d7:ad:f0:cb:7d:2a:88:e6:1c:02:53:d2:f7:0d:39:3d:5f:72: a6:42:6c:06:35:aa:4d:23:80:7f:08:37:2a:a9:c1:b3:3a:10: 19:81:59:da:58:32:3d:b6:8c:bb:d8:cb:60:94:11:cc:35:78: 01:c0:d5:3a:35:1f:65:4e:63:ca:b1:fe:26:75:aa:f7:b0:05: c8:e6:0a:2d:62:07:aa:2e:8b:ef:92:e4:67:f2:ec:a8:32:fa: ae:7a:f7:2f:8c:b6:0e:bf:36:6e:f0:53:02:98:ba:9d:8c:57: 88:94:37:45:b8:c4:b0:27:24:72:d7:bc:5f:c4:d7:b3:f0:8a: 29:b6:50:e5 the vm-support file is 35Mb and can't uploaded here.. i'll upload it on my gdrive and will provide a link shortly. I've generated vm-support only for 1 hosts let me know if you need for all of them. Many thanks Alan

alancivita commented 7 years ago

i've split the tgx in 4 zip files. thanks esx-hdesx42e-2016-12-06--19.57-001.zip esx-hdesx42e-2016-12-06--19.57-002.zip esx-hdesx42e-2016-12-06--19.57-003.zip esx-hdesx42e-2016-12-06--19.57-004.zip

schellappan commented 7 years ago

Thanks Alan,

Host certificate is signed by a root certificate from 10.90.44.54 and the root for that is not getting pulled from lightwave server. Endpoint certificate store also does not have the certificate in it's TRUSTED_ROOTS.

10.90.44.54 and 10.90.44.56 are they replicating each other?

Can you run /usr/lib/vmware/vmafd/bin/dir-cli nodes list to see what are the lightwave servers in the cluster?

alancivita commented 7 years ago

Hi, that's odd because 10.90.44.54 is an old ip address from a previous deployment. photon controller has ip 10.90.44.55 and lightwave 10.90.44.56

[root@hdesx42e:~] /usr/lib/vmware/vmafd/bin/dir-cli nodes list Enter password for administrator@esxcloud: dir-cli failed, error= Could not connect to VMware Directory Service via LDAP.

alancivita commented 7 years ago

the same command from the lightwave container provides: root [ /opt/vmware/bin ]# ./dir-cli nodes list Enter password for administrator@esxcloud: Node: ec-mgmt-10-90-233-196ccbef.esxcloud Type: PSC Site: Default-first-site

Node: ec-mgmt-10-90-233-1955a0d0 Type: Management

Node: hdesx42e Type: Management

Node: hdesx42f Type: Management

Node: hdesx42c Type: Management

Node: hdesx42d Type: Management

alancivita commented 7 years ago

Hi there, a quick update: i was able to get the right certificates, signed by 10.90.44.56 and not 10.90.44.54 which was an old ip from a previous deployment: First i've removed the .0 and .r0 certificates from /etc/vmware/ssl/ and then removed the afd.db from /etc/vmware/vmafd/db/ directory. i've done this because this was the only place referring to 10.90.44.54 [root@hdesx42c:/etc/vmware/vmafd] grep -r "10.90.44.54" * ��5afd.d#10.90.44.54Default-first-siteesxcloudXJ��� ��/afd.desxcloudX5b� 10.90.44.54esxcloud

Then i've redeploy photon-controller from scratch and now cerficates are correctly signed by 10.90.44.56: [root@hdesx42d:/etc/vmware/ssl] openssl x509 -in 3df40f5b.0 -noout -text WARNING: can't open config file: /usr/ssl/openssl.cnf Certificate: Data: Version: 3 (0x2) Serial Number: ed:e0:29:ba:25:df:57:96 Signature Algorithm: sha256WithRSAEncryption Issuer: CN=CA,DC=esxcloud, C=US, O=10.90.44.56 Validity Not Before: Dec 6 12:41:33 2016 GMT Not After : Dec 4 12:41:33 2026 GMT Subject: CN=CA,DC=esxcloud, C=US, O=10.90.44.56 Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c5:9b:6d:7e:4c:48:24:c9:21:23:92:d4:e5:40: 52:f5:24:05:a6:1d:32:e7:9a:60:a9:1f:90:04:fb: e5:5a:07:02:53:05:8d:ef:38:9d:6b:4e:6b:13:25: 98:bd:ed:50:92:fb:a4:68:3e:0a:d8:0c:c1:33:41: f6:ce:1a:4d:be:97:7c:68:b7:a0:7d:dd:58:0c:59: a4:39:f7:b1:97:97:0d:e2:a8:2e:11:9b:33:e4:f7: 73:f8:c4:66:67:ce:4c:9d:9e:ea:67:b0:0d:ff:2a: 31:db:87:54:ff:86:e7:aa:96:96:51:af:df:9a:0b: e0:e2:5d:03:36:cd:54:cb:01:e8:4b:6c:0a:01:ea: 44:87:38:e0:c9:4a:75:63:e0:ff:84:bc:b8:de:ef: 30:4b:71:17:18:73:cf:e4:08:16:1d:f6:1c:a7:d9: 8b:ef:46:71:7b:bf:91:3a:4a:7d:bd:7a:3b:73:f7: 44:8e:e1:cf:94:ea:8c:42:16:60:7a:50:da:2a:5e: 24:93:af:06:04:85:c6:81:7d:89:fa:a5:9d:40:a5: 96:1a:b3:49:3b:84:ed:f7:93:8d:9e:44:d9:bc:82: a1:66:59:13:d0:c7:0a:8b:21:4d:f7:19:a3:fb:dd: 36:77:2c:d7:e1:2a:46:f5:28:8c:ee:cd:5d:42:a1: b0:93 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Subject Key Identifier: 27:72:98:88:D8:7C:9B:F1:A9:10:47:71:75:58:74:09:96:02:45:D6 X509v3 Key Usage: critical Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 Signature Algorithm: sha256WithRSAEncryption 8f:95:53:c3:d4:cd:a9:ff:48:e8:b4:1d:7e:8f:c6:5a:32:b8: 60:38:e2:ab:d5:94:50:56:51:99:65:cd:e9:d1:cb:9e:ef:bd: 1e:91:3e:61:26:3b:53:26:8c:ea:15:77:46:69:d1:70:03:cd: 32:fa:80:79:43:28:f0:38:c9:f3:51:e4:23:bb:7f:c9:de:da: ba:40:81:e4:b2:59:47:9d:99:ee:30:b2:4f:40:9e:f9:a2:36: c2:23:e2:b5:4a:ba:5d:ab:92:9e:16:2b:d5:4d:ec:07:06:93: 56:44:3e:85:75:87:0f:64:b2:01:ba:92:1b:dc:d0:0b:a5:68: c7:e8:51:5d:65:0d:b5:8f:aa:3d:69:0f:83:ea:72:08:60:e9: bd:a5:e3:04:78:42:b3:5f:df:7d:a7:ed:b5:3f:29:67:1b:82: db:5f:f4:7d:40:04:43:02:63:bc:f6:42:a1:de:74:df:11:77: a3:75:02:8e:e6:e0:ea:45:38:53:96:f5:58:ae:c1:a7:26:9a: 7c:90:06:45:2c:e1:91:9a:20:d9:70:10:fa:d8:7c:87:9b:a5: bc:0d:bb:af:00:25:94:e7:29:07:7a:29:3f:c6:cd:c1:96:40: b8:27:29:1f:77:98:fa:b2:e6:fa:98:01:d4:b4:4a:f9:49:68: 95:29:9c:fc

Finally, i can see images being replicated across all the hosts! and can create VMs

ID Name State Size(Byte) Replication_type ReplicationProgress SeedingProgress 67a5f7d8-0502-4cee-bd0c-9762e0564c60 photon-OS.ova READY 16777216098 EAGER 100% 100% 73c07279-be38-48b0-92aa-4f0c7d4cb11a photon-management-vm-disk1.vmdk READY 41943040000 ON_DEMAND 100% 100%

I still have a copy of afd.db and certificates.

Alan

longzhou commented 7 years ago

Thanks for the update, Alan! I am glad it is working for you now.

Please send us the afd.db file and we can find out why certificates sync did not work.

Thanks, Long

alancivita commented 7 years ago

i've attached the file.. just one question, shouldn't this file be deleted by photon system destroy? or overwritten by a new deployment? thanks Alan

longzhou commented 7 years ago

Our design is to overwrite the certificates by new deployment. We need to investigate why it did not work. I had opened a bug.

alancivita commented 7 years ago

thanks for the explanations, just a note, before deleting the adf.db I've only deleted the certificates and redeployed, but the certs were recreated with 10.90.44.54 (the old ip); After that I've destroyed the environment, deleted the certs and the afa.db...and this worked fine.

longzhou commented 7 years ago

I did not find afd.db attached in your previous comment. Did I miss anything?

alancivita commented 7 years ago

sorry, the original extension was unsupported :) afd.db.orig.zip

longzhou commented 7 years ago

Great! thanks!