search

pivotal-cf / om

General command line utility for working with VMware Tanzu Operations Manager
Apache License 2.0
135 stars 104 forks source link

`om staged-config` may not generate all configured elements #364

Closed brightzheng100 closed 5 years ago

brightzheng100 commented 5 years ago

While using om staged-config for some products, like newrelic-broker, some dynamically generated elements (e.g. the plans) couldn't be generated properly -- some with null values and some were simply missing.

ENV:

The one generated:

product-name: newrelic-broker
product-properties:
  .properties.allow_paid_service_plans:
    value: false
  .properties.apply_open_security_group:
    value: false
  .properties.newrelic_broker_enable_global_access_to_plans:
    value: true
  .properties.nrplans:
    value:
    - insightsInsertKey: null
      orgs: null
      planOldGuid: null
      rpmAccountId: null
  .properties.org:
    value: NewRelicServiceBroker-service-org
  .properties.space:
    value: NewRelicServiceBroker-service-space

The right config (after manually detecting and adding up):

product-name: newrelic-broker
product-properties:
  .properties.allow_paid_service_plans:
    value: false
  .properties.apply_open_security_group:
    value: false
  .properties.newrelic_broker_enable_global_access_to_plans:
    value: true
  .properties.nrplans:
    value:
    - planName: nrtrialplan
      planDescription: testplan
      licenseKey: 0b10d7fb14cd4decdea417adad093eacee7f9466
      insightsInsertKey: ""
      orgs: ""
      planOldGuid: ""
      rpmAccountId: ""
 .properties.org:
    value: NewRelicServiceBroker-service-org
  .properties.space:
    value: NewRelicServiceBroker-service-space

So you can see the .properties.nrplans part is wrongly generated.

cf-gitbot commented 5 years ago

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

kcboyle commented 5 years ago

There was a discussion about this in private channels. The correspondence is below:

brightzheng100 ENV:

jtarchie Were you expecting other values besides the null values? if so, what property name and the expected value.

brightzheng100 it's kind of dynamic plans there but the plan's elements, like plan_name, couldn't be exported out -- so if we unstage -> stage -> configure product again, the data would be lost Now the workaround was to detect the elements and values and manually compile the missing parts in generated config file @jtarchie (edited)

jtarchie Are you sure the properties are not there? They could've also been migrated by the tile into a new format. It looks like newrelic broker v1.2 has a migration in it to transform the data. It looks like the new property name is planName.

brightzheng100 yes planName -- at this point my memory was wrong. But the issue is 100% sure as of now -- you may have a try. the reason I found this again is that what ever I changed the value in the plan, I generated the config which would be automatically checked in to Git, but logs told me that nothing was changed -- so I dig more into it and found this issue

@brightzheng100 was there anything further you wanted to address on this issue, or are we good to close?

jtarchie commented 5 years ago

@brightzheng100 are you able to provide us a om --trace staged-config. It would be great to have the API payloads that are returned from OpsManager.

brightzheng100 commented 5 years ago

This is fully reproducible so you can also try it out.

As per @jtarchie 's request:

# om -e env/sb/env/env.yml --trace staged-config -p newrelic-broker

GET /api/v0/staged/products HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:32 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: 7849bd34-fc68-4adc-a07e-f4b29517f318

X-Runtime: 0.255742

X-Xss-Protection: 1; mode=block

664

[{"installation_name":"p-bosh","guid":"p-bosh-c07757076f3dd98a419c","type":"p-bosh","product_version":"2.3-build.258"},{"installation_name":"cf-7eb3135223205a72d4d8","guid":"cf-7eb3135223205a72d4d8","type":"cf","product_version":"2.3.8"},{"installation_name":"pivotal-mysql-1e01891d063361b2ee15","guid":"pivotal-mysql-1e01891d063361b2ee15","type":"pivotal-mysql","product_version":"2.5.3-build.7"},{"installation_name":"p-redis-370648d20721ebe8cfc0","guid":"p-redis-370648d20721ebe8cfc0","type":"p-redis","product_version":"1.14.4"},{"installation_name":"p-rabbitmq-96cdc28d035cc1d3a434","guid":"p-rabbitmq-96cdc28d035cc1d3a434","type":"p-rabbitmq","product_version":"1.14.6"},{"installation_name":"p-spring-cloud-services-80e5d9c9c08eac825d1c","guid":"p-spring-cloud-services-80e5d9c9c08eac825d1c","type":"p-spring-cloud-services","product_version":"2.0.5"},{"installation_name":"p-scheduler-66a7cc32932c9f85a940","guid":"p-scheduler-66a7cc32932c9f85a940","type":"p-scheduler","product_version":"1.2.23"},{"installation_name":"apmPostgres-414605d69604f0ed14c7","guid":"apmPostgres-414605d69604f0ed14c7","type":"apmPostgres","product_version":"1.6.0-build.41"},{"installation_name":"newrelic-broker-221cc28ca55e8f7c130d","guid":"newrelic-broker-221cc28ca55e8f7c130d","type":"newrelic-broker","product_version":"1.12.20"},{"installation_name":"p-event-alerts-bc5fb82a0e072ec9f0b3","guid":"p-event-alerts-bc5fb82a0e072ec9f0b3","type":"p-event-alerts","product_version":"1.2.6"},{"installation_name":"p-healthwatch-03254288187128ae5905","guid":"p-healthwatch-03254288187128ae5905","type":"p-healthwatch","product_version":"1.4.4-build.1"}]

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/properties HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:32 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: c3feea51-c950-4e96-991d-01b6e8595117

X-Runtime: 0.226318

X-Xss-Protection: 1; mode=block

ccb

{"properties":{".properties.org":{"type":"string","configurable":true,"credential":false,"value":"NewRelicServiceBroker-service-org","optional":false},".properties.space":{"type":"string","configurable":true,"credential":false,"value":"NewRelicServiceBroker-service-space","optional":false},".properties.apply_open_security_group":{"type":"boolean","configurable":true,"credential":false,"value":false,"optional":false},".properties.allow_paid_service_plans":{"type":"boolean","configurable":true,"credential":false,"value":false,"optional":false},".properties.newrelic_broker_enable_global_access_to_plans":{"type":"boolean","configurable":true,"credential":false,"value":true,"optional":false},".properties.tile_version":{"type":"string","configurable":false,"credential":false,"value":"1.12.20","optional":false},".properties.tile_previous_version":{"type":"string","configurable":false,"credential":false,"value":"1.12.8","optional":false},".properties.create_open_security_group":{"type":"boolean","configurable":false,"credential":false,"value":false,"optional":false},".properties.servicebroker_serviceid":{"type":"string","configurable":false,"credential":false,"value":"xxxxxx","optional":false},".properties.nrplans":{"type":"collection","configurable":true,"credential":false,"value":[{"guid":{"type":"uuid","configurable":false,"credential":false,"value":"xxxx","optional":false},"planName":{"type":"string","configurable":false,"credential":false,"value":"nrtrialplan","optional":false},"planDescription":{"type":"string","configurable":false,"credential":false,"value":"testplan","optional":false},"licenseKey":{"type":"string","configurable":false,"credential":false,"value":"0b10d7fb14cd4decdea417adad093eacee7f9466","optional":false},"oldPlan":{"type":"boolean","configurable":false,"credential":false,"value":false,"optional":false},"planOldGuid":{"type":"string","configurable":true,"credential":false,"value":null,"optional":true},"rpmAccountId":{"type":"string","configurable":true,"credential":false,"value":null,"optional":true},"insightsInsertKey":{"type":"string","configurable":true,"credential":false,"value":null,"optional":true},"orgs":{"type":"string","configurable":true,"credential":false,"value":null,"optional":true}}],"optional":false},".properties.org_quota":{"type":"integer","configurable":false,"credential":false,"value":4096,"optional":false},".properties.memory":{"type":"integer","configurable":false,"credential":false,"value":1024,"optional":false},".deploy-all.vm_credentials":{"type":"salted_credentials","configurable":false,"credential":true,"value":{"identity":"vcap","salt":"***","password":"***"},"optional":false},".deploy-all.app_credentials":{"type":"salted_credentials","configurable":false,"credential":true,"value":{"identity":"18e24c7a0ef21fa8","salt":"***","password":"***"},"optional":false},".delete-all.vm_credentials":{"type":"salted_credentials","configurable":false,"credential":true,"value":{"identity":"vcap","salt":"***","password":"***"},"optional":false},".delete-all.app_credentials":{"type":"salted_credentials","configurable":false,"credential":true,"value":{"identity":"3bb601ef53a94791","salt":"***","password":"***"},"optional":false}}}

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/networks_and_azs HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:33 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: 54825858-168d-4500-936e-e0de16690600

X-Runtime: 0.510294

X-Xss-Protection: 1; mode=block

8d

{"networks_and_azs":{"singleton_availability_zone":{"name":"AZ1"},"other_availability_zones":[{"name":"AZ1"}],"network":{"name":"Services"}}}

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/jobs HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:33 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: 0e68a987-3ab2-4924-b89f-10ec44418051

X-Runtime: 0.235995

X-Xss-Protection: 1; mode=block

88

{"jobs":[{"name":"deploy-all","guid":"deploy-all-d0c252287ca3d515f3ac"},{"name":"delete-all","guid":"delete-all-93ff4bcb16ae1475c8d4"}]}

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/jobs/deploy-all-d0c252287ca3d515f3ac/resource_config HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:34 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: dd47b7e7-faeb-45c7-b2c8-c40d58bd433e

X-Runtime: 0.200821

X-Xss-Protection: 1; mode=block

af

{"instance_type":{"id":"automatic"},"instances":"automatic","nsx_security_groups":null,"nsx_lbs":[],"additional_vm_extensions":[],"swap_as_percent_of_memory_size":"automatic"}

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/jobs/delete-all-93ff4bcb16ae1475c8d4/resource_config HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:35 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: d3489091-4ffc-4c83-a2b6-2f99c0f10ba8

X-Runtime: 0.621844

X-Xss-Protection: 1; mode=block

af

{"instance_type":{"id":"automatic"},"instances":"automatic","nsx_security_groups":null,"nsx_lbs":[],"additional_vm_extensions":[],"swap_as_percent_of_memory_size":"automatic"}

0

GET /api/v0/staged/products/newrelic-broker-221cc28ca55e8f7c130d/errands HTTP/1.1

Content-Type: application/json

HTTP/1.1 200 OK

Transfer-Encoding: chunked

Cache-Control: no-cache, no-store

Connection: keep-alive

Content-Type: application/json; charset=utf-8

Date: Thu, 16 May 2019 06:12:35 GMT

Expires: Fri, 01 Jan 1990 00:00:00 GMT

Pragma: no-cache

Server: Ops Manager

Strict-Transport-Security: max-age=15552000; includeSubDomains

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-Request-Id: a408e5f2-d2c5-49ab-a940-92c0fb9e489c

X-Runtime: 0.624893

X-Xss-Protection: 1; mode=block

89

{"errands":[{"name":"deploy-all","label":"deploy-all","post_deploy":false},{"name":"delete-all","label":"delete-all","pre_delete":true}]}

0

product-name: newrelic-broker

product-properties:

  .properties.allow_paid_service_plans:

    value: false

  .properties.apply_open_security_group:

    value: false

  .properties.newrelic_broker_enable_global_access_to_plans:

    value: true

  .properties.nrplans:

    value:

    - insightsInsertKey: null

      orgs: null

      planOldGuid: null

      rpmAccountId: null

  .properties.org:

    value: NewRelicServiceBroker-service-org

  .properties.space:

    value: NewRelicServiceBroker-service-space

network-properties:

  network:

    name: Services

  other_availability_zones:

  - name: AZ1

  singleton_availability_zone:

    name: AZ1

resource-config:

  delete-all:

    instances: automatic

    instance_type:

      id: automatic

  deploy-all:

    instances: automatic

    instance_type:

      id: automatic

errand-config:

  delete-all:

    pre-delete-state: true

  deploy-all:

    post-deploy-state: false
geofffranks commented 5 years ago

Does it seem like this would apply to things like customized sidebar links for apps manager in the PAS tile? We have them set in the tile, but can find no reference to them after running om staged-config. Not sure if this should go here, or in a separate issue for om.

jpmcb commented 5 years ago

Hi @geofffranks, this issue appears to be around a property changing from plan_name to planName in the config which caused some checked in property mappings to be incorrect.

Unless you are experiencing the same problem, I would recommend opening a new issue so that we can fully capture and keep eyes on what you're experiencing.

kcboyle commented 5 years ago

@brightzheng100 thank you for the context!

I'd like to close this issue, as it looks like we were able to solve the problem using other channels. Is this okay with you?

brightzheng100 commented 5 years ago

@kcboyle, the solution / workaround?

kcboyle commented 5 years ago

@brightzheng100 From this response: https://github.com/pivotal-cf/om/issues/364#issuecomment-492441231

It looked like the solution was to change plan_name to planName. Did that not work?

jtarchie commented 5 years ago

@brightzheng100, in the API output that you have above. The meta information for planName get's returned as ..."planName":{"type":"string","configurable":false,"credential":false,"value":"nrtrialplan","optional":false}....

This shows that the value is not settable via the API or a config file. It does have a value, but that value could be being set programmatically via the tile, not directly by the user.

Since configurable is false the command filters it out from the staged-config.

jtarchie commented 5 years ago

@brightzheng100, we were able to determine the issue. We wrote up a similar exploration in this comment. We are awaiting on the OpsManager team for the resolution.

Thanks for being patient with us while we hunted down the issue.

nbconklin commented 5 years ago

Hi there - we've discussed with the OpsManager team and they are planning to backport the API fix from 2.3-2.7. Stay tuned for the OpsManager release notes for more information. Closing this issue. Thanks for reporting!