yaml: unknown anchor 'cc_install_buildpacks' referenced

[roninby]

Hello,

I'd like to test cf on a vsphere cluster using cf-deployment. The deploy failed at the very beginning as below:

[gcps@ansiblehost cf-deployment]$ bosh2 -e bosh-1 -d cf-1 deploy cf-1/cf-deployment.cf-1.yml \
> -o operations/scale-to-one-az.yml \
> --vars-store cf-1/deployment-vars.yml \
> -v system_domain=cf.gcps.lab
Using environment '10.10.0.6' as client 'admin'

Using deployment 'cf-1'

Evaluating manifest:
  yaml: unknown anchor 'cc_install_buildpacks' referenced

Exit code 1

I guess there're some misconfiguration in my cf-deployment.yml in connection with special characters "&" and "*", but I cannot figure out how to fix it. Is there any documentation about it? Or is there any sample cf-deployment.yml for vsphere environment?

From cf-deployment.yml:

        install_buildpacks: &cc_install_buildpacks
        - name: binary_buildpack
          package: binary-buildpack
        - name: dotnet_core_buildpack
          package: dotnet-core-buildpack

       ...
       ...

      cc:
        db_encryption_key: "((cc_db_encryption_key))"
        default_to_diego_backend: true
        install_buildpacks: *cc_install_buildpacks
        default_running_security_groups: *cc_default_running_security_groups
        default_staging_security_groups: *cc_default_staging_security_groups
        security_group_definitions: *cc_security_group_definitions

Thanks in advance.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/149345155

The labels on this github issue will be updated when the story is started.

[roninby]

I just found there were some lines missing in my cf-deployment.yml. After it's recreated, the problem was gone. Sorry for the trouble.

[dsabeti]

Hey @roninby, no worries. How did those lines go missing from cf-deployment.yml? Are you editing the manifest somehow?

Also, I have an unrelated question because you're deploying to vsphere. How did you build the cloud-config for your deployment? I'm asking because we want to provide instructions for all vsphere deployers of cf-deployment, and your input could be very valuable.

[roninby]

Hi, dsabeti.

Yes, I falsely deleted some lines in the manifest, it's just the part include links.

I know operations file is good, but it need a lot of background knowledge and skills like ERB, spiff, cloud properties, etc. As a newcomer to CF, the simplest way to me is to modify the config file directly. By the way, I'm still struggling with manifest syntax, like the link concept of consume/provider, is there any good reference on this?

For cloud-config, it's quite good to keep manifest IaaS agnostic in bosh v2. I tried to minimize the consumed resources, and some good references do help a lot. These practical samples make the configuration items more understandable. https://brianragazzi.wordpress.com/tag/bosh-v2/ http://engineering.pivotal.io/post/nsx_with_bosh/

I have to say it's quite painful to learn the manifest from the beginning, unfamiliar with the syntax of manifest, not clear about details on CF infrastructure integration with Cloud, etc. I mean the bosh guide is quite cool, but it's not enough to build up a application infrastructure. The problem possibly comes from lack of guidance, especially on bosh v2. Actually, I'm now stuck with cf-mysql-deployment. So, complete scenarios and procedures will be indeed very helpful, just like VMware Hands on Lab with detailed procedure on certain scenarios. A good example on PCF is: http://pcf-ford-workshop.cfapps.io/2016/03/15/lab--build-and-deploy-apps-on-pcf/

Glad to talk with you here, pls let me know if there's anything else you want to know.

Cheers.

[dsabeti]

Lots of good questions and feedback here. One by one:

• I totally understand that there's a learning curve for understanding operations files (I call them ops-files for short). However, you shouldn't need any ERB or spiff knowledge to understand ops-files -- they replace spiff tooling for manifests. This guide should help you learn the basics.
• The same thing goes for links. You can find some official documentation here to get started, but there's also an interesting discussion on this gist about good practices for links. It's a bit of a deep dive, but you might find it helpful.
• As for manifest syntax, I'd love to get some specific thoughts from you about what made sense and what was confusing. Our goal for cf-deployment is for it to be easily understood. Would you be willing jump on a call some time to get feedback?

[roninby]

@dsabeti , Thanks for the info, it really do help with understanding the manifest structure.

I totally agree with you. Maybe the learning curve is coming from that the manifest (mostly in the job part) is not much intuitive: From the job description, I have no idea about which file is modified, and how it is applied, that's different from other tools like Ansible. It doesn't mean Bosh is not good, on the contrary the abstraction is efficient and powerful, just need more time to understand.

In my view, one half of the confusion coming from Bosh/CF infrastructure, the connections between components decide which properties should be provided. The concept is documented well, but practices are still needed to understand the details; The other half is related with variables/properties/spec/link definition. For example:

• What kind of data should be stored in spec/properties/metadata/variables? What's the difference?
• How to list/check all the available properties of a job? Where the properties are written to?
• Understand the sub layers of a property. For example, where to insert "ssl_enabled: false" under job: cf-mysql-broker?
• How to check all cloud_properties for vsphere?

I guess the answers are behind release. So, if the learning curve is possibly not inevitable, then more documents on release mechanism and more App use case deployment samples will do a lot help.

Thanks again.

[dsabeti]

The answer to most of your questions is that the releases decide how to manage their properties.

• All available properties are defined in "spec" files in a release; here's an example for the MySQL broker. If a property is not defined in that file, then BOSH will ignore the property even if the deployer provides it.
• The jobs that are defined in the releases decide how the properties get used. For example, the ssl_enabled property you were asking about gets used in this file to decide to use http or https.
• The best way to ask questions about job properties is to contact the release authors. Most of them have a channel in the Cloud Foundry Slack, but you can also open GitHub issues on their repos.

Cloud properties are different, because they are fundamentally tied to the IaaS you choose. In v2 manifests, the cloud_properties should mostly be referenced in the cloud-config rather than the manifest, and there are three ways to get help for cloud_properties/cloud-config questions:

1. bosh.io docs. For example, the vSphere docs live here, and explain a lot of the different configuration options for cloud_properties on vsphere.
2. CPI repositories. Each IaaS has a related CPI -- Cloud Provider Interface -- whose repos live on GitHub. Here's the vsphere CPI. Again, you can open github issues.
3. Slack channels. Some CPI's have their own channels in the Cloud Foundry slack and some don't. If you can't find what you're looking for, you can jump into the #bosh channel, and they'll help you find someone who can help.

Hope that helps!

[roninby]

@dsabeti ,

The guidelines are awesome! It may take me days to figure all of these out by myself. The linkage behind is more and more clear. Thanks for your kindness！

I've just built up a small CF+Mysql environment, gone through all the demo procedures.

Next, to make it more practical, more external services are being considered, like Concourse, Artifactory, Jenkins, etc. and Active Directory integration as well. They are all necessary parts in a on-premise PaaS platform. I noticed CloudBee and JFrog tiles are EOA in PCF. Is there alternates for CF users? Any advice on these services?

I'm not sure if I'm asking a lot here. If it's not related with the original topic, pls feel free to stop me. :-)

Best Regards

[dsabeti]

Yeah, I'd say most of this is a bit out of scope for this repo. Concourse is open source, and you can always reach to the team in their repo. Otherwise, most of what you mentioned are Pivotal-specific products, so you'd need to reach out to Pivotal for help with those.

[roninby]

@dsabeti

That's fine, you've helped a lot. Your help is greatly appreciated.

Best Regards