plumbis
commented
1 year ago Background
A 10 question survey focused on Crossplane documentation was conducted from November 28, 2002 through January 1, 2023. 24 survey responses were received.
The survey was anonymous, but participants were given the option of providing their email address to be contacted by Crossplane maintainers if they were okay with follow-up questions. These email address won't be shared unless a maintainer has a specific follow-up.
The survey was sent directly to 250 randomly selected Crossplane members on Slack and later posted on the landing page of docs.crossplane.io
Questions
The survey asked the following questions.
1. How long have you been using Crossplane?
- Less than 6 months
- Between 6 months and 1 year
- More than 1 year
2. Do you use Crossplane in production?
- Yes
- No
3. Which Provider(s) do you use?
Multiple choices were allowed
- AWS
- Azure
- GCP
- Helm
- Terraform
- Other
4. How satisfied are you with the Crossplane documentation?
- Unsatisfied
- Neither satisfied or unsatisfied
- Satisfied
5. How important are docs on the following topics to you?
Rate each between 1 and 3 points. 1 point being "Not Important" and 3 being "Very Important"
- Getting Started Guides
- Install and Upgrade
- Crossplane Components
- Crossplane Packages
- Reference Implementations
- Provider Configurations
- Integrating Crossplane with other software
- Migrating to Crossplane from another platform
- Troubleshooting
6. Crossplane docs need more content on...
this is a freeform answer
7. I prefer documentation organized by...
- Provider. Each provider has their own chapter on a topic like "authentication"
- Topic. Each topic, like "authentication" highlights one or more providers.
- Other
8. The Crossplane topic I've struggled with most is....
this is a freeform answer
9. Imma let you finish but the best docs of all time were....
this is a freeform answer
10. Anything else we should know?
this is a freeform answer
Responses
1. How long have you been using Crossplane? (n=24) |
Question | Responses | % |
|---|---|---|---|
| Less than 6 months | 18 | 75% | |
| Between 6 months and 1 year | 2 | 8.3% | |
| More than 1 year | 4 | 16.7% |
2. Do you use Crossplane in production? (n=24) |
Question | Responses | % |
|---|---|---|---|
| No | 16 | 66.7% | |
| Yes | 8 | 33.3% |
3. Which Provider(s) do you use? (n=24)Multiple choices were allowed |
Question | Responses | % |
|---|---|---|---|
| AWS | 19 | 41.3% | |
| Azure | 6 | 13% | |
| GCP | 7 | 15.2% | |
| Helm | 4 | 8.7% | |
| Terraform | 2 | 4.3% | |
| Other | 8 | 17.4% |
| Individual answers: | Participant | Answers |
|---|---|---|
| P24 | AWS, Azure, GCP, Helm, Terraform | |
| P23 | Other, planning to build a new provider | |
| P22 | Azure | |
| P21 | AWS | |
| P20 | AWS, k8s | |
| P19 | AWS | |
| P18 | AWS | |
| P17 | AWS, Azure, GCP, Terraform, SAP BTP | |
| P16 | AWS | |
| P15 | Custom, self-written | |
| P14 | GCP, Helm, k8s | |
| P13 | AWS | |
| P12 | AWS, GCP, Helm | |
| P11 | AWS | |
| P10 | Azure | |
| P9 | AWS, datadog-jet | |
| P8 | AWS | |
| P7 | AWS, Azure, Helm | |
| P6 | AWS, GCP | |
| P5 | AWS, k8s | |
| P4 | AWS, Azure, GCP | |
| P3 | AWS, GCP | |
| P2 | AWS | |
| P1 | AWS |
4. How satisfied are you with the Crossplane documentation? (n=24)
| Question | Responses | % |
|---|---|---|
| Unsatisfied | 10 | 41.7% |
| Neither satisfied or unsatisfied | 9 | 37.5% |
| Satisfied | 5 | 20.8% |
5. How important are docs on the following topics to you? (n=24)
Rate each between 1 and 3 points. 1 point being "Not Important" and 3 being "Very Important"
| Question | Total Score | Average Score |
|---|---|---|
| Getting Started Guides | 66 | 2.75 |
| Install and Upgrade | 65 | 2.71 |
| Crossplane Components | 66 | 2.75 |
| Crossplane Packages | 60 | 2.5 |
| Reference Implementations | 71 | 2.96 |
| Provider Configurations | 66 | 2.75 |
| Integrating Crossplane with other software | 52 | 2.17 |
| Migrating to Crossplane from another platform | 44 | 1.83 |
| Troubleshooting | 69 | 2.88 |
6. Crossplane docs need more content on... (n=19)
this is a freeform answer
- Creating a new provider from a CRUD API, explaining how Crossplane works to someone who is not familiar with Kubernetes but just needs to use it
- More examples on how to create XRDs for specific providers, e.g. Azure, AWS. It should be more explicit what you can do with crossplane and what you can’t do (e.g. wait for X to be created before Y, reference the outbound IP address of in a vnet) and how you can work around those issues (e.g. create multiple XRDs)
- E2E production grade examples. Getting started with compositions is still bit too abstract
- disaster recovery (trouble shooting more scenarios), patching
- Different use-cases
- multi-tenant setup, security tuning
- Control Plane Getting started, and a why for newbies
- examples & integrations with other tools
- if You really want to succeed make good docs about writing own providers, how to nicely use Your api etc. - crossplane is not a bad software, but it's still "too" open-source :P allow people easily add their logic
- Known limitations and pitfalls
- Examples on how to put things together and how to troubleshoot why it doesn't seem to be working.
- more examples, use cases, problem statement and how to solve it with crossplane, day 2 stuff security, observability, troubleshooting, reliability
- What to actually use and type into a yaml file, their definitions, expected uses. Great, I can install crossplane with your documentation, but past that your guides just say, create a configuration as if its magic and the verbiage is known. Maybe I just missed something super basic, but if so it highlights the confusion the site creates.
- How to initialize a provider, it's hard to find documentation around this
- We're trying to migrate from terraform. It has a lot of content about reference implementations, community discussions and best practices. Cloud providers have hundreds of services but crossplane mostly focus on how crossplane works and how to configure it. But terraform focus on how to create efs or eks cluster or rds cluster. I prefer the terraform approach.
- Developing/Contributing, Upjet needs focus
- Getting started. Specifically need more real world examples that developers can start with so they can start iterating sooner and see how you should architect your compositions. Without these examples it is too time consuming to get started and most people abandon the process. Things like platform-ref-gcp are great! (however, it uses the unbound crossplane which is confusing and a whole other problem to address).
- High availability/multi-cluster/multi-region deployments
- My biggest pitfall i have run into is when something needs order. Example: Creating an IAM Role and Policy with Tieing the policy to the role. I always get caught up in that and am sure with composition can help.
7. I prefer documentation organized by... (n=24) |
Question | Responses | % |
|---|---|---|---|
| Provider. Each provider has their own chapter on a topic like "authentication" | 16 | 66.7% | |
| Topic. Each topic, like "authentication" highlights one or more providers. | 7 | 29.2% | |
| Other | 1 | 4.2% |
- Other: What is relevant for developers, what is relevant to infrastructure teams
8. The Crossplane topic I've struggled with most is.... (n=21)
this is a freeform answer
- Compositions
- The overall relationship between Crossplane components: XRDs, XRs, claims, compositions, etc.
- Naming conventions, understanding how I should be writing compositions, complex topics like copying parameters between compositions (like when creating a cluster), getting started, how to integrate with common systems like vault for secret storage.
- Patching fields in compositions, implementing lateinit / referencers in providers
- I've developed couple of k8s operator in my previous job so I've expertise about how controllers work. It was easy for me to understand how crossplane works. But I've struggled with most are to explain the concepts to my team(cloud organization) and how to debug a problem.
- debugging while trying to create my own provider or using terrajet to do so
- rate limit in AWS. It is terrible. We are not able to recover a cluster fast enought because of all the rate limitting. We are actually moving out because of this and that state gets stored in k8s (pros and cons)
- Getting started. I tried to approach as a noob (which I am to CP)...yet I cannot for my life figure out how to create my own custom yaml deployment without just modifying some example I found.
- missing tutorials, docs, debugging, and compositions creation
- Writing a package in a way that mimics what I have been doing with terraform.
- One irritating issue was https://github.com/crossplane/crossplane/issues/3468
- composite, composition - why here is Xsomething why if I write provider, then I have to check cooperation between 3 resources to receive working example, makes sensu after You spend longer while, but it's not intuitive or simple
- The integration with External tools as Vault
- examples
- rbac-manager
- creating disaster recovery use cases and trying to test them all
- Creating custom compositions
- Creating XRDs was not difficult per se, but we are struggling to migrate our production IaC for Azure to crossplane because crossplane is lacking functionality and/or is not supposed to work in same way as Bicep/ARM.
- Getting an overview that is understandable to someone who is not familiar with Kubernetes
- How is this different ane why do things this way.
9. Imma let you finish but the best docs of all time were.... (n=14)
this is a freeform answer
- https://github.com/getify/You-Dont-Know-JS
- https://doc.rust-lang.org/book/
- I love the upjet videos on youtube, integrate them where it makes sense
- gobyexample
- Cloudfoundry opensource docs at https://docs.cloudfoundry.org/ and associated Vmware TAS (proprietary) https://docs.pivotal.io/application-service/3-0/concepts/overview.html is a great example for me of opensource docs that support proprietary offers. As an opensource user and contributor, I have enjoyed contributing opensource documentation that also helped improved the proprietary product. Associated repos https://github.com/cloudfoundry/docs-book-cloudfoundry https://github.com/cloudfoundry/?q=docs&type=all&language=&sort= https://github.com/pivotal-cf?q=docs&type=all&language=&sort=
- Nate Reids blog was the most useful documentation so far related to crossplane
- https://argo-cd.readthedocs.io/en/stable/
- terraform
- https://developer.hashicorp.com/boundary/docs
- youtube video of muvaf on provider code generation
- Love almost all the stripe docs: https://stripe.com/docs/api
- It would be too much to say I love their docs but I find the official Kubernetes docs well written and useful. https://kubernetes.io/docs/home/
10. Anything else we should know? (n=11)
this is a freeform answer
- Keep up the good work!
- Maybe a section on highly wanted features that have not yet been done (and maybe never will). Been following the discussion for example on "depends_on" type of thing in github issues
- real life examples, code snippets etc. - I have to integrate 300 various softwares, update 500 ticekt during daily basics, I don't have time to go through Your codebase to learn how things work
- More doc content should end up in the crossplane CRDs, possibly with permalinks to doc website related sections. This way, developers can discover related documentation sections from their IDE as they code XRD, Composition etc... Reference compositions such as https://github.com/upbound/platform-ref-gcp/tree/main would be quite useful for the opensource version
- All the examples! I think terraform's documentation is fairly decent both to get started and for ongoing development. I just copy paste what is there and then modify the defaults to meet what I need.
- The new font makes things too crowded and looks like something from early days of the internet.
- is difficult to find the getting started, then after a very high level getting started, there are no more tutorial docs for the next level
- The documentation is very sporadic, some on the site, some it github, some buried in the API spec...there is no source of truth and one place to go.
- In the docs, in meetings, kubecon, YouTube videos, the focus is on high level things and core concepts, ie you can compose things. This is very clear. There needs to be more focus on how. How do I get started? How do I create compositions that are maintainable? How do I use crossplane with Flux, ArgoCD, Vault, or other common technologies that my company may require? How do I troubleshoot crossplane when things go wrong, like something wasn't deleted or something is stuck? This may sound like a rant, but the addition of the unbound crossplane version and marketplace has added a lot of confusion. I love the idea of the Marketplace, but it's using a different version of crossplane in its examples. Even when you do work out that they are the compatible versions of crossplane and try to just replace API versions from unbound API to crossplane-contrib APIs you will find not everything is the same. For example "ProjectIAMMember" is missing from the crossplane contrib version and makes the platform-ref-gcp unusable as a getting started example unless you use the unbound crossplane version. I know upbound is focused on building out their platform for platforms, but don't cannibalize the crossplane open source experience. Try to keep that experience first class. You all are so close, crossplane is going to blow away Hashicorp and others. Thank you all for what you build!
- I would love to see more docs showing real-world implementations of Crossplane, especially those in production environments.
Take Aways
Unfortunately the sample size was lower than hoped. Based on the Slack membership a 90% +/- 5% margin of error would require n≈250. With n=24 our margin of error is ~20%. There are trends in this data and it shouldn't be ignored, but it's important to be aware of the caveat that comes with such a low response rate.
- Most users are new and not yet in production
- Provider AWS is the most commonly used provider
- Provider k8s was mentioned by 3 respondents, making it more used than Provider Terraform
- Provider GCP was always used with another cloud provider (primarily AWS)
- Only 20% of respondents are satisfied with the documentation.
- Guides on migrating to Crossplane are considered unimportant.
- Getting Started, Troubleshooting and Reference Implementations are considered most important.
- Users are looking for more practical examples.
- Discoverability of content seems to be a challenge. Multiple requests for content that exists today and complaints that docs are not centralized into a single source of truth.
- Users express confusion/frustration around the relationship between components and how to do it themselves.
- Providing unique documentation for platform teams and developer teams is in both CloudFoundry and ArgoCD documentation that was called out.
Actions
- Search was recently implemented with #226 that will assist in discoverability
- A large reorganization effort, focused on deduplicating and consolidating information has been proposed in #254
- New "Introduction to Crossplane" section draft exists in #247
- New quick starts were rejected in crossplane/crossplane #3396, but using them to expand and cover use of Compositions and Claims is a top docs priority. This shows evidence of their importance and the need to approach "getting started" in a very step-by-step manner.
- We need to engage the community to bring in examples of more comprehensive use cases. Upbound has a collection of public Reference Platform Implementations that would be good first candidates.
- Docs should include "expected behavior" outputs and, where possible, some common problems/troubleshooting examples.
Related to #251 . The results should be organized and shared on the docs site and with the community.