Closed caniszczyk closed 4 years ago
Hey there!
On today's SC call we decided that the best approach would be for @caniszczyk and @LawrenceHecht to finish their flurry of changes and, once the survey is no longer a moving target, for one of them to create a PR that the SC can review. That PR would then become the final, approved version of the survey.
Here is the pull request for people to review. https://github.com/todogroup/survey/pull/63. @caniszczyk or @vmbrasseur how do we go about getting sign off?
I'll do a final look at things tonight before I ask the SC to sign off starting tomorrow.
We're almost there, thanks
On Thu, Apr 16, 2020 at 4:47 PM Lawrence Hecht notifications@github.com wrote:
Here is the pull request for people to review. #63 https://github.com/todogroup/survey/pull/63. @caniszczyk https://github.com/caniszczyk or @vmbrasseur https://github.com/vmbrasseur how do we go about getting sign off?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/todogroup/survey/issues/62#issuecomment-614914018, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPSINVAXU3ZIDPVA23UKLRM54ILANCNFSM4MH7QQ2A .
-- Cheers,
Chris Aniszczyk http://aniszczyk.org +1 512 961 6719
OK, survey questions are here for review: https://github.com/todogroup/survey/blob/master/2020/questions.md
Question 9 needs an answer which includes both formal and informal; a number of us have both formal (assigned roles) and informal (volunteers, champions, etc.) in our overall programs. It's very common.
Question 13's answer '0 (no dedicated staff yet)' conflicts with answers of a purely informal program; orgs with informal programs should supply a non-zero answer even though the staff is not dedicated.
Question 21 should offer the same additional answer as requested for Question 9.
How are companies selected for question 32's answers?
@kpfleming, here are some responses:
-Here is text explaining how they were picked: "We asked respondents this question about 11 TODO Group members, which represent a broad cross-section of technology companies. All TODO Group members have a significant investment in open source and many have dedicated open source programs to set policies and encourage contributions."
I like the second option for Question 9, and agree then that Question 13 would follow on only if Question 9 is answered "yes". I do think there could be some value in asking how many additional people are involved in informal roles, but it will be hard to ask the question in such a way to gather only people who have a responsibility in the program, and aren't just people who hang out in chats/lists. In my case I have ~10 people who are volunteers but who have primary responsibility for reviewing outbound contribution requests.
For question 32 - companies included; in prior surveys they were selected from the TODO group member companies, with an attempt to select a "like" industry cohort - in this case "tech" companies. Is this true again this time around? If Oracle is included, are they a TODO group member? We need a defensible selection criterion
@SuzanneA300 correct -- we took Oracle out of the list because we couldn't easily come up with a defensible criterion.
Ah - I see that Uber is now substituted. Thanks.
Question 43: suggest that you add the LF ACT Initiative (automated compliance tooling)
Question 46: add innovation speed as a benefit ?
@SuzanneA300 for Question 43, this is how we can update the choices:
Regarding question 32: there are a lot of TODO Group members who are 'tech' companies, if that is defined by the current list. I would include at least these:
As you can see, it's going to be hard to have a reasonably-sized list and also have defensible criteria for who is included in the list; even if you tried to limit the list to 'large' companies (by market value or number of employees) you'd only remove a few.
Question 5 - Recommend changing the phrase "spare time" to "personal time." No one has spare time.
Question 7 - I understand why we are asking this, but I'd like to see us iterate on the language, to frame it as more of a positive and to remove the subtle time-bound nature of the question (by the time the survey is released, I expect most companies will already have done some kind of reevaluation, which makes this sort of a future-tense question about past events). Recommendation below:
@kpfleming 1) The phrase "represent a broad cross-section of large technology companies" seems defensible. Perhaps if Tencent were added, then that would represent a large Chinese non-tech.
Question 9 - How do folks feel about framing this by asking them to choose a description?
"How would you describe the structure of your open source program/initiative?"
Question 32 - As a data point, of the 11 companies in that list of TODO group members, 1-3 of them are not what I would describe as active participants.
+1 to the call for a defensible selection criteria. A consideration here is that when measuring things like reach and share of voice, there is some credence put on proximity mentions (being mentioned in the same sentence as your peers). I don't have a good recommendation here, but I wonder what results we would see if we asked this as a two part, free form question
"What companies would you consider to be good open source community members?"
"What companies would you consider to be bad open source community members?'
In previous years I found the company list for question 32 to be non-inclusive and not understandable, and I don't think it's gotten better. The general population doesn't consider Comcast to be a 'tech' company any more than they consider Bloomberg to be a 'tech' company. The list of TODO Group members who are also 'large technology companies' is much longer than what is included in the survey, and at least some of the missing companies have contributed as much (or more) to the open source community as companies who are on the list. There are also a number of companies who should be on this list but are not TODO Group members (HashiCorp comes to mind, I'm sure I could easily find 6 more without much effort). When I've filled out this survey in the past, the list of companies read as if it was a 'sponsor list' because it was so short; I know it's not, the sponsors of this survey are well indicated, but I've completed enough analyst/research surveys in the past to get that impression from this question.
In fact only one of the companies represented by the current TODO Group Steering Committee are in this list (AWS). The remainder are GitHub, Bloomberg, Indeed, Spotify and Juniper Networks.
In any case, at a minimum VMware needs to be changed to Pivotal / VMware, just like IBM / Red Hat.
Question 36 - It seems weird to have both the LF and an LF Umbrella Foundation in this list
Question 39 - I would LOVE to see this paired with "How large is your company's engineering organization" and "How many employees does your company have"
Also recommend asking how many employees contribute open source, rather than focusing only on developers. Season of Docs would be an argument for broadening the language here.
Questions 43 and 44 - Can you clearly articulate the difference here between a Methodology/Initiative" and a Tool? ClearlyDefined isn't so clearly one or the other (ironically)
Question 44 - Given the reach of some of these tools, I question Tidelift's absence from this list.
WRT to changing the VMware entry to Pivotal/VMware -- in this instance it should be ONLY VMware. With the acquisition the Pivotal brand is no more. In the case of Red Hat and IBM -- the Red Hat brand was retained and in essence forms a pseudo-subsidiary. Pivotal is fully integrated. So, please leave VMware as is.
WRT to company selection in Q32 - does anyone have suggested criteria for inclusion or exclusion? TODO Group membership... LF Member (Silver or above)... company size... industry... searching for a valid set of criteria that we'd feel comfortable with and comfortable defending. But also don't want to have a LONG list of names for respondents to wade through...
Can we attack that from another angle? What is anyone going to do with the answers from question 32? This isn't a brand reputation survey for the companies in question, so looking at the broader open source ecosystem I can't figure out how the answers to that question are valuable. If we know the answer to that question we can come up with a list of candidate companies more easily.
As a reminder, we used Q32 last time for this article: https://thenewstack.io/survey-shows-how-developers-and-their-employers-measure-good-open-source-citizenship/.
Others have tried a different approach. See how I wrote up Digital Ocean's attempt: https://thenewstack.io/the-value-of-big-tech-in-open-source-sustainability/.
Thanks, I had forgotten about that. Given that, I see that this is solely a brand-reputation exercise, and there's no practical way to extend it to the long list of companies who deserve to be on the list. If this year's results show Salesforce getting a better rating, for example, that's not actionable in any particular way, it's just good news for Salesforce.
@DuaneOBrien re: Q43 and Q44 -- we were not trying to ask about specific tools.
For Q44, I was only trying to include the major software composition analysis players. I don't think Tidelift fits into this mix, but the definition of this market is crazy. If we add any more companies (and I don't really want to, I think these would be the two at the top of my list:
Hi All,
Want to weigh in on the discussion. Yes the list needs to be defensible, be ToDO group members and also diverse.
Open source is adopted and supported by not just tech companies but also enterprises like Bloomberg, Capital One, Comcast etc. and furthermore more verticals are using and building OSPOs. I would like to see this be a more inclusive and defensible list as well. I know I would like to understand how our open source efforts and investments are being perceived just as much as tech companies. And we are an active member of the community contributing to open source as well.
Thank You, Nithya
Sent from my IPhone
On Apr 17, 2020, at 5:51 PM, Kevin P. Fleming notifications@github.com wrote:
In previous years I found the company list for question 32 to be non-inclusive and not understandable, and I don't think it's gotten better. The general population doesn't consider Comcast to be a 'tech' company any more than they consider Bloomberg to be a 'tech' company. The list of TODO Group members who are also 'large technology companies' is much longer than what is included in the survey, and at least some of the missing companies have contributed as much (or more) to the open source community as companies who are on the list. There are also a number of companies who should be on this list but are not TODO Group members (HashiCorp comes to mind, I'm sure I could easily find 6 more without much effort). When I've filled out this survey in the past, the list of companies read as if it was a 'sponsor list' because it was so short; I know it's not, the sponsors of this survey are well indicated, but I've completed enough analyst/research surveys in the past to get that impression from this question.
In fact only one of the companies represented by the current TODO Group Steering Committee are in this list (AWS). The remainder are GitHub, Bloomberg, Indeed, Spotify and Juniper Networks.
In any case, at a minimum VMware needs to be changed to Pivotal / VMware, just like IBM / Red Hat.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
@nruff, Right now we are asking about 11 companies. For me it is not acceptable to include more than 14 in the list. Here is my suggestion for the companies to include:
AWS Microsoft Google
Facebook Tencent
SAP Salesforce
IBM VMware
Uber Netflix
Comcast Verizon
And, I suggest adding "The following list represents a range of large companies that use open source." to the front of Q32, so it would now read "The following list represents a range of large companies that use open source. To what degree do you perceive each of them to be “good open source community citizens” in terms of contributions, collaboration and leadership on open source projects and initiatives within the open source ecosystem?"
@caniszczyk, @SuzanneA300, and others:
Understand that is is characterized as large companies and not just tech. And has some telecom with the tech companies. Is there room to include fintech? Like Bloomberg and CapitalOne? If not, we should consider it for next year.
Thank You, Nithya
Sent from my IPhone
On Apr 17, 2020, at 7:42 PM, Lawrence Hecht notifications@github.com wrote:
And, I suggest adding "The following list represents a range of large companies that use open source." to the front of Q32, so it would now read "The following list represents a range of large companies that use open source. To what degree do you perceive each of them to be “good open source community citizens” in terms of contributions, collaboration and leadership on open source projects and initiatives within the open source ecosystem?"
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
Here is a revised list.
AWS Microsoft Google
SAP Salesforce
IBM / Red Hat VMware
Facebook Uber
Comcast Verizon
Bloomberg CapitalOne
Thanks Lawrence. That looks good to me. Appreciate it.
Thank You, Nithya
Sent from my IPhone
On Apr 17, 2020, at 8:06 PM, Lawrence Hecht notifications@github.com wrote:
Here is a revised list.
AWS Microsoft Google
SAP Salesforce
IBM / Red Hat VMware
Facebook Uber
Comcast Verizon
Bloomberg CapitalOne
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.
While Red Hat is now owned by IBM, the open source practices of both companies are still very separate. I'd encourage not listing them as a single unit, particularly because community perceptions are likely to vary for them based on their separate histories.
Overall good. I like it. Lots of nuance things in the comments below.
On thing that stuck out for me was the relative emphasis on licensing and compliance over security. Historically licensing has been the hot topic but security is foremost on many people's minds. I'd like to have better understanding of how people are viewing security and what they are (or are not) doing about it.
Detailed comments:
On first read it is ambiguous whether we're talking about how many products or how much open source is in each.
"Program" has proven to be a challenging word in past surveys. Many people are not familiar with it in the way we mean. Suggest either using a different term or backhandedly defining like
Does your company have a formal or informal management initiative or program around open source?
Nit but suggest moving this later around the size and originating date questions (# 12 or so)
Suggest adding an answer for "Security team" or some such. That would also need to go in 22
Suggest using a different term for "Program Manager". At Microsoft we had 3 people whose title was "Program Manager". Suggest Program Lead or Program Director or something that implies the leader of the program. If changed, also update wording in answers for 23
Nit: Little odd having "Strategy:" as the only answer in that format
Suggest defining out "program" again to avoid confusion similarly to my comment on 6. Perhaps:
Why doesn't your company have a formal or informal management initiative or program around open source?
Very interesting.
Should clarify the roles here. perhaps
To what degree does a potential supplier's participation in, and contributions to, the open source community influence your organization’s buying decisions?
The meaning of this question is unclear. Is this asking about reallocation from in-person events to online/async material from open source folks?
Reality check! need options for quarterly and annually. Most companies have yet to achieve devops release cadence.
Do we want to include "and or DCO" in this? That is, why the focus on CLA? At the high level it is really "do y'all understand what it means to take contributions"? or is there something else we're after with this question?
More of a curiosity: Do we care about the difference between 2 and 3 clause BSD? I'd think we cared more about GPL 2 vs GPL 3
While Red Hat is now owned by IBM, the open source practices of both companies are still very separate. I'd encourage not listing them as a single unit
@mekkim I understand your point of view. If we ask about Red Hat separately, then we should also ask about GitHub separately. Both choices will almost assuredly get high ratings based on their long association with open source.
We are not asking "Microsoft / GitHub" (nor do I think we should) so the comparison is not the same. It really depends what we are trying to get out of this question. @LawrenceHecht, what is the goal of the question? If it is member brand recognition (for example) then we should be asking using the member's brand (e.g., IBM). If we want to get into all the subsidiaries, most of these companies have many well known subs.
@jeffmcaffer, the purpose of the question is to measure reputation and then see if it matters. I'm going to table discussion on this one question for now.
@DuaneOBrien
Question 39 - I would LOVE to see this paired with "How large is your company's engineering organization" and "How many employees does your company have"
We will look at the data based on # of employees
Also recommend asking how many employees contribute open source, rather than focusing only on developers. Season of Docs would be an argument for broadening the language here.
I understand your point. Non-developer contributions are important. That said, I think we should keep the question for two reasons: 1) to allow time series comparisons, and 2) this question already had a lot of people (17%) saying they don't know, and changing the question will increase that.
I'll go through @jeffmcaffer's comments tonight and make some changes to questions, a lot of the suggestions are good
@LawrenceHecht we should add a question regarding open source + security, essentially are people using automated tools to look at security issues on top of other concerns.
@caniszczyk I just made 2 pull requests: https://github.com/todogroup/survey/pull/64 took care of some easy changes. https://github.com/todogroup/survey/pull/65 has my recommendation for the final list of companies for the citizenship question.
Outstanding things I wanted to address are:
Chris, tomorrow I'll review your changes, with particular attention to how they many affect time series comparisons.
addressed a ton of @jeffmcaffer's concerned in https://github.com/todogroup/survey/commit/0a0adf96633c2bebfe87e304f2b6e1b08346b299
@LawrenceHecht feel free to suggest any more changes but we are getting closer
Question 41
Do we want to include "and or DCO" in this? That is, why the focus on CLA? At the high level it is really "do y'all understand what it means to take contributions"? or is there something else we're after with this question?
@jeffmcaffer Last year, 16% required a CLA and 41% didn't know the answer to this question. My preference is to keep the question for time series purposes or get rid of it. If we did include DCO, I would prefer to break that out as separate from a CLA.
@caniszczyk & everyone else. I believe we addressed most people's comments. I am closing this issue. The next steps are copy editing, updating the survey's coding, and testing the survey before launching it.
Going to give the SC a bit more time, we're almost there
We are making final modifications to the OSPO survey and would like to get the SC to sign off :)