search

Devographics / Monorepo

Monorepo containing the State of JS apps
surveyform-sigma.vercel.app
Other
119 stars 50 forks source link

State of JS 2022 ToDo/Feedback #61

Closed SachaG closed 7 months ago

SachaG commented 2 years ago

To consider:

SachaG commented 2 years ago
SachaG commented 2 years ago
SachaG commented 2 years ago

Add YouTube category

IgorMinar commented 2 years ago

consider some type of open governance so that more people can participate in preparing the survey, and raising awareness of the survey.

IgorMinar commented 2 years ago

consider publishing a description of the methodology used in this survey

SachaG commented 2 years ago

consider some type of open governance so that more people can participate in preparing the survey, and raising awareness of the survey.

How would that work? What would be different from the current way the survey is run practically speaking?

I feel like maybe you're under the impression that the main barrier to more people being involved is that I don't want them to be, but nothing could be further from the truth. I've spent a lot of time bugging various people to give me their input on the survey outline, and many of them did so here.

Similarly I'm all for raising awareness but a look at my Twitter feed and DMs, Gmail inbox, or dev.to profile would show that I'm already pretty much working on that full time. And these kind of inquiries usually have a very low success rate to begin with.

consider publishing a description of the methodology used in this survey

I did that for the State of CSS survey, although this being an open survey there is not a ton of "methodology" to speak of.

By the way sorry if I was abrasive on Twitter, but it's just not a good platform for constructive dialog.

IgorMinar commented 2 years ago

@SachaG I have no illusion that running a survey like this doesn't take a considerable amount of your time. I truly appreciate your willingness to put a lot of your energy and time into this project. I love it when community members feel empowered to do things that the community benefits from. I honestly have only the utmost respect for you and the team around you even if I have gripes with StateOfJS in its current form.

The thing that I, as well as many others involved in the web community struggle with, is that this survey is divisive, and comes across as a biased popularity contest rather than a valuable feedback mechanism that could be used to improve the JS ecosystem for everyone. This, after all, should be our goal, and I hope you agree.

Unfortunately, there are four main areas of concern with StateOfJS in its current form:

  1. The sample selection used by StateOfJS doesn't represent the JavaScript community.
  2. Additionally, the way some of the questions are formulated lead to biased comparisons that don't reflect the real-world decision making well.
  3. As a result of how the questions are formulated, more often than not, the results are misinterpreted in ways that lead to unhealthy division in the Web community.
  4. Lastly, the survey collects PII and private information, but the stateofjs.com site doesn't have any privacy policy, or info on how the data will be used, or who will have access to it. This most certainly makes the whole survey GDPR non-compliant.

The fact is that the results of the survey try to portray trends in the JS ecosystem and are often cited in various publications, blog posts, presentations, etc. And many of us waste days, if not weeks responding to various inquiries or confusions related to the StateOfJS results.

Please make no mistake, I'm not trying to ensure that my favorite tool/framework/etc is announced to be the "winner" of StateOfJS. I don't need that kind of validation. Most of us don't need it.

What I would however appreciate from a survey like this are valuable insights and feedback that would effectively enable the community to draft actionable items, to improve the JS ecosystem.

There are well-constructed, insightful, and valuable surveys out there that I find extremely useful, two come to my mind: — the StackOverflow survey and Octoverse by GitHub. These use a combination of observational studies and surveys, and their survey questions are very well designed.

I fully understand that these are reports funded by wealthy corporations, and that they have the resources to hire data scientists and survey design experts, organize thorough data collection and data processing, and carefully interpret the results.

StateOfJS is however very prestigious and equally regarded. That's a huge accomplishment! Congratulations to you and the team! That accomplishment however translates into a lot of power y'all have, and with that power comes responsibility.

After all, all of us want the JS ecosystem to thrive and improve. So how about we explicitly use the power of StateOfJS to support this mission by turning it into an unbiased and neutral feedback mechanism that would make the JS ecosystem better?

SachaG commented 2 years ago

First of all I want to thank you for taking the time to write up such a detailed comment, I often get annoyed by people on Twitter who do drive-by criticism without taking the time to dig more into the project but that's clearly not the case here, so thank you!

Regarding your concerns:

1.

The sample selection used by StateOfJS doesn't represent the JavaScript community.

This is a great post, and maybe @davideast would also have feedback on this whole topic they'd like to share here. But I would argue that the survey doesn't really aim to represent the "whole" JavaScript community, for two reasons:

  1. It's really hard to even define what that is.
  2. The survey is squarely aimed at identifying upcoming trends and not interested in identifying current usage levels, since other surveys and datasets already do a much better job of this. Maybe this can be clarified somewhere.

But if you think even with those caveats the survey still has sample selection issues, then I would welcome suggestions of how to change/improve the sample.

2.

Additionally, the way some of the questions are formulated lead to biased comparisons that don't reflect the real-world decision making well.

Could you provide some examples?

3.

As a result of how the questions are formulated, more often than not, the results are misinterpreted in ways that lead to unhealthy division in the Web community.

Could you provide some examples?

4.

Lastly, the survey collects PII and private information, but the stateofjs.com site doesn't have any privacy policy, or info on how the data will be used, or who will have access to it. This most certainly makes the whole survey GDPR non-compliant.

There is a privacy policy here but you're right that it should be present on all survey-related sites.

Overall my general impression of your comment is that you point out a range of issues but without A) providing the basis for what you're saying in terms of proofs, reasons, examples, etc. and B) providing suggestions of what to do about the issue. I think if you can address those two concerns it will be easier for me to understand how you'd like things to change.

IgorMinar commented 2 years ago

One examples of how hard it is to understand the results / how easy it is to misinterpret them is TypeScript adoption according to SoJS.

Take a look at javascript flavors and switch to "usage".

Screen Shot 2022-02-03 at 6 11 06 AM

So while providing this data summary is visually pleasing, it confuses people more than helps. If I were to display anything in this results section, I'd just say that TypeScript seems to be trending up, while all the other JS flavors don't appear to have any significant mainstream adoption. That's all we know. All the other graphs and diagrams (while very very cool looking) try to tell a story that is based on very unreliable data, so we either should improve the data, or not try to tell that story at all.

Let's take a look at another example, this time satisfaction with these JS flavors:

Screen Shot 2022-02-03 at 6 11 20 AM

This graph suggests that the satisfaction with Elm has plummeted over the years (all while, the usage/adoption has increased as the previous graph suggests). And Clojure's story as told by SoJS is similar.

I'm not an Elm or Clojure developer, but I find it hard to believe that these graphs reflects the reality or even trends within these communities. So by trying to present these numbers as "state of JavaScript" you are unintentionally creating division, and in fact forming opinions that Elm and Clojure are going down hill just because a non-representative sample of the community says so.

I'm intentionally picking examples from the JS flavors section, because I have no skin in the game there. But the same kind of issues are seen through all of "Technologies" sections of SoJS where competitive solutions are being "compared".

I previously mentioned that how the questions are formed is a bit unfortunate and leads collection of data that is not that helpful. What I mean by that is "would you use again/would you not use" style of questions assumes greenfield solo development. That scenario is extremely rare in practice. Most of the code today is being built with constraints such as corporate policies, legacy code base, business requirements, team skillset, skillset available for hire, existence of integrations, etc.

A crushing majority of professional developers don't get to pick tech they use on the project based on their personal preference as the survey suggests. And if they do that, the project more often than not fails to meet business requirements or fails to launch at all. Devs do have a say in which tech becomes prevalent over long periods of time by learning the tech or building out integrations, but this influence plays only partial role in the overall short-to-mid-term decision making.

Now combine this with the previous sampling issue I highlighted, and you get data set that is not at all related to reality on the ground — something I really want SofJS to be.

Does this make sense at all?

And yes, I can simply ignore the results, and move on. I did that for years. But what troubles me is that developers, managers, and even execs that are not familiar with the blind spots of SoJS, consciously or unconsciously accept the results as "reality on the ground", and use the results in planning processes and decision making, turning SoJS into an opinion forming tool that forms opinions using very shaky and often biased data.

And again, please understand that I don't bring this up because I don't respect your work. I really do. I just can't stand still any more and ignore the unintentional negative effect that SoJS has on the JS community. And this effect is getting only stronger as the survey becomes more popular.

And lastly a few concrete suggestions that would improve the SoJS via small tweaks to the front page of stateofjs.com that might be high ROI items:

Thanks for listening and considering the feedback.

SachaG commented 2 years ago

Thanks for writing such a detailed post! To sum up your argument I would say that you have a certain vision of reality, and the survey doesn't match up with it.

So I think this is a disagreement on a philosophical level more than anything: you believe the goal of the survey is to reflect the "true" reality that you yourself perceive. But I think this assumes that there is one objective truth that will satisfy everybody, and I'm not sure that this itself is true… As an example, you asserted a lot of things in your post without providing any sources. What's to stop me from asserting the opposite? And what if both visions are valid from our own subjective point of views?

So for me, the main responsibility of the survey is to instead accurately reflect what its respondents say while also being as transparent as possible about who these respondents are.

So I would again suggest you create a new survey (or some other form of research) that better matches up with your vision. If you do so I'll be happy to help promote it to our audience!

Thanks for the action items though, those are all great ideas and things I will add to our roadmap.

SachaG commented 2 years ago

Also I think we've both made our point of views clear now so I will consider this specific matter (on the fundamental nature of the survey's accuracy compared to your own reality) closed, although I still welcome feedback on other issues of course.

IgorMinar commented 2 years ago

All reality is subjective. All I tried to achieve was to make the results of SoJS a bit more useful for the JS community. But ultimately you get to decide if any of my feedback is helpful.

natorion commented 2 years ago

Rekindling the conversation about adding Selenium from the 2021 survey at https://github.com/StateOfJS/Monorepo/issues/56#issuecomment-999524885.

I am happy to help in crafting the Testing section in the survey.

cc @foolip @jgraham

foolip commented 2 years ago

Yes, I hoped that selenium-webdriver would be included in State of JS 2021, but now I'm hoping for the next one :)

SachaG commented 2 years ago

Yeah sorry for dropping the ball on this! Let's add it in 2022 then.

NiklasBr commented 2 years ago

Can you offer a dark-on-light theme for next year? The light-on-dark is hard for me to read

amalv commented 2 years ago

With the elimination of the data layers section it seems that this also has lead to the disappearance of GraphQL and Apollo Client from the State of JS survey, however I think it's quite important and very interesting to continue to know the trends of data layers. So my suggestion / feedback is to bring it back again next year.

test

SachaG commented 2 years ago

@amalv I'm hoping to do a separate State of GraphQL survey this year!

cliffordfajardo commented 2 years ago

Idea

  1. It would be interesting to see how the javascript is being used at a high level. For example are folks mostly using it for: web development, machine learning, desktop apps, mobile (ios/android/ipad etc)

Jetbrains Python Yearly Survey

CleanShot 2022-02-20 at 19 54 48@2x
  1. Work style patterns

Jetbrains Python Yearly Survey

CleanShot 2022-02-20 at 19 57 25@2x
jpvajda commented 1 year ago

@SachaG 👋 hello again! I hope you are well.

I was curious if this year's state of JS survey could explore questions around which approach developers are considering in 2022 as it relates to SSR (Server side rendering) or CSR (client side rendering)? For example this review of CSR was really interesting by @theninthsky. It got me thinking, how many other developers are facing this decision in 2022, what is their approach and what are some of the considerations being made as they are asked to support more SSR use cases?

jpvajda commented 1 year ago

@SachaG One other question I have to consider adding: "How much does bundle size matter to developers when choosing different packages in 2023?"

With the growing interest in SSR bundle size can be more of CSR concern and I'm really curious to know what people are thinking about in relation to their package bundle sizes in 2023 and the SSR approach, and how much of bundle size is still a concern.

jpvajda commented 1 year ago

@SachaG One more idea comes to mind for the 2022's survey. With the Next.js 13 announcement of support for React Server Components I'm really curious to know how this is going to be adopted by the JS ecosystem and what people think about this approach.