State of AI 2023 Preliminary Discussions

[SachaG]

Although it's still early days, we are thinking about holding our first ever "State of AI" developer survey this year, so I'm opening this thread to collect feedback, suggestions, ideas, or anything else you'd like to share about the topic.

Some important things to note:

• There is already a State of AI Report (which is a written report, not a survey). We'll try to do our best to avoid any confusion due to the similar naming, or ideally outright collaborate with them.
• This survey would be focused specifically on how "regular" developers use AI, e.g. tools like ChatGPT, CoPilot, etc. In other words, how non-AI-specialists are leveraging AI tools to help with coding, design, management, or any other aspect of their job.

Some questions the survey could try to answer include:

• What percentage of developers use AI tools?
• What new skills/capabilities are these tools unlocking that people just didn't possess before?
• Which tools are the most popular?
• Which apps/services/jobs/etc. will these tools potentially replace?
• How worried are non-AI-specialists about the risks of AI?

[swyxio]

would like to make it as specific to the "AI Engineer" profile that is making use of tooling to put LLM apps in production!

[rogeriochaves]

hey there! I completely agree with the focus of having non AI-expert focus, but I do think we can dig deeper, there is a large chunk of "regular" developers that migrated to building a lot on top of AI, they are NOT ML/DS engineers, but they are building chat apps, RAGs and so on, which at some point lead to having to serve it production, monitor, doing prompt engineering, finetuning, etc. The sheer amount of super popular tools that popped up this last year screams "fatigue" to me, so a survey would be awesome 😄

I would really like to contribute to make this happen for 2024, I think a lot has happened throughout 2023, but still we only have written reports like those and not a wide survey, and if I may, I already thought of some topics we could ask about and tried to collect the (currently) most popular ones for each:

What application types have you developed (in the most abstract level) Chat Application Content Generation, Summarization or Augmentation Code or Data Generation, Transformation or any deep tech integration LLM-enabled Automation

What LLM capabilities are you using Function Calling Retrieval Augmented Generation (RAG) Synthetic Dataset Generation Data Evaluation and Classification Multi-modal capabilities

Models GPT-4 (turbo or not) GPT-3.5 (turbo or not) Claude 2 Llama2 Code Llama Mistral 8x7B Gemini

Providers OpenAI Anthropic Cohere Google Mistral HuggingFace Replicate DeepInfra Azure AWS Bedrock & SageMaker

Running Locally GPT4all Ollama llama.cpp LocalAI

Serving Traditional Web Server (flask, express, etc) OpenLLM Fastchat vLLM

LLM Frameworks LangChain LlamaIndex Haystack Semantic Kernel (https://github.com/microsoft/semantic-kernel)

UI Interface FastChat (https://github.com/lm-sys/FastChat) Chainlit (https://github.com/Chainlit/chainlit) Serge (https://github.com/serge-chat/serge)

Copilot Tools GitHub Copilot Cursor Mentat

Vector Databases Chroma Pinecone Weaviate ElasticSearch OpenSearch MongoDB Postgres

Prompting Techniques Chain-of-Thought Few-shot Self-consistency Self-generated knowledge Tree of Thoughts Automatic Prompt Engineer ReAct Medprompt

Evaluation Guardrails (https://github.com/guardrails-ai/guardrails) Ragas (https://github.com/explodinggradients/ragas) Promptfoo (https://github.com/promptfoo/promptfoo) Uptrain (https://github.com/uptrain-ai/uptrain)

Monitoring LangSmith LangFuse Weights & Biases

LLM Utilities Guidance (https://github.com/guidance-ai/guidance) Instructor Outlines (https://github.com/outlines-dev/outlines)

Finetuning Manual Scripts (HuggingFace or otherwise) OpenAI API Ludwig LLaMA-Factory OpenPipe