search

EthicalML / awesome-production-machine-learning

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
https://ethicalml.github.io/awesome-production-machine-learning
MIT License
17.31k stars 2.22k forks source link

Proposal to add prompt engineering tools #408

Open zhimin-z opened 1 year ago

zhimin-z commented 1 year ago

https://github.com/EthicalML/awesome-production-machine-learning/pull/416

axsaucedo commented 1 year ago

I'm not yet clear how on prompt engineering in this list, I do see huge value in this field but even looking at the tools proposed in the PR, I fail to see how these are tools for production machine learning as opposed to application-level tooling to leverage / interact with LLMs (e.g. https://github.com/promptslab/Awesome-Prompt-Engineering). I would see something along the lines of New Section on GenAI via #339 as more relevant to be explored in favour of something specifiaclly around prompt engineering tooling.

zhimin-z commented 1 year ago

I'm not yet clear how on prompt engineering in this list, I do see huge value in this field but even looking at the tools proposed in the PR, I fail to see how these are tools for production machine learning as opposed to application-level tooling to leverage / interact with LLMs (e.g. https://github.com/promptslab/Awesome-Prompt-Engineering). I would see something along the lines of New Section on GenAI via #339 as more relevant to be explored in favour of something specifiaclly around prompt engineering tooling.

Good catch, @axsaucedo . The logic behind prompt engineering tools is that they could foster the deployment of LLM/StableDiffusion in industry production. For example, the artists could try with various painting style using the prompt engineering tools and this not only save their time and efforts but also bring the underlying power of StableDiffusion models to real-world applications. So does LLM-related prompt tools as listed in the PR.

axsaucedo commented 1 year ago

the artists could try with various painting style using the prompt engineering tools and this not only save their time and efforts but also bring the underlying power of StableDiffusion models to real-world application

Just to clarify I am not challenging the usefulness of production usecases of promp engineering, but I don't see it as a section for production ML frameworks at this stage, especially looking at the PR.

The tools proposed in the PR seem to be: 1) click-prompt is only a UI for running models - not relevant for this list; 2) promptify is just an SDK to interact with OpenAI API - not relevant for this list; 3) Promptsource is an sdk to save templates for prompts - closest section would be data mgmt but I wouldnt' consider this relevant for this list at this stage.

There is quite a lot of hype going around with these tools so I don't want to jump on the bandwagon and add sections just because a lot of talk around it or a lot of potential - certainly keen to revisit eventually, but at this stage based on these examples I don't see a prompt engineering section being added at this stage, instead I woudl suggest we continue the discussion on how a GenAI or LLMs section that focuses on productionisation tools for these type of AI - however as per the discussion it does seem like there's ambiguity there but keen to continue exploring.

zhimin-z commented 3 weeks ago

Even rag workflow [RAG denotes a class of LLM applications that use external data to augment the LLM’s context.] somewhat belongs to the prompt engineering domain. Since the output of the rag, namely extra context would be fed into the LLM as part of the prompt.

zhimin-z commented 1 week ago

How about tools such as https://github.com/bigscience-workshop/promptsource?