Add section on LLMs, chatGPT, and prompt engineering

[svenvanderburg]

Development on LLMs and specifically chatGPT now goes so quickly that we cannot ignore it in our lesson I think. In fact, some of the research projects that I assisted in can probably be solved very well with ChatGPT prompt engineering. To be clear I mean constructing some question based on the input, ask that to ChatGPT, and do something with the answer to get an output, I don't mean asking ChatGPT to type some keras code for you.

I believe that many machine learning problems that researchers have will be easily solved by doing some simple prompt engineering with chatGPT. One might go as far to say that, to train researchers for the future we might have to teach them prompt engineering instead of deep learning....

With the newest chatGPT version one can now also upload images, so with some prompt engineering the CIFAR-10 task can also be solved by chatGPT..., see: https://openai.com/blog/chatgpt-can-now-see-hear-and-speak

I suggest to add a section in the introduction on chatGPT answering this learning objective: 'understand how I can use chatGPT in combination with prompt engineering to solve machine learning tasks'. It does not have to be long, maybe 5 minutes extra. Maybe we can add an exercise 'Which problem could you solve with ChatGPT prompt engineering?' Then we can refer to https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/ for people that want to go into this.

What do others think? @colinsauze @psteinb @dsmits @cpranav93

[carschno]

Very generally, I see a huge problem with reproducibility there. I have seen a few papers in NLP comparing their own models to GPT; the workaround is typically a disclaimer because GPT can return different results per run, and they depend on the exact prompt too.

Open models should at least make it possible to reproduce the same results when you use the same random seed etc.; if you can run the model on your own infrastructure.

I am very sceptical about prompt engineering; for me, it is essentially trying to operate an undocumented black box. Everything you find out about the best prompts will be useless for future generations. However, ignoring this might not be a good option either, so perhaps some higher level discussion about the issues along with some examples could be a compromise.

[dsmits]

I like the idea of a higher level discussion. It might also be a way to collect experiences from researchers from various domains that we can use to improve the content of the episode.

[colinsauze]

I feel like prompt engineering could be a lesson all of its own. We could get learners to do a set of exercises to solve some research problems.

[svenvanderburg]

Yes, I also like the idea of a higher level discussion, thanks @carschno! Reproducibility is an issue. I think you should compare chatGPT to a team of humans doing data annotation, which is also not reproducible at all, but can in many fields be super useful.

I worked on a politics/linguistics project were interns would annotate a few 100 of transcribed political speeches according to some rules. With chatGPT this would be 10.000s with presumably the same level of quality and reproducibility.

But let's have this discussion in the classroom it's super interesting!

[svenvanderburg]

@colinsauze just to double check you mean an entire different lesson separate of the deep learning lesson right? Like https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/

[colinsauze]

@colinsauze just to double check you mean an entire different lesson separate of the deep learning lesson right? Like https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/

I was thinking an entire lesson and that short course looks like the sort of thing I had in mind.