Rallio67
commented
1 year ago Hello,
I will clarify a little what is going on with this work.
I have been experimenting as a hobbyist with large language models for a few years now. However, my professional background is in experimental scientific research, not in academic or corporate NLP research. I am not following any typical behaviors people do when they try to publish a research paper or get some hype for a company or new software. I do not intend to publish any research papers or professional quality software, so if you are looking for that level of support with anything on this repo, I will tell you up front I am not planning to do anything like that.
With regards to your specific questions about more details on everything that was done to generate these datasets. There are several people associated with LAION that I am collaborating with on these efforts to generate instruction tuned models. Right now there is a lot of rapid changes and experiments we are doing to try to address the various problems with datasets and the deficiencies of large language models. Very many different approaches are being tested and tried. These models and scripts that I am posting are just a "snapshot" or "sneak peak" of what we are trying (which is why I put the label experimental and alpha on everything).
When we complete our efforts or feel we have made a significant discovery we will document and share the results on the LAION github repo ASAP. Anything that I am heavily involved in I will also write up on here or provide some better documented code so others can replicate the results. If you want to see some more detailed examples of the components we are using to do rejection sampling you can checkout some of the work by Ken (https://github.com/kenhktsui/) who is also doing a lot of work in this area. Things have been changing fast though and something we use one week is often replaced by a new / better approach the following week.
With UL2 20B it is using few shot prompting to convert an academic NLP reasoning example into a natural instruction. This is used on a case by case basis to reformat any academic text into a more natural instruction.
For instance converting this Com2Sense example:
"id": "c55e23cb21b44ce",
"sent": "Ruby was seven months pregnant in October, so she expected to be even bigger in November, but smaller by January.",
"label": "True",
"domain": "temporal",
"scenario": "comparative",
"numeracy": "True"Into this natural instruction: User: If Ruby was seven months pregnant in October, would she expect to be even bigger in November, but smaller by January?'
Agent: Yes, she would expect to be bigger in November because she was seven months pregnant in October. She would expect to be smaller in January because she would have given birth to her baby.
robinsongh381
Hi, its me again. Your work is of so much fun, cant stop asking questions.
When you prepare training datasets, you seem to obtain from data augmentation by few shot prompting on large language models.
For example, you use UL2 20B or GPT-NeoX-20B model (Chip) (but which Chip model? Chip 1 or 2 ) for Natural Instruction examples. However, it is unclear how many shots you used and how you formatted/designed the prompt examples. Also, it is unclear which LLMs you used for the rest of datasets such as for Reasoning Instructions or Character and Scene Descriptions.
It would be very helpful if you could specify the few-shot prompt formats as well as the LLMs you used.
Thank you