Open-source and open science for large-scale generalist learning and transferable foundation models

[JeniaJitsev]

Description

Major recent breakthroughs in generalist, transferable learning were executed by training and using large-scale language, vision or language-vision foundation models like GPT, ViT, CLIP or Stable Diffusion. Strong transferability of these models in zero-shot and few-shot regimes makes it possible to adapt those to various tasks where only small amounts of training data or no training data is available, while the pre-training of such models require both large compute and large volumes of generic data.

However, the study of such foundation models at larger scales is still mostly conducted in few large industry labs (openAI, Google, Facebook). To counteract this trend of power concentration and allow broader research community to study these important model class, various grassroot organizations self-organized into networks of research labs, machine learning enthusiasts and citizen scientists to gain expertise and resources necessary for such studies. Some important examples are

1. BigScience, France https://bigscience.huggingface.co/
2. ELeutherAI, USA https://www.eleuther.ai/
3. LAION, Germany https://laion.ai/

Following those examples, we would like to discuss what is necessary to bring back research on foundation models at scale to academia and what are the perspectives of open-source and open science for studying transferable foundation models at larger scale. Speficially, we would like to discuss collaborative efforts

• to obtain necessary large-scale datasets for training such models (see for instance LAION-5B: https://openreview.net/forum?id=M3Y74vmsMcY, Outstanding NeurIPS paper award 2022; DataComp: https://www.datacomp.ai)
• to enable model training that ensures reproducibility using common resources (see for instance https://github.com/LAION-AI/scaling-laws-openclip)
• to put in place research repositories where data, models and workflows for their handling can be shared among broad research community (see for instance https://github.com/mlfoundations/open_clip, https://github.com/mlfoundations/open_flamingo)

Organizational

Organizer(s)

Jenia Jitsev (JSC, LAION), j.jitsev@fz-juelich.de Christoph Schuhmann (LAION), christoph.schuhmann@laion.ai

Format

Open discussion & world cafe with smaller groups according to topics (datasets, model training)

Timeframe

1.5 h

Number of participants

Min. 3 participants

[SusanneWenzel]

@JeniaJitsev Would you need a screen? We possibly don't have one for each session. Flipchart will be available

[ilsenatorov]

Would it be possible to start at 14:30?

[SusanneWenzel]

Sure, already noted

[JeniaJitsev]

@JeniaJitsev Would you need a screen? We possibly don't have one for each session. Flipchart will be available

Screen would be actually very helpful, yes

[JeniaJitsev]

We start 14:30 and are looking forward fruitful discussion and exchange! As a good starter point - if you have a story of using a pre-trained foundation model (LLM, vision, language-vision) for transferring to your particular research, feel free to share it.

[SusanneWenzel]

@JeniaJitsev if possible, please make a note here on the (rough) number of participants. Also don't forget to make a note here about the outcome of the session and, if applicable, future plans that came out of this session.