[Paper] Outline

[ch3njust1n]

Dynamic Metaprogramming with Large Language Models

Sections:

1. Abstract

2. Introduction

   • Metaprogramming -- What is metaprogramming -- Types of metaprogramming -- Why use metaprogramming

• Code injection -- What is code injection -- What are the different types of code injection -- How to mitigate most common attacks -- How does this relate to metaprogramming

3. Related work

4. Dynamic Metaprogramming with Large Language Models

5. Results

6. Considerations

   • Practicality -- Latency -- Compute & Cost
   • Secure code
   • Best practices

7. Risks for run-time metaprogramming

   • Difficulty in directing outputs of LLM
   • Difficulty of guaranteeing correctness of LLM outputs

8. References

[ch3njust1n]

Section: Generative API examples

Given the function name and behavior:

{
        "actions": [
            {
                "component": "device",
                "action": "save-custom-mode",
                "parameter": "study-mode",
                "commands": [
                    {
                        "component": "app",
                        "action": "disable-notifications",
                        "parameters": "",
                    },
                    {
                        "component": "device",
                        "action": "set-volume",
                        "parameters": "0",
                    },
                    {"component": "call", "action": "block-all", "parameters": ""},
                    {"component": "text", "action": "block-all", "parameters": ""},
                    {
                        "component": "call",
                        "action": "unblock",
                        "parameters": "555-555-5555",
                    },
                    {
                        "component": "text",
                        "action": "unblock",
                        "parameters": "555-555-5555",
                    },
                ],
            }
        ]
    }

text-davinci-003 generated:

# Candidate 1:
def save_custom_mode(modes):
    for mode in modes['study-mode']:
        if mode['action'] == 'disable-notifications':
            disable_notifications(mode['component'])
        elif mode['action'] == 'set-volume':
            set_volume(mode['component'], mode['parameters'])
        elif mode['action'] in ['block-all', 'unblock']:
            manage_block(mode['component'], mode['action'], mode['parameters'])

# Candidate 2:
def save_custom_mode(mode_data):
  # Parse mode data
  mode_config = mode_data['study-mode']
  # Iterate over mode data
  for config in mode_config:
    # Retrieve parameters
    component = config['component']
    action = config['action']
    parameters = config['parameters']

    # Execute action for component
    if component == 'app':
      if action == 'disable-notifications':
        # Disable notifications
        pass
    elif component == 'device':
      if action == 'set-volume':
        # Set the device volume
        pass
    elif component == 'call':
      if action == 'block-all':
        # Block all incoming calls
        pass
      elif action == 'unblock':
        # Unblock specific incoming call
        pass
    elif component == 'text':
      if action == 'block-all':
        # Block all incoming texts
        pass
      elif action == 'unblock':
        # Unblock specific incoming text
        pass

# Candidate 3:
def save_custom_mode(study_mode):
    mode_list = eval(study_mode)
    for mode in mode_list:
        component = mode['component']
        action = mode['action']
        parameters = mode['parameters']
        # Perform logic to save components and parameters

[ch3njust1n]

Section: Generative API examples

Given the function name and behavior:

{
        "actions": [
            {
                "component": "device",
                "action": "save-custom-mode",
                "parameter": "study-mode",
                "commands": [
                    {
                        "component": "app",
                        "action": "disable-notifications",
                        "parameters": "",
                    },
                    {
                        "component": "device",
                        "action": "set-volume",
                        "parameters": "0",
                    },
                    {"component": "call", "action": "block-all", "parameters": ""},
                    {"component": "text", "action": "block-all", "parameters": ""},
                    {
                        "component": "call",
                        "action": "unblock",
                        "parameters": "555-555-5555",
                    },
                    {
                        "component": "text",
                        "action": "unblock",
                        "parameters": "555-555-5555",
                    },
                ],
            }
        ]
    }

claude-instant-v1 generated:

# Candidate 1:
def save_custom_mode({'study-mode': "[{'component': 'app', 'action': 'disable-notifications', 'parameters': ''}, {'component': 'device', 'action': 'set-volume', 'parameters': '0'}, {'component': 'call', 'action': 'block-all', 'parameters': ''}, {'component': 'text', 'action': 'block-all', 'parameters': ''}, {'component': 'call', 'action': 'unblock', 'parameters': '555-555-5555'}, {'component': 'text', 'action': 'unblock', 'parameters': '555-555-5555'}]"}):    
    pass

# Candidate 2:
def save_custom_mode(study_mode):
    pass

# Candidate 3:
def save_custom_mode({'study-mode': "[{'component': 'app', 'action': 'disable-notifications', 'parameters': ''}, {'component': 'device', 'action': 'set-volume', 'parameters': '0'}, {'component': 'call', 'action': 'block-all', 'parameters': ''}, {'component': 'text', 'action': 'block-all', 'parameters': ''}, {'component': 'call', 'action': 'unblock', 'parameters': '555-555-5555'}, {'component': 'text', 'action': 'unblock', 'parameters': '555-555-5555'}]"}):   
    pass

[ch3njust1n]

Mention that to give the model greater context, the entire repo could be given to the model before deployment. The model can then use that context and the current input for generating code at run-time.

[ch3njust1n]

"[Metaprogramming] ... enables expressing anticipated variability in a domain explicitly" - page 10 of Meta-programming and Model-Driven Meta-program Development