(Some?) MLX models won't stop generating most of the time

[daaain]

I'm using runtime 0.0.13 with 0.3.4 build 2.

So far I've tried mlx-community/Phi-3.5-MoE-instruct-4bit and mlx-community/Qwen2.5-Coder-7B-4bit, their answers start out good, but then they just keep going until I manually stop generation.

I had this happen 2/2 times with Qwen and 2/3 times with Phi.

Here's a chat example with Phi, I manually stopped after it kept on generating variations of the answer endlessly:

{
  "name": null,
  "systemPrompt": "",
  "messages": [
    {
      "versions": [
        {
          "type": "singleStep",
          "role": "user",
          "content": [
            {
              "type": "text",
              "text": "Please implement a React component with MUI that includes toggling between horizontal and vertical interface between two panels."
            }
          ],
          "preprocessed": {
            "role": "user",
            "content": [
              {
                "type": "text",
                "text": "Please implement a React component with MUI that includes toggling between horizontal and vertical interface between two panels."
              }
            ]
          }
        }
      ],
      "currentlySelected": 0
    },
    {
      "versions": [
        {
          "type": "multiStep",
          "role": "assistant",
          "steps": [
            {
              "type": "contentBlock",
              "stepIdentifier": "1728856216437-0.6894782743610182",
              "content": [
                {
                  "type": "text",
                  "text": "<|assistant|>Below is a simple React component that uses Material-UI (MUI) to toggle between horizontal and vertical layout for two panals using the Grid component:\n\n```javascript\nimport React, { useState } from 'react';\nimport Grid from '@mui/material/Grid';\nimport Box from '@mui/material/Box';\nimport Panel1 from './Panel1';  but you can replace it with your own panel component\nimport Panel2 from './Panel2'; // Replace this import with the location of your actual panel component\nimport ToggleButton from '@mui/material/ToggleButton';\nimport ToggleButtonGroup from '@mui/material/ToggleButtonGroup';\nimport ToggleButtonLabel from '@mui/material/ToggleButtonLabel';\n\nconst LayoutSwitcher = () => {\n  const [layout, setLayout] = useState('horizontal');\n\n  return (\n    <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"vertical\">\n      <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"horizontal\">\n        <ToggleButton value=\"horizontal\" aria-label=\"Horizontal layout\">Horizontal Layout</ToggleButton>\n        <ToggleButton value=\"vertical\" aria-label=\"Vertical layout\">Vertical Layout</ToggleButton>\n      </ToggleButtonGroup>\n      <Grid container spacing={2} direction={layout}>\n        <Box component=\"div\">\n          <Panel1 />\n        </Box>\n        <Box component=\"div\">\n          <Panel2 />\n        </Box>\n      </Grid>\n    </ToggleButtonGroup>\n  );\n};\nexport default LayoutSwitcher;\n```\nThis component uses two `ToggleButtonGroup` components to switch between horizontal and vertical layout. The state of the layout is stored in a `layout` variable, which can be either 'horizontal' or 'vertical'. The layout is applied to the `Grid` component by setting its direction prop.\n\nPlease replace `./Panel1` and `./Panel2` with the actual imports of your panel components. You can further customize this component according to your needs and preferences.\n\nMake sure you have the MUI library installed in your project:\n```bash\nnpm install @mui/material\n```\nor if you are using yarn:\n```bash\nyarn add @mui/material\n```\nThis component is a simple example and may require further adjustments to fit specific requirements. The panels are represented by `Panel1` and `Panel2`. You can replace them with the actual components you want to use. The layout switching functionality is provided by the `ToggleButtonGroup` and its associated buttons (`Horizontal Layout` and `Vertical Layout`).\n\nThis example assumes that you have the MUI library installed in your project. If not, make sure to add it using `npm install @mui/material` or the equivalent command for your package manager.\n\nPlease note that this code might not work properly if you copy and paste it directly to your project. You' may need to adapt the paths of imports, install required dependencies for MUI or other libraries you are using and ensure that all project settings are correct.\nHere's a more complete example considering spacing, responsiveness and using the `makeStyles` hook for applying styles:\n```javascript\nimport React, { useState } from 'react';\nimport Grid from '@mui/material/Grid';\nimport Box from '@mui/material/Box';\nimport ToggleButtonGroup from '@mui/material/ToggleButtonGroup';\nimport ToggleButtonLabel from '@mui/material/ToggleButtonLabel';\nimport Panel1 from './Panel1'; // Replace this import with the location of your actual panel component\nimport Panel2 from './Panel2'; // Replace this import with the location of your actual panel component\nimport { makeStyles } from '@mui/styles';\n\nconst useLayoutSwitcherStyles = makeStyles({\n  box: {\n    flexGrow: 1,\n    border: '1px solid gray',\n    padding: 20,\n  },\n});\n\nconst LayoutSwitcher = () => {\n  const [layout, setLayout] = useState('horizontal');\n  const classes = useLayoutSwitcherStyles();\n\n  return (\n    <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"vertical\">\n      <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"horizontal\">\n        <ToggleButton value=\"horizontal\" aria-label=\"Horizontal layout\">Horizontal Layout</ToggleButton>\n        <ToggleButton value=\"vertical\" aria-label=\"Vertical layout\">Vertical Layout</ToggleButton>\n      </ToggleButtonGroup>\n      <Grid container spacing={2} direction={layout}>\n        <Box className={classes.box}>\n          <Panel1 />\n        </Box>\n        <Box className={classes.box}>\n          <Panel2 />\n        </Box>\n      </Grid>\n    </ToggleButtonGroup>\n  );\n};\nexport default LayoutSwitcher;\n```\nThis example offers more control over the layout and panels styling. The `Box` components have custom styles applied using the `makeStyles` hook from MUI. The borders, padding and flex grow properties can be adjusted according to your needs. You may replace the `Panel1` and `Panel2` components with actual panels or other content you want to display. The layout switching functionality is provided by the `ToggleButtonGroup` component and its buttons (`Horizontal Layout` and `Vertical Layout`).\nRemember to replace the imports with actual component locations, adjust spacing as needed and apply styles according to your project's requirements. To ensure this code works well in your environment, adjust paths of imports, install required dependencies and confirm project settings. You might also want to adapt the layout switching functionality based on your specific needs or preferences.\nIn this example, we also use the `makeStyles` hook to apply custom styles on the components. You can replace it with other styling methods, such as `styled` or plain CSS, depending on your project's needs. The `Box` components have custom styles applied using the `makeStyles` hook, which can be adjusted to provide more control over their appearance. You may replace the `Panel1` and `Panel2` components with actual panels or other content as needed. The layout switching functionality is provided by the `ToggleButtonGroup` component and its buttons (`Horizontal Layout` and `Vertical Layout`).\nPlease adapt the layout switching functionality according to your needs or preferences. Remember also to adjust paths of imports, install required dependencies and confirm project settings as needed for the code to work properly in your environment. This example can serve as a starting point and you might need further adjustments or customizations to meet specific requirements. Here's a version using the `useTheme` hook and responsive breakpoints:\n```javascript\nimport React, { useState } from 'react';\nimport Grid from '@mui/material/Grid';\nimport Box from '@mui/material/Box';\nimport ToggleButtonGroup from '@mui/material/ToggleButtonGroup';\nimport Panel1 from './Panel1'; // Replace this import with the location of your actual panel component\nimport Panel2 from './Panel2'; // Replace this import with the location of your actual panel component\nimport { useTheme, ThemeProvider } from '@mui/material';\nimport { createMuiTheme } from '@mui/material/styles';\n\nconst theme = createMuiTheme({\n  breakpoints: {\n    values: {\n      xs: 0,\n      sm: 640,\n      md: 960,\n      lg: 1280,\n    },\n  },\n});\n\nconst LayoutSwitcher = () => {\n  const [layout, setLayout] = useState('horizontal');\n  const themeContextValue = { layout };\n  const classes = useLayoutSwitcherStyles();\n\n  return (\n    <ThemeProvider value={themeContextValue}>\n      <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"vertical\">\n        <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"horizontal\">\n          <ToggleButton value=\"horizontal\" aria-label=\"Horizontal layout\">Horizontal Layout</ToggleButton>\n          <ToggleButton value=\"vertical\" aria-label=\"Vertical layout\">Vertical Layout</ToggleButton>\n        </ToggleButtonGroup>\n        <Grid container spacing={2} direction={layout}>\n          <Box className={classes.box}>\n            <Panel1 />\n          </Box>\n          <Box className={classes.box}>\n            <Panel2 />\n          </Box>\n        </Grid>\n      </ToggleButtonGroup>\n    </ThemeProvider>\n  );\n};\nexport default LayoutSwitcher;\n```\nThis example uses the `useTheme` hook to access MUI's theme and provides a responsive layout based on the current breakpoint. The `ThemeProvider` component wraps the entire UI, which allows access to the theme's values. The `layout` state is stored in a variable and can be either 'horizontal' or 'vertical'. The `ThemeProvider` component provides an access point to the theme by using a context value object that is passed down and managed through `useTheme`. The Grid component has responsive styles applied using breakpoints. This approach allows for fine-tuning the layout to various screen sizes, and you can swap between horizontal and vertical as needed.\n\nTo replace the imports with actual component locations, adapt spacing as needed, apply styles according to your requirements and adjust paths of imports. Remember that this code might not work properly if directly copied into your project, and may require some adjustments or additional settings. Use `npm install @mui/material` to ensure the MUI library is installed in your project.\nThis example serves as a starting point and you might need further adjustments or customizations to meet specific requirements. Don't forget to adapt the layout switching functionality according to your needs or preferences and ensure paths of imports, required dependencies are correctly configured.\nHere's an example with `useMediaQuery`:\n```javascript\nimport React, { useState } from 'react';\nimport Grid from '@mui/material/Grid';\nimport Box from '@mui/material/Box';\nimport ToggleButtonGroup from '@mui/material/ToggleButtonGroup';\nimport Panel1 from './Panel1'; // Replace this import with the location of your actual panel component\nimport Panel2 from './Panel2'; // Replace this import with the location of your actual panel component\nimport { useMediaQuery } from '@mui/material';\n\nconst LayoutSwitcher = () => {\n  const [layout, setLayout] = useState('horizontal');\n  const isSmallScreen = useMediaQuery('(max-width:640px)');\n\n  return (\n    <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"vertical\">\n      <ToggleButtonGroup value={layout} onChange={(e, layout) => setLayout(layout)} orientation=\"horizontal\">\n        <ToggleButton value=\"horizontal\" aria-label=\"Horizontal layout\">Horizontal Layout</ToggleButton>\n        <ToggleButton value=\"vertical\" aria-label=\"Vertical layout\">Vertical Layout</ToggleButton>\n      </ToggleButtonGroup>\n      <Grid container spacing={2} direction={layout}>\n        <Box component=\"div\">\n          {isSmallScreen ? (\n            <Panel1 />\n          ) : (\n            <Panel2 />\n          )}\n        </Box>\n      </Grid>\n    </ToggleButtonGroup>\n  );\n};\nexport default LayoutSwitcher;\n```\nThis example uses the `useMediaQuery` hook to determine screen size and switch between layouts accordingly. The `Panel1` component is displayed on small screens, while the `Panel2` component is visible for larger screens. The useMediaQuery hook can be used to determine the size of a screen, and it returns `true` for screens smaller than or equal to the specified breakpoint (640px in this example). This approach allows for fine-tuning the layout to various screen sizes and swapping between horizontal or vertical as needed.\nTo replace imports with actual component locations, adapt spacing as needed and apply styles according to your requirements. Remember that direct copying of this code might not work properly in your project, and may require additional adjustments or settings. Use `npm install @mui/material` to ensure the MUI library is installed in your project.\nThis example serves as a starting point and can help you achieve responsive layouts, but further adjustments or customizations might be needed to meet specific requirements. Ensure paths of imports, required dependencies are correctly configured and adapt layout switching functionality according to your preferences.\n```<|assistant|> Here's a more complete example using `useMediaQuery` and responsive breakpoints to switch between two panels using MUI:\n```javascript\nimport React, { useState } from 'react';\nimport Grid from '@mui/material/Grid';\nimport Box"
                }
              ],
              "defaultShouldIncludeInContext": true,
              "shouldIncludeInContext": true
            }
          ]
        }
      ],
      "currentlySelected": 0
    }
  ],
  "usePerChatPredictionConfig": true,
  "preset": "",
  "perChatPredictionConfig": {
    "fields": []
  },
  "tokenCount": 2938,
  "createdAt": 1728856104256,
  "pinned": false,
  "clientInput": "",
  "clientInputFiles": [],
  "userFilesSizeBytes": 0,
  "lastUsedModel": {
    "indexedModelIdentifier": "mlx-community/Phi-3.5-MoE-instruct-4bit",
    "identifier": "phi-3.5-moe-instruct-4bit",
    "instanceLoadTimeConfig": {
      "fields": []
    },
    "instanceOperationTimeConfig": {
      "fields": []
    }
  },
  "notes": []
}

I tried the CLI log stream, but only got this:

➜  ~ lms log stream

I Streaming logs from LM Studio

timestamp: 13/10/2024, 22:54:58
type: llm.prediction.input
modelIdentifier: phi-3.5-moe-instruct-4bit
modelPath: mlx-community/Phi-3.5-MoE-instruct-4bit
input: "<|user|>
Please implement a React component with MUI that includes toggling between horizontal and vertical interface between two panels.<|end|>
<|assistant|>
"

[daaain]

Oh wait, it might be a model issue: https://huggingface.co/microsoft/Phi-3.5-MoE-instruct/discussions/48