unraid app development

[Steel-skull]

was wanting to make sure your cool with using and forking your repo to make a unraid community app

[Atinoda]

Definitely cool with that - the more ways for people to easily install the better! Is there anything you'd like to share back to this repo, or any minor change requests here that would help with the Unraid app?

[Steel-skull]

Awesome and not yet I'm gonna look into this more soon but if I have anything I'll let you know!

[simplesisu]

@Steel-skull Hope all is well! Really looking forward to the community app. How is it going for you?!

Br,

[SurvivaLlama]

Any update to the community app?

[RTStump]

Definitely cool with that - the more ways for people to easily install the better! Is there anything you'd like to share back to this repo, or any minor change requests here that would help with the Unraid app?

I know this was from a while ago, but changing the volume mapping where dependencies and such are installed in order to prevent the install from blowing up the Docker image size would be great. Right now it's adding roughly 15 GB to people's docker images with default configuration.

[Atinoda]

@RTStump Thanks for the suggestion! I have an extensive refactor intended for the near future - and I would definitely like to slim down the image, in addition to creating some more contemporary variants. Could you please provide more details on what folders are blowing up the size? For instance, are you referring to extension dependencies or other post-install actions?

[Steel-skull]

Sorry for never getting back on this I've been working on family issues and haven't been able to work on this.

[Steel-skull]

<?xml version="1.0"?>
<Container version="2">
  <Name>text-generation-webui</Name>
  <Repository>atinoda/text-generation-webui:latest</Repository>
  <Registry/>
  <Network>bridge</Network>
  <MyIP/>
  <Shell>sh</Shell>
  <Privileged>false</Privileged>
  <Support/>
  <Project/>
  <Overview/>
  <Category/>
  <WebUI>http://[IP]:[PORT:7860]</WebUI>
  <TemplateURL/>
  <Icon/>
  <ExtraParams>--runtime=nvidia</ExtraParams>
  <PostArgs/>
  <CPUset/>
  <DateInstalled>1709426164</DateInstalled>
  <DonateText/>
  <DonateLink/>
  <Requires/>
  <Config Name="WebUI" Target="7860" Default="7860" Mode="tcp" Description="" Type="Port" Display="always" Required="true" Mask="false">7860</Config>
  <Config Name="Open AI API" Target="5000" Default="5000" Mode="tcp" Description="" Type="Port" Display="always" Required="true" Mask="false">5000</Config>
  <Config Name="Characters" Target="/app/characters" Default="./config/characters" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/disk1/docker/compose/text-generation-webui-docker/config/characters/</Config>
  <Config Name="Loras" Target="/app/loras" Default="./config/loras" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/loras/</Config>
  <Config Name="Models" Target="/app/models" Default="./config/models" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/models/</Config>
  <Config Name="Presets" Target="/app/presets" Default="./config/presets" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/presets/</Config>
  <Config Name="Prompts" Target="/app/prompts" Default="./config/prompts" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/prompts/</Config>
  <Config Name="Training" Target="/app/training" Default="./config/training" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/training/</Config>
  <Config Name="Extensions" Target="/app/extensions" Default="./config/extensions" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/docker/compose/text-generation-webui-docker/config/extensions/</Config>
  <Config Name="EXTRA_LAUNCH_ARGS" Target="EXTRA_LAUNCH_ARGS" Default="" Mode="" Description="" Type="Variable" Display="always" Required="false" Mask="false">"--listen --verbose --api"</Config>
  <Config Name="NVIDIA_VISIBLE_DEVICES" Target="NVIDIA_VISIBLE_DEVICES" Default="all" Mode="" Description="" Type="Variable" Display="always" Required="false" Mask="false">all</Config>
  <Config Name="NVIDIA_DRIVER_CAPABILITIES" Target="NVIDIA_DRIVER_CAPABILITIES" Default="all" Mode="" Description="" Type="Variable" Display="always" Required="false" Mask="false">all</Config>
</Container>

template for unraid to load web-ui with --api

sorry for the delay life got in the way

@Atinoda

[Atinoda]

@Steel-skull apologies for my delayed reply - thank you for sharing your template! I have it on my list to include it in the repo somewhere, or perhaps to submit to the community repository. However, I need to spin up an Unraid VM or put it on some spare hardware to understand the OS better before I can properly support it.

[Steel-skull]

@Steel-skull apologies for my delayed reply - thank you for sharing your template! I have it on my list to include it in the repo somewhere, or perhaps to submit to the community repository. However, I need to spin up an Unraid VM or put it on some spare hardware to understand the OS better before I can properly support it.

Hey man, no prob! Sorry for the delay in getting it done. I can probably help with that as I already tinker with the container's and unraid on the regular, I was looking into submitting it to the community store altho I've been putting off looking into the submission process. I'll probably do that this weekend.

[Atinoda]

Awesome - give us a heads up if you get it published to the community store and I'll add a link to it in the README.md!

P.S. I also had my planned refactor get backburnered for a few months by family life - but I think that just means you're prioritising the right things :)

[Basicallyapenguin]

Hello any update on this? Would love for this to be in CA for unraid!

[Steel-skull]

Sadly, I've stopped using and working on it as updates from webui kept breaking the container on the regular, and there was no fix in site, so I moved to Ollama for easier deployment.

Not to mention, one update crashed my server so hard I had to reinstall unraid on a separate usb drive as it corrupted the old drive.

However, I'll take a look and see if it was ever fixed.

[Basicallyapenguin]

Okay no worries, thanks for the response!

[Atinoda]

I'm planning a big update to the container this week to bring it back in line with the upstream project and close off a few feature requests. If I have the time, I'll also look at what's involved in publishing an Unraid version.

@Steel-skull - sorry to hear that you had issues with it. I've also been experimenting with ollama as a back-end for a new project... I don't see how this (or any other container) could brick your server though - that's kind of the point of docker container sandboxes! Barring sensitive bind mounts or resource starvation, of course. Have you had specific issues with the stable versions? I would really appreciate you sharing them so I can investigate and fix for the big update.

[Steel-skull]

I'm planning a big update to the container this week to bring it back in line with the upstream project and close off a few feature requests. If I have the time, I'll also look at what's involved in publishing an Unraid version.

@Steel-skull - sorry to hear that you had issues with it. I've also been experimenting with ollama as a back-end for a new project... I don't see how this (or any other container) could brick your server though - that's kind of the point of docker container sandboxes! Barring sensitive bind mounts or resource starvation, of course. Have you had specific issues with the stable versions? I would really appreciate you sharing them so I can investigate and fix for the big update.

Good to hear, Ive been playing with making an Unraid container again as well and have a functional container running.

Honestly I thought the same thing lol I personally think it was a complete driver failure that led to corruption but with out the logs from the failure its hard to diagnose.

id say no specific issues currently with the container but im testing it further in the next few days.

Current working Unraid docker config:

<?xml version="1.0"?>
<Container version="2">
  <Name>text-generation-webui-test</Name>
  <Repository>atinoda/text-generation-webui</Repository>
  <Registry>https://hub.docker.com/r/atinoda/text-generation-webui/</Registry>
  <Network>bridge</Network>
  <MyIP/>
  <Shell>sh</Shell>
  <Privileged>false</Privileged>
  <Support>https://hub.docker.com/r/atinoda/text-generation-webui/</Support>
  <Project/>
  <Overview>Oobabooga's Text Generation Web UI - a gradio web UI for running Large Language Models (LLMs).&#xD;
&#xD;
Container made by Atinoda&#xD;
https://hub.docker.com/r/atinoda/text-generation-webui/&#xD;
&#xD;
Cuda: "--runtime=nvidia --gpus=all"</Overview>
  <Category/>
  <WebUI>http://[IP]:[PORT:7860]</WebUI>
  <TemplateURL/>
  <Icon/>
  <ExtraParams>--runtime=nvidia --gpus=all </ExtraParams>
  <PostArgs/>
  <CPUset/>
  <DateInstalled>1721098068</DateInstalled>
  <DonateText/>
  <DonateLink/>
  <Requires/>
  <Config Name="Container Port 1" Target="5000" Default="5000" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">5000</Config>
  <Config Name="Container Port 2" Target="5005" Default="5005" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">5005</Config>
  <Config Name="Container Port 3" Target="7860" Default="7860" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">7860</Config>
  <Config Name="Container Variable 1" Target="VIRTUAL_ENV" Default="/venv" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">/venv</Config>
  <Config Name="Container Variable 2" Target="PYTHONUNBUFFERED" Default="1" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">1</Config>
  <Config Name="Container Variable 5" Target="EXTRA_LAUNCH_ARGS" Default="" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">--listen --verbose --api</Config>
  <Config Name="Cashe" Target="/root/.cache" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/.cashe</Config>
  <Config Name="Characters" Target="/app/characters" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/characters</Config>
  <Config Name="Instruction-Templates" Target="/app/instruction-templates" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/instruction-templates</Config>
  <Config Name="Loras" Target="/app/loras" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/loras</Config>
  <Config Name="Models" Target="/app/models" Default="" Mode="rw" Description="I reccomend changing this to a spot on your array not cashe drive" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/models</Config>
  <Config Name="Prompts" Target="/app/prompts" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/prompts</Config>
  <Config Name="Training" Target="/app/training" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/training</Config>
  <Config Name="Extensions" Target="/app/extensions" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/extensions</Config>
</Container>

ill update again once im able to confirm its all working no issue

[Steel-skull]

@Atinoda ok so no big issue with the template, i am however receiving this:

Traceback (most recent call last): File "/app/modules/callbacks.py", line 61, in gentask ret = self.mfunc(callback=_callback, args, self.kwargs) File "/app/modules/text_generation.py", line 393, in generate_with_callback shared.model.generate(kwargs) File "/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, **kwargs) File "/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1758, in generate result = self._sample( File "/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2437, in _sample next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1) RuntimeError: probability tensor contains either inf, nan or element < 0

when i prompt with api through sillytavern regardless of model

[nwithan8]

I'm planning a big update to the container this week to bring it back in line with the upstream project and close off a few feature requests. If I have the time, I'll also look at what's involved in publishing an Unraid version. @Steel-skull - sorry to hear that you had issues with it. I've also been experimenting with ollama as a back-end for a new project... I don't see how this (or any other container) could brick your server though - that's kind of the point of docker container sandboxes! Barring sensitive bind mounts or resource starvation, of course. Have you had specific issues with the stable versions? I would really appreciate you sharing them so I can investigate and fix for the big update.

Good to hear, Ive been playing with making an Unraid container again as well and have a functional container running.

Honestly I thought the same thing lol I personally think it was a complete driver failure that led to corruption but with out the logs from the failure its hard to diagnose.

id say no specific issues currently with the container but im testing it further in the next few days.

Current working Unraid docker config:

<?xml version="1.0"?>
<Container version="2">
  <Name>text-generation-webui-test</Name>
  <Repository>atinoda/text-generation-webui</Repository>
  <Registry>https://hub.docker.com/r/atinoda/text-generation-webui/</Registry>
  <Network>bridge</Network>
  <MyIP/>
  <Shell>sh</Shell>
  <Privileged>false</Privileged>
  <Support>https://hub.docker.com/r/atinoda/text-generation-webui/</Support>
  <Project/>
  <Overview>Oobabooga's Text Generation Web UI - a gradio web UI for running Large Language Models (LLMs).&#xD;
&#xD;
Container made by Atinoda&#xD;
https://hub.docker.com/r/atinoda/text-generation-webui/&#xD;
&#xD;
Cuda: "--runtime=nvidia --gpus=all"</Overview>
  <Category/>
  <WebUI>http://[IP]:[PORT:7860]</WebUI>
  <TemplateURL/>
  <Icon/>
  <ExtraParams>--runtime=nvidia --gpus=all </ExtraParams>
  <PostArgs/>
  <CPUset/>
  <DateInstalled>1721098068</DateInstalled>
  <DonateText/>
  <DonateLink/>
  <Requires/>
  <Config Name="Container Port 1" Target="5000" Default="5000" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">5000</Config>
  <Config Name="Container Port 2" Target="5005" Default="5005" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">5005</Config>
  <Config Name="Container Port 3" Target="7860" Default="7860" Mode="tcp" Description="" Type="Port" Display="always" Required="false" Mask="false">7860</Config>
  <Config Name="Container Variable 1" Target="VIRTUAL_ENV" Default="/venv" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">/venv</Config>
  <Config Name="Container Variable 2" Target="PYTHONUNBUFFERED" Default="1" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">1</Config>
  <Config Name="Container Variable 5" Target="EXTRA_LAUNCH_ARGS" Default="" Mode="{3}" Description="" Type="Variable" Display="always" Required="false" Mask="false">--listen --verbose --api</Config>
  <Config Name="Cashe" Target="/root/.cache" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/.cashe</Config>
  <Config Name="Characters" Target="/app/characters" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/characters</Config>
  <Config Name="Instruction-Templates" Target="/app/instruction-templates" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/instruction-templates</Config>
  <Config Name="Loras" Target="/app/loras" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/loras</Config>
  <Config Name="Models" Target="/app/models" Default="" Mode="rw" Description="I reccomend changing this to a spot on your array not cashe drive" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/models</Config>
  <Config Name="Prompts" Target="/app/prompts" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/prompts</Config>
  <Config Name="Training" Target="/app/training" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/training</Config>
  <Config Name="Extensions" Target="/app/extensions" Default="" Mode="rw" Description="" Type="Path" Display="always" Required="false" Mask="false">/mnt/user/appdata/text-webui/extensions</Config>
</Container>

ill update again once im able to confirm its all working no issue

Well bummer, that's what I get for waiting too long to get around to making an Unraid template for the project (although it looks like this has been in the works for a while). When you're ready to publish @Steel-skull , if you don't have your own repo in the CA store, feel free to submit it to the SelfHosters repo: https://github.com/selfhosters/unRAID-CA-templates