How can I add custom embedding model when running web

[rjwharry]

I have custom huggingface embedding model in my pipeline.yaml.
When running as api server, I can add my custom model like below

autorag.embedding_models["custom model"] = LazyInit(
    HuggingFaceEmbedding, model_name="custom_model"
)
runner = Runner.from_yaml("./6/pipeline.yaml")
runner.run_api_server()

But run_web support cli only, so I can't add custom model like above.
How can I add custom model? Is there any reason why run_web is not supported by the Python SDK, like run_api_server()?

[vkehfdl1]

@bwook00 You might can help this issue.

@rjwharry Great point! I think there is no docs for this situation.

First, we might do experiment and leave a comment here again. Thanks. (We missed this situation)

[rjwharry]

If you're willing to work on this issue once you've decided, can I contribute?
Seems like a good fit for a “good first issue”.

[vkehfdl1]

@rjwharry That will be lovely. I will assign you to assignee.

[vkehfdl1]

There is a streamlit file at autorag/web.py At cli.py we execute the streamlit with streamlit run web.py command. Because the only thing to execute streamlit was cli, we made just cli method. You might find a way to start streamlit without using cli.

[rjwharry]

I have some suggestions and requests

Request for style guide

If you have any documentation for the style guide or the '.pylintrc' file, please let me know. My preferences are quite different from yours.

Suggestions

I'm working on this issue and I noticed that there is a limitation in implementing it with streamlit. Even if I set up a new custom LLM or embedding model, the streamlit application will not recognize the custom model because the streamlit application has to run in a different process.
So here is my suggestion. Why don't you use Gradio instead? I think it is a good alternative to streamlit and I can run it in the same process, so Gradio knows my custom model.
I have already tested it on my local and it works fine, so I hope you will consider this. Here's a screenshot of my test

[rjwharry]

In the process of changing to gradio, I ran into an issue with event loops. When I run gradio and run the pipeline, all of the loop.get_event_loop code throws an error saying: There is no current event loop in thread 'AnyIO worker thread'.
I thought the cause of this error was that there was no event loop running, as shown in the logs, so I changed that code to asyncio.run() so that it can be automatically generated when the event loop is not present and now I don't get the error. Consequently I changed all modules like below.

# original
loop.get_event_loop()
results = loop.run_until_complete(process_batch(tasks, batch_size=batch))

# new
results = asyncio.run(process_batch(tasks, batch_size=batch))

I'm testing to make sure this fix works. I'd appreciate it if you could share your ideas.

[vkehfdl1]

I have some suggestions and requests

Request for style guide

If you have any documentation for the style guide or the '.pylintrc' file, please let me know. My preferences are quite different from yours.

Suggestions

I'm working on this issue and I noticed that there is a limitation in implementing it with streamlit. Even if I set up a new custom LLM or embedding model, the streamlit application will not recognize the custom model because the streamlit application has to run in a different process. So here is my suggestion. Why don't you use Gradio instead? I think it is a good alternative to streamlit and I can run it in the same process, so Gradio knows my custom model. I have already tested it on my local and it works fine, so I hope you will consider this. Here's a screenshot of my test

1. I made issue for that #621 Thanks for the suggestion.
2. Gradio will be great. Just please keep in mind users can select between two of them and it is not becoming too much complex.

[vkehfdl1]

@rjwharry Maybe gradio is using anyio, which is not asyncio. So the library itself have some conflict at asyncio event loop.

But yeah, it looks like when you changed all asyncio.get_event_loop() to asyncio.run, it looks like there is no error from now. So it will be great you can fix this.

[vkehfdl1]

@rjwharry Can you make a draft PR to our repo? It will be great to see your code 👍

[rjwharry]

@vkehfdl1 During test, I got same error(There is no current event loop in thread 'AnyIO worker thread') for specific case. So let me research more about this, and then I'll share you with draft PR as soon as possible

[vkehfdl1]

@rjwharry Okay thanks. I hope we can find some way that gradio can use asyncio not anyio

Plus, at #622, I integrate ruff Linter and reformat all codes. Maybe the code style will be consistent.

[vkehfdl1]

It might be have conflict when merging between two because of all reformat.. So be careful...

I think it is better you get ruff setting and reformat first before merge this. The setting is in pyproject.toml And use below code.

ruff check --fix ruff format