scrape a list of URLs from a text file / text box (many json items in one json file)

dodeeric / langchain-ai-assistant-with-hybrid-rag

This is a LLM chatbot coded with LangChain. The web interface is coded with Streamlit. It implements a hybrid RAG (keyword and semantic search) and chat memory.

https://bmae-ragai-webapp.azurewebsites.net

GNU General Public License v3.0

8 stars 1 forks source link

scrape a list of URLs from a text file / text box (many json items in one json file) #41

Closed dodeeric closed 2 weeks ago

dodeeric commented 1 month ago

if not a list of urls from a text files, then a text box (copy/paste urls from the text file)

the present url field will become an urls text field (with 1 or more urls)

which name to give to the file?

collectionname-firsturldnspart-YYYYMMDDHHMM-swp.json

two possibilities:

1 json file per URL
1 json file for all the URLs

dodeeric commented 2 weeks ago

in streamlit, how to enter a list of URLs in a text box, then use each URL one by one? https://chatgpt.com/share/b1c84aa2-38e5-449b-9154-f9489596c94d

dodeeric commented 2 weeks ago

1 json file per url