search

spyglass-search / spyglass

A personal search engine: Create a searchable library from your personal documents, interests, and more!
https://docs.spyglass.fyi
GNU Affero General Public License v3.0
2.55k stars 58 forks source link

Crawling private sites that require login #306

Open haydenflinner opened 1 year ago

haydenflinner commented 1 year ago

It would be nice to be able to index work sites like Confluence, Jira, and internal documentation sites. Is there any way to configure the crawler with my login cookies? Or add the crawler as a plugin to my Firefox so that it just sort of always works rather than needing the cookies updated daily?

a5huynh commented 1 year ago

Hey @haydenflinner, we recently added a Firefox & Chrome add-on that syncs with your bookmarks and you can manually add a single page with a button. Let me know if that fits what you're looking for?

Are you looking to index the entire internal wiki? If so, we'd need to add in an integration to handle something like that.

haydenflinner commented 1 year ago

Hey @haydenflinner, we recently added a Firefox & Chrome add-on that syncs with your bookmarks and you can manually add a single page with a button. Let me know if that fits what you're looking for?

Awesome! Thank you for this, it will be useful as-is for a few regular queries!

Are you looking to index the entire internal wiki? If so, we'd need to add in an integration to handle something like that.

Indeed, I'm mainly looking for something like a locally-hosted version of Needl, where I can self-host a search engine across all of the internal data that I care about and have access to. JIRA, Slack, Confluence, typical enterprise stuff.