s3 storage for the index files

sourcegraph / doctree

First-class library documentation for every language (based on tree-sitter), with symbol search & more. Lightweight single binary, run locally or self-host. Surfaces usage examples via Sourcegraph.

https://doctree.org

Apache License 2.0

872 stars 25 forks source link

s3 storage for the index files #25

Open jippi opened 2 years ago

jippi commented 2 years ago

Hey!

Would be fantastic if doctree serve could use s3 as filesystem backend - would allow me to use doctree index in CI and upload the files to s3 where they could be served, allowing a central highly-available doctree frontend for all projects

Maybe via a filesystem abstraction layer like https://github.com/usmanhalalit/gost , https://github.com/lytics/cloudstorage , https://github.com/graymeta/stow

Note: my use-case only covers frontend part, assuming i can s3 upload the indexed file to the right path and it would just work

slimsag commented 2 years ago

Thanks for filing this! I can't say for sure right now if this will make sense or not. The reason being: search.

I really need to figure out how to do good/nice search indexing for the docs, and that will likely involve leveraging local SSD performance (otherwise it'd require a lot of memory.)

I do think finding a way for this to be super simple to deploy, easily turned into an HA service, and (most importantly) easy to keep your projects indexed is extremely important though.

I'll keep this issue open and revisit once I have a better idea of how search in doctree will look / be implemented.