Push Files to Index from Obsidian, Emacs & Desktop Clients using Multi-Part Forms Method

[debanjum]

Overview

• Add ability to push data to index from the Emacs, Obsidian clients
• Switch to standard mechanism of syncing files via HTTP multi-part/form. Previously we were streaming the data as JSON
  • Benefits of new mechanism
  • No manual parsing of files to send or receive on clients or server is required as most have in-built mechanisms to send multi-part/form requests
  • The whole response is not required to be kept in memory to parse content as JSON. As individual files arrive they're automatically pushed to disk to conserve memory if required
  • Binary files don't need to be encoded on client and decoded on server

Code Details

Major

• 60e9a61 Use multi-part form to receive files to index on server
• 68018ef Use multi-part form to send files to index on desktop client
• fc99431 Send files to index on server from the khoj.el emacs client
  • 292f042 Send content for indexing on server at a regular interval from khoj.el
• f2e293a: Send files to index on server from the khoj obsidian client
• bed3aff 6baaaaf Update tests to test multi-part/form method of pushing files to index

Minor

• 6aa69da Put indexer API endpoint under /api path segment
• bea196a Explicitly make GET request to /config/data from khoj.el:khoj-server-configure method
• 9ba173b Improve emoji, message on content index updated via logger
• 7b9161e Don't call khoj server on khoj.el load, only once khoj invoked explicitly by user
• Improve indexing of binary files
  • 541cd59 Let fs_syncer pass PDF files directly as binary before indexing
  • d27dc71 Use encoding of each file set in indexer request to read file
• 99a2c93 Add CORS policy to khoj server. Allow requests from khoj apps, obsidian & localhost
• 84654ff Update indexer API endpoint URL toindex/update from indexer/batch

Resolves #471 #243