search

VauxIo / core

Document Liberation Network
MIT License
4 stars 1 forks source link

Reduce Redundant Work #4

Open citruspi opened 9 years ago

citruspi commented 9 years ago

CC: @rossdylan @thequbit

At this point every node does all the work - when a node ingests a file it

In the future, this work will also include generating a preview of the content.

Ideally, this redundant work load will be reduced - once a node has performed metadata extraction and preview generation, each of its peers should be able to make use of the metadata and preview.

One idea for how this would work...

Uploading a file simply means making a POST request to the /documents/ endpoint. The name for the file input is simply file. If only file is defined, the node would perform metadata extraction and preview generation. However, if file, preview, and metadata were also included in the form, the node would use the included preview and metadata. So initial uploads would simply include the file. Once the node performs the initial ingestion, it would POST the file, and the metadata and preview, to its peers.

— @citruspi

thequbit commented 9 years ago

100% agree. The cost of conversation can be super high. Symantics, but I would suggest tar.gz'ing the thumbnails when POSTing. You can just include the converted text as a field within the POST I would think.