Closed hunterhacker closed 12 years ago
Have support for storing and retrieving binary documents. Can also retrieve binary documents that are managed outside of Corona. Still need to support: searching binary documents and automatic extraction of content and metadata from binaries.
Binary documents are now searchable as well.
When extracting out the content of a binary file, what should be done when said content is extremely large? For example, say someone inserts a PDF file that's 10,000 pages of pure text. It's likely that the server won't be configured with enough resources to handle that large of an XML document. Should we truncate the file?
What docs the standard CPF pipeline do? My guess is it assumes it can handle anything, and lets the underlying memory error propagate up if it's wrong.
Which resources are you concerned about exactly, Ryan? If memory, truncation will be hard because it needs to be in memory first before truncation.
Good point. Specifically I was thinking about the in-memory stand size. But if we'd like to just let things fly and allow resource limits to crop up, I'm okay with that. If it proves to be too annoying we can come up with options to help alleviate the problem.
Our binary support now includes inserting, fetching, updating binary documents. Users can also include their own document (XML or JSON) representation of the binary document. If available, metadata and content will be extracted from the binary. This extraction can be turned off. Users can search the extracted content via the wordInBinary structured query constraint.
It would be good to support storing binary documents.
It would also be good to optionally extract metadata and text content from them and make it available for query.