Indexing file content in ElasticSearch

[ducoquelicot]

Issue The search portion works, with mock data. But what I haven't created yet is a way to index the .txt files (or .pdfs) that are the public records that I want to be able to search.

Thoughts

• Should I only index the content in Elasticsearch or index it into the SQLite database as well?
• For the ES index, I want to add metadata about the city, document type, and the id - which should be the same ID as the ID in the SQLite database, otherwise search doesn't work.
• For the SQLite index, I want to add metadata about the city, document type, date, and record name [Question should the content of the file be in the SQLite database as well? Might simplify the workflow]

Ideas

• I have looked at Logstash but I found it to be pretty confusing
• I have also looked at the Ingest plugin from ES again, but that's also still confusing me
• I have tried to index the content of one of the files that I have [just copy + pasted into a Kibana search] and it throws an error. Error is as follows: "Illegal unquoted character ((CTRL+CHAR, code 10)): has to be escaped using backslash to be included in string value \n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@6b3825e5; line: 4; column: 64]"

I'm guessing the above error is that there's a character in the document that throws the string formatting out of whack, but I can't check all the files manually for those characters so either I have to write a script that ensure the characters are processed in the right way, or I have to figure out something else.

Let me know!

[zstumgoren]

Hey, First, HAPPY BIRTHDAY!

And some thoughts on the questions:

• Should be fine to insert content (metadata and extracted text) into both db and ES, especially if that simplifies the Flask integration
• Don't think Logstash or Ingest plugin are needed. Let's stick with the ORM-layer integration for CRUD operations, following the strategy detailed in the Flask Megatutorial
• When indexing multi-line content through Kibana, you have to ensure the JSON document that you're sending is properly escaped (as the error message suggests). You could do this perhaps most easily by pre-generating the JSON using a Python script and then transferring the query to Kibana. But as we discussed, this might be more trouble than it's worth for a large document. Further, it seems like you're at the point where it makes sense to start working at the Python layer, which should handle this more gracefully. And that will set you up to begin integration into the flask layer. I'd recommend starting with a simple, stand-alone Python script that just inserts one document. That should let you work through the kinks. Once that's figured out, you should be able to begin Flask integration following the roadmap laid out in the tutorial.

[ducoquelicot]

Thanks!

I think the problem is not so much in indexing in ES (I figured that out) but writing a script that will automatically index all the necessary information into the SQLite database. That part isn't covered by the Flask tutorial because he only indexes posts, which are written on the website, so that goes through a form of sorts - and we talked about that last week.

So far so good though, I was able to update the txt files into ES. So indexing into the SQLite database is the next step.

[zstumgoren]

Aha, ok, I get it now. So the secret ingredient is the SQLAlchemy ORM layer. Once you've integrated the CRUD operations for Elastic into the database model(s), you can use those models in any context. In a Flask web request context, they'd likely be used in one of your routes (perhaps when a user uploads a document manually, if you decide to support that feature). In a similar fashion, you can also write a command-line script that imports those same models and inserts documents. The code logic will likely be highly similar if not identical, so you'll be setting the stage for flask app integration by implementing the command-line client.