Elasticsearch fulltext report

adsabs / AutomatedIngestReport

Gather data and create report on the state of ingest data.

MIT License

0 stars 1 forks source link

Elasticsearch fulltext report #5

Closed krisbukovi closed 4 years ago

krisbukovi commented 4 years ago

Update fulltext section of AIR to handle inconsistent extraction times as well use elastic search instead of log files.

Benefits of this PR:

No concern of losing logs because of the 10MB cap on storage we have for log files
Better parsing using regex that is not as manual as previous version
Report can be run at anytime without concern of the fulltext pipeline processing schedule

spacemansteve commented 4 years ago

I like the clear PR summary and the ability to run report anytime. Does it make any sense to modify the test code?

krisbukovi commented 4 years ago

Yes it does! I added a unit test for fulltext compute, let me know what you think.

side note: A unit test for computing solr errors is failing, but I didn't make any edits to solr or any of the unit tests besides the one I made for fulltext. Is this because I am missing the "solrUpdates" scripts that I see in your repo on adsqb?