Add Error Reporting for ES Inserts + Add URL Keywords to Report

[tfnribeiro]

• Improved the error reporting in the bulk inserts by printing the errors found and the total Success + Errors
• Added the URL keywords to report where any URL keywords seen more than 100 times with no Topic assigned will be shown in the table.

Errors might have been due to the attempt to index broken articles with empty fields, which fail when adding to ES. To avoid these situations, I also filter the broken articles in the query now. I expect less errors now, but if we do we should get them printed out with some information about what was the cause.

[github-actions[bot]]

Module view diffs:

[mircealungu]

Comment suffices!

On Mon, 21 Oct 2024 at 13.03, Tiago Ribeiro @.***> wrote:

@.**** commented on this pull request.

In tools/mysql_to_elastic_new_topics.py https://github.com/zeeguu/api/pull/261#discussion_r1808471605:

@@ -26,8 +26,8 @@ DELETE_INDEX = False

First we should only index with topics so we can do

inference based on the articles that have topics.

-INDEX_WITH_TOPIC_ONLY = False -TOTAL_ITEMS = 50000 +INDEX_WITH_TOPIC_ONLY = True +TOTAL_ITEMS = 10000

Because I didn't want to run it for all documents in my environment. I can add a comment that number should be adjusted as needed, I guess it could even be made into a system argument.

— Reply to this email directly, view it on GitHub https://github.com/zeeguu/api/pull/261#discussion_r1808471605, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADRNB3THHI5EUSCV5PRIEDZ4TGQZAVCNFSM6AAAAABQJVSJ72VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGOBRGU2DGNJXG4 . You are receiving this because you modified the open/close state.Message ID: @.***>