LuminosoInsight / sales-engineering-code

Code for sales engineering, particularly for code that will be given to customers
MIT License
0 stars 0 forks source link

doc_downloader requires sentiment built to download #188

Open wharter opened 11 months ago

wharter commented 11 months ago

doc_downloader.py currently requires that sentiment be built on a project to download all the documents even if the script isn't using the sentiment data.

https://github.com/LuminosoInsight/sales-engineering-code/blob/9a47c787ced5892d13540d5c636467b3eb272f83/se_code/doc_downloader.py#L13

The issue is that we only need the include_sentiment_on_concepts flag if the user specified --concept_relations_sentiment parameter. If the user just wants to download the documents, it shouldn't require that sentiment be built.

I think the change looks something like this in the function get_all_documents, then where you call get_all_documents, pass in the whether the --concept_relations_sentiment is True/False.

def get_all_docs(client, include_sentiment_on_concepts):
    docs = []
    while True:
        new_docs = client.get(
            "docs", include_sentiment_on_concepts=inclue_sentiment_on_concepts, limit=2500, offset=len(docs)
        )
        if new_docs["result"]:
            docs.extend(new_docs["result"])
        else:
            return docs