Fewer results than expected when running ontogpt run_multilingual_analysis

[justaddcoffee]

When running multilingual analysis, we are observed many fewer results than expected (less than 1000 when running about 5000 prompts)

For example, when I run this command on this directory of 5267 prompts:

~/PythonProject/malco/data/all/prompts/en/ $ ls -1 | wc -l 
    5267

I only see 758 .result files in the output directory

en/differentials_by_file $ ls -1 *result | wc -l 
     758

and I only see 758 entries in results.yaml:

/en  $ grep -c '\-\-\-' results.yaml 
758

One possible explanation is here: maybe there are many API failures that are being silently ignored?

@caufieldjh what do you think - maybe we could walk through this if you have time?

[caufieldjh]

Yes, let's look into this. Does that "OPENAI API CALL FAILED" string appear anywhere in the output?

[justaddcoffee]

Thanks @caufieldjh

Yes, let's look into this. Does that "OPENAI API CALL FAILED" string appear anywhere in the output?

No, not that I can see - the code linked above wouldn't actually output anything AFAICT

[caufieldjh]

OK, I think the issue is that the OpenAIClient catches all exceptions upstream of this operation: https://github.com/monarch-initiative/ontogpt/blob/0b1031d7e17dc49d3bda5aefaf702ce16b1284cb/src/ontogpt/clients/openai_client.py#L98 I think the API client object should be handling all error handling here, but I'll have it write to an error log.

[caufieldjh]

On second thought I'm just going to focus on getting the error handling in place for the multilang analysis for now, as I think the other half of the issue is in malco.

[justaddcoffee]

thanks @caufieldjh!