First input file is the only one processed

[mjh624]

Our input folder contains 11 files. All appear to be read in: Successfully read file: ./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/innovationqkb.WordPress.2024-07-26.xml.md JSON file saved successfully. Successfully read file: ./input/ipcomkb.WordPress.2024-07-26.xml.md JSON file saved successfully. Successfully read file: ./input/iqideaskb.WordPress.2024-07-26.xml.md JSON file saved successfully. Successfully read file: ./input/ipcomkb.faq.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/priorartdatabasekb.WordPress.2024-07-26.xml.md JSON file saved successfully. Successfully read file: ./input/iqideaskb.faq.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/innovationqkb.faq.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/priorartdatabasekb.glossary.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/iqideaskb.glossary.WordPress.2024-07-27.xml.md JSON file saved successfully. Successfully read file: ./input/priorartdatabasekb.faq.WordPress.2024-07-27.xml.md Pretraining set created.

However, only the first file: ./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md has question/answer pairs produced.

The augmentoolkit output messages do not appear to give an indication as to whether there is an issue. COMPLETED PHASE 0 Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/a0db9260-500e> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/a7bc5e7d-950c> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/bd42e735-6bba> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/c6ddcde3-8678> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/15a96815-222f> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/91488b07-8c85> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/f8913bd2-afb5> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/8ef1f903-2906> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/ee46ce5b-8461> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/86b05937-c2a0> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/1b6b185c-b3fd> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/0adb9229-3210> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/d51cf6a9-1745> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/921636fe-4076> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/456d0805-b8c7> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/f09a3fd9-3fcf> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/bf45b67b-81da> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/01b56723-abdf> Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/b89ac365-1286> FAILED TO GENERATE QUESTIONS! Output written to /tmp/augmentoolkit/original/output/question_generation_generations/question_generation_generations/1f0d3583-f9e7> COMPLETED PHASE 1

Each file written in phase 1 appears to correspond to questions/answers related to a paragraph in the document: ./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md

What are some possible reasons that files in the input folder are skipped?

[mjh624]

A check of the metadata field in the qa_tuples_filtered folder shows only the first file lead to question/answer pairs:

grep -r metadata qatuples_filtered qatuples_filtered/para_6_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_5.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_0_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_19_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_18_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_19_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_6.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_6_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_4_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_15_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_3_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_12_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_12_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_6_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_5.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_3_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_2_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_18_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_3_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_1_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_0_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_6_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_19_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_18_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_4_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_12_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_1_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_3_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_0_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_15_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_19_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_6.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_2_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_2_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_18_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_2_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_15_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_8_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_1_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_6.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_5.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_9_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_1_q_3.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_6_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_0_q_4.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_1_q_0.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_7_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_0_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_3_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_4_q_2.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md", qatuples_filtered/para_15_q_1.json: "metadata": "./input/innovationqkb.glossary.WordPress.2024-07-27.xml.md",

[e-p-armstrong]

Hmm that's strange. I'm not able to reproduce this using the toy example the repo starts with, so that leaves a few possibilities: 1) we're running into an edge case with the code that isn't triggered with the three default input files 2) somehow everything sourced from the other files is failing validation and never gets to question generation 3) All questions made from those files get dropped for some reason 4) something else

Would you be against sharing your input files and maybe your config so I can try to repro it on my end, or is that stuff confidential?

[mjh624]

Here is the config file: API: API_KEY: xxxx BASE_URL: http://localhost:11434/v1
LARGE_LOGICAL_MODEL: llama3.1:70b LOGICAL_MODEL: llama3.1:70b HUGGINGFACE: HUB_PATH: < our info here > PRIVATE: False PUSH_TO_HUB: False PATH: DEFAULT_PROMPTS: ./prompts INPUT: ./input OUTPUT: ./output PROMPTS: ./prompts PHASE: PHASE_INDEX: 3 WORK_IN_PHASES: False SKIP: ANSWER_RELEVANCY_CHECK: False FILTER_CHUNKS: True QUESTION_CHECK: False SYSTEM: CHUNK_SIZE: 1900 COMPLETION_MODE: False CONCURRENCY_LIMIT: 3 CONVERSATION_INSTRUCTIONS: For this conversation, you are generating a chat between a generalist, generic AI assistant, and a human. DOUBLE_CHECK_COUNTER: 1 DO_NOT_USE_SYSTEM_PROMPTS: True FINAL_ASSISTANT_PROMPT_NO_RAG: 'You are a helpful AI assistant.

'

FINAL_ASSISTANT_PROMPT_RAG: 'You are a helpful AI assistant.

Context information is below:

----------------------

{data}

'

MODE: api STOP: True SUBSET_SIZE: 15 USE_FILENAMES: False USE_SUBSET: False

Unfortunately, I cannot share our input files.

I found the config file that was used to process army training manuals:

https://github.com/e-p-armstrong/augmentoolkit/blob/master/original/config_overrides/army_model/config.yaml

I modified the config to use our input files and model and it appears that most, if not all input files now are being processed. However, processing started 10/1/2024 and after 4 days, it is still processing. I would like to understand which settings may have allowed the other files to process, and, why is it taking so much longer.