system message being included in chosen & rejected when chat_template inserts system message

[dctanner]

Some models include a chat template that inserts a default system message. E.g: {% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful assistant.' %}{% endif %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{{'<|im_start|>system\n' + system_message + '<|im_end|>\n'}}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}

The problem is that this causes the system message to be included into the chosen and rejected samples. E.g:

2024-02-06 14:26:39 - INFO - __main__ - Chosen sample 204 of the raw training set:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>assistant
There could be a number of reasons why a green star polyp coral is not opening. Some common causes include stress from water quality issues, lack of food, or disease. It's also possible that the coral is simply in a resting phase. If you are concerned about the health of your coral, it's recommended to consult with an aquarium specialist or marine biologist who can help diagnose and address the issue.<|im_end|>

Another issue is that the current logic to insert a system message into the prompt if it's missing, overwrites the default system message that is set using a chat template like the one above.

[BramVanroy]

This PR should alleviate the issue. #123 By disabling auto_insert_empty_system_msg, the system message should not be auto inserted. So the argument would be something like --no_auto_insert_empty_system_msg.

[BramVanroy]

Since #123 has been merged, this can be closed I think @dctanner