feat(model): Combined dataset to HF pipeline

[boss-chanon]

Why this PR

merge dataset in hf dataset by can set weight of dataset

Changes

• hf trainer pipeline can merge dataset

Related Issues

Close #

Checklist

• [ ] PR should be in the Naming convention
• [ ] Assign yourself in to Assigneees
• [ ] Tag related issues
• [ ] Constants name should be ALL_CAPITAL, function name should be snake_case, and class name should be CamelCase
• [ ] complex function/algorithm should have Docstring
• [ ] 1 PR should not have more than 200 lines changes (Exception for test files). If more than that please open multiple PRs
• [ ] At least PR reviewer must come from the task's team (model, eval, data)

[new5558]

Can you test is this data loader actually can load two datasets with the specified weight

[codecov[bot]]

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (2f8f284) 94.15% compared to head (5a27d13) 19.39%. Report is 30 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #316 +/- ## =========================================== - Coverage 94.15% 19.39% -74.77% =========================================== Files 10 25 +15 Lines 291 1392 +1101 =========================================== - Hits 274 270 -4 - Misses 17 1122 +1105 ``` | [Flag](https://app.codecov.io/gh/OpenThaiGPT/openthaigpt-pretraining/pull/316/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenThaiGPT) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/OpenThaiGPT/openthaigpt-pretraining/pull/316/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenThaiGPT) | `19.39% <ø> (-74.77%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenThaiGPT#carryforward-flags-in-the-pull-request-comment) to find out more. [see 35 files with indirect coverage changes](https://app.codecov.io/gh/OpenThaiGPT/openthaigpt-pretraining/pull/316/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=OpenThaiGPT)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[boss-chanon]

Can you test is this data loader actually can load two datasets with the specified weight

after test this solution don't exact match by weight and some output data are duplicate.

[new5558]

Can you test is this data loader actually can load two datasets with the specified weight

after test this solution don't exact match by weight and some output data are duplicate.

Approve krub