ludwig-ai / ludwig

Low-code framework for building custom LLMs, neural networks, and other AI models
http://ludwig.ai
Apache License 2.0
11.21k stars 1.19k forks source link

[Maintenance] Bump minimum tokenizers to 0.15 #3856

Closed arnavgarg1 closed 11 months ago

arnavgarg1 commented 11 months ago

Required for transformers 4.36

The conflict is caused by:
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    ludwig 0.9.dev0 depends on tokenizers>=0.13.3
    ludwig[test] 0.9.dev0 depends on tokenizers>=0.13.3
    transformers 4.36.2 depends on tokenizers<0.19 and >=0.14
    ludwig 0.9.dev0 depends on tokenizers>=0.13.3
    ludwig[test] 0.9.dev0 depends on tokenizers>=0.13.3
    transformers 4.36.1 depends on tokenizers<0.19 and >=0.14
    ludwig 0.9.dev0 depends on tokenizers>=0.13.3
    ludwig[test] 0.9.dev0 depends on tokenizers>=0.13.3
    transformers 4.36.0 depends on tokenizers<0.19 and >=0.14

Tokenizers 0.14 is linked to huggingface hub < 0.17, but transformers 4.36 requires huggingface hub >= 0.19

github-actions[bot] commented 11 months ago

Unit Test Results

  6 files  ±0    6 suites  ±0   13m 54s :stopwatch: +2s 12 tests ±0    9 :heavy_check_mark: ±0    3 :zzz: ±0  0 :x: ±0  60 runs  ±0  42 :heavy_check_mark: ±0  18 :zzz: ±0  0 :x: ±0 

Results for commit 85a80f34. ± Comparison against base commit d45566b0.