Import functions from the ilab repository

[aakankshaduggal]

Some functions that are being called in the generate_data.py file are in this file - https://github.com/instructlab/instructlab/blob/main/src/instructlab/utils.py

---

to-do list:

• [ ] DEFAULT_MULTIPROCESSING_START_METHOD
  • Used in lab.py but also this is just a variable
• [ ] DEFAULT_API_KEY
  • Used in many places but also this is just a variable
• [ ] DEFAULT_MODEL_OLD
  • Used in lab.py but also this is just a variable
• [ ] get_model_family()
  • Used by generate_data.py but also server.py
  • https://github.com/instructlab/sdg/pull/46

From utils, we import the following:

• [x] chunk_document()
  • Only used by generate_data.py

  • 35

• [x] max_seed_example_tokens()
  • Only used by generate_data.py

  • 35

• [x] num_chars_from_tokens()
  • Only used by generate_data.py

  • 35

• [ ] read_taxonomy()
  • Used by generate_data.py but also lab.py
  • https://github.com/instructlab/sdg/pull/46
• [ ] get_sysprompt()
  • Used by generate_data.py but also lab.py, chat.py and make_data.py

[russellb]

related to #6

[nathan-weinberg]

@aakankshaduggal @russellb is this something we want to try and preserve commits for? if not I'm happy to take this (and the other issue I guess, seems one PR can resolve both)

[russellb]

@aakankshaduggal @russellb is this something we want to try and preserve commits for? if not I'm happy to take this (and the other issue I guess, seems one PR can resolve both)

In general, yes, I would always prefer keeping history. I can also walk you through how I would do it.

I haven't looked at these specifics. If the code in question is only used here, then moving it here seems like an easy decision. If it's used in other places, too, we should discuss further.

[nathan-weinberg]

ack @russellb - fine either way - AFAIK the code in question is only used in the CLI currently

[russellb]

ack @russellb - fine either way - AFAIK the code in question is only used in the CLI currently

question is it called from only code other than what got moved here

[nathan-weinberg]

Okay looked into this a bit - we import from two locations within instructlab - config and utils

From config, we import the following:

• DEFAULT_MULTIPROCESSING_START_METHOD
  • Used in lab.py but also this is just a variable
• DEFAULT_API_KEY
  • Used in many places but also this is just a variable
• DEFAULT_MODEL_OLD
  • Used in lab.py but also this is just a variable
• get_model_family()
  • Used by generate_data.py but also server.py

From utils, we import the following:

• chunk_document()
  • Only used by generate_data.py
• max_seed_example_tokens()
  • Only used by generate_data.py
• num_chars_from_tokens()
  • Only used by generate_data.py
• read_taxonomy()
  • Used by generate_data.py but also lab.py
• get_sysprompt()
  • Used by generate_data.py but also lab.py, chat.py and make_data.py

[tiran]

You can write rules for Ruff or PyLint to detect these types of imports and raise an error. InstructLab uses Ruff for that:

[tool.ruff.lint.flake8-tidy-imports.banned-api]
"yamllint".msg = "yamllint is for CLI usage only."