search

microsoft / DeepSpeedExamples

Example models using DeepSpeed
Apache License 2.0
6.02k stars 1.02k forks source link

ModuleNotFoundError: No module named 'utils.data' #639

Open xtu-xiaoc opened 1 year ago

xtu-xiaoc commented 1 year ago

I know this is a path problem, and I have modified the code to know the parent directory, but I still can't find the package, step1_supervised_finetuning and utils are at the same level, how do you solve it?

question: from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data' [2023-07-14 05:54:32,744] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 88084 Traceback (most recent call last): File "/202221633050/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 27, in from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data' Traceback (most recent call last): File "/202221633050/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 27, in from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data' [2023-07-14 05:54:33,056] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 88085 Traceback (most recent call last): File "/202221633050/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 27, in from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data' Traceback (most recent call last): File "/202221633050/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 27, in from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data' Traceback (most recent call last): File "/202221633050/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/main.py", line 27, in from utils.data.data_utils import create_prompt_dataset ModuleNotFoundError: No module named 'utils.data'

yanshanjing commented 1 year ago

just put a empty file named init.py in utils floder

roy-mzh commented 1 year ago

I solved the problem by adding the following statements:

import sys sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), os.path.pardir))) sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '..', 'utils'))) sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '..', 'utils.model'))) sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '..', 'utils.data'))) sys.path.append(os.path.abspath(os.path.join(os.path.dirname(file), '..', 'utils.module')))