When a bootstrap build fails, it's possible that the traceback chain includes a bunch of exceptions related to all of the packages from the bottom of the stack of dependencies to the top. This can make it confusing to figure out which error is most relevant/important and where to start looking to fix the problem.
could not handle install dependency accelerate for [('container-dependencies/rocm-ubi9/requirements.txt', <Requirement('instructlab-eval>=0.0.4')>, <Version('0.0.9')>)]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 224, in handle_requirement
handle_requirement(ctx, dep, next_req_type, why)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 144, in handle_requirement
sdist_root_dir = sources.prepare_source(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 307, in prepare_source
source_root_dir = preparer(ctx, req, source_filename, version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/package_plugins/torch.py", line 102, in prepare_source
sources._patch_source(ctx, source_root_dir)
^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'fromager.sources' has no attribute '_patch_source'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 224, in handle_requirement
handle_requirement(ctx, dep, next_req_type, why)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 226, in handle_requirement
raise ValueError(
ValueError: could not handle install dependency torch>=1.10.0 for [('container-dependencies/rocm-ubi9/requirements.txt', <Requirement('instructlab-eval>=0.0.4')>, <Version('0.0.9')>), ('install', <Requirement('accelerate')>, <Version('0.32.1')>)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/fromager/__main__.py", line 156, in invoke_main
main(auto_envvar_prefix="FROMAGER")
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fromager/commands/bootstrap.py", line 63, in bootstrap
sdist.handle_requirement(wkctx, Requirement(toplevel), req_type=origin)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 226, in handle_requirement
raise ValueError(
ValueError: could not handle install dependency accelerate for [('container-dependencies/rocm-ubi9/requirements.txt', <Requirement('instructlab-eval>=0.0.4')>, <Version('0.0.9')>)]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 224, in handle_requirement
handle_requirement(ctx, dep, next_req_type, why)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 144, in handle_requirement
sdist_root_dir = sources.prepare_source(
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fromager/sources.py", line 307, in prepare_source
source_root_dir = preparer(ctx, req, source_filename, version)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/package_plugins/torch.py", line 102, in prepare_source
sources._patch_source(ctx, source_root_dir)
^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'fromager.sources' has no attribute '_patch_source'. Did you mean: 'patch_source'?
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 224, in handle_requirement
handle_requirement(ctx, dep, next_req_type, why)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 226, in handle_requirement
raise ValueError(
ValueError: could not handle install dependency torch>=1.10.0 for [('container-dependencies/rocm-ubi9/requirements.txt', <Requirement('instructlab-eval>=0.0.4')>, <Version('0.0.9')>), ('install', <Requirement('accelerate')>, <Version('0.32.1')>)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/fromager", line 8, in <module>
sys.exit(invoke_main())
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fromager/__main__.py", line 156, in invoke_main
main(auto_envvar_prefix="FROMAGER")
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 45, in new_func
return f(get_current_context().obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fromager/commands/bootstrap.py", line 63, in bootstrap
sdist.handle_requirement(wkctx, Requirement(toplevel), req_type=origin)
File "/usr/local/lib/python3.11/site-packages/fromager/sdist.py", line 226, in handle_requirement
raise ValueError(
ValueError: could not handle install dependency accelerate for [('container-dependencies/rocm-ubi9/requirements.txt', <Requirement('instructlab-eval>=0.0.4')>, <Version('0.0.9')>)]
We should think about better ways to report the errors.
For example, do we need the tracebacks on the console, or is having those in the log files sufficient?
Could we emit some details before, or after, the tracebacks to try to diagnose the problem? Showing the full chain of dependencies that is failing, the one that actually triggered the error, that error, etc.
When a bootstrap build fails, it's possible that the traceback chain includes a bunch of exceptions related to all of the packages from the bottom of the stack of dependencies to the top. This can make it confusing to figure out which error is most relevant/important and where to start looking to fix the problem.
We should think about better ways to report the errors.
For example, do we need the tracebacks on the console, or is having those in the log files sufficient?
Could we emit some details before, or after, the tracebacks to try to diagnose the problem? Showing the full chain of dependencies that is failing, the one that actually triggered the error, that error, etc.