Closed AndyWanna closed 8 months ago
Did you pull from the upstream?
I did, and it solved that issue. I am now getting the following issue:
Traceback (most recent call last):
File "/content/mase/machop/./ch", line 6, in <module>
ChopCLI().run()
File "/content/mase/machop/chop/cli.py", line 249, in run
self._run_search()
File "/content/mase/machop/chop/cli.py", line 373, in _run_search
search(**search_params)
File "/content/mase/machop/chop/actions/search/search.py", line 103, in search
strategy.search(search_space)
File "/content/mase/machop/chop/actions/search/strategies/optuna.py", line 153, in search
self._save_best(study, self.save_dir / "best.json")
File "/content/mase/machop/chop/actions/search/strategies/optuna.py", line 234, in _save_best
].map(
File "/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py", line 5902, in __getattr__
return object.__getattribute__(self, name)
AttributeError: 'DataFrame' object has no attribute 'map'. Did you mean: 'mad'?
I am also running into an issue when trying to run the search function using a GPU. I have set the accelerator to automatic instead of CPU in the .toml file. This is the error:
Traceback (most recent call last):
File "/content/mase/machop/./ch", line 6, in <module>
ChopCLI().run()
File "/content/mase/machop/chop/cli.py", line 249, in run
self._run_search()
File "/content/mase/machop/chop/cli.py", line 373, in _run_search
search(**search_params)
File "/content/mase/machop/chop/actions/search/search.py", line 86, in search
search_space.build_search_space()
File "/content/mase/machop/chop/actions/search/search_space/quantization/graph.py", line 77, in build_search_space
mase_graph = self.rebuild_model(sampled_config=None, is_eval_mode=True)
File "/content/mase/machop/chop/actions/search/search_space/quantization/graph.py", line 59, in rebuild_model
mg, _ = add_common_metadata_analysis_pass(
File "/content/mase/machop/chop/passes/graph/analysis/add_metadata/add_common_metadata.py", line 382, in add_common_metadata_analysis_pass
graph = graph_iterator_for_metadata(graph, **pass_args)
File "/content/mase/machop/chop/passes/graph/analysis/add_metadata/add_common_metadata.py", line 203, in graph_iterator_for_metadata
result = modules[node.target](*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/batchnorm.py", line 171, in forward
return F.batch_norm(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 2478, in batch_norm
return torch.batch_norm(
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper_CUDA__native_batch_norm)
If I am forced to use a CPU runtime will it not be too slow to train many different models?
Can you check PR #38 where I just enabled you to pick different --accelerator for that. I can't really reproduce the error you got with pandas though, what command did you use for that?
I am also receiving an error when trying to run the search command;
./ch search --config configs/examples/jsc_toy_by_type.toml --load ../mase_output/jsc-tiny_classification_jsc_2024-01-23/software/training_ckpts/best.ckpt
The error can be seen below;
Traceback (most recent call last):
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/./ch", line 2, in <module>
from chop.cli import ChopCLI
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/__init__.py", line 1, in <module>
from .tools.logger import root_logger
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/__init__.py", line 11, in <module>
from .get_input import get_cf_args, get_dummy_input
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/get_input.py", line 3, in <module>
from ..models.utils import ModelSource
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/__init__.py", line 6, in <module>
from .patched import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/__init__.py", line 2, in <module>
from .opt_patched import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/__init__.py", line 3, in <module>
from .modeling_opt_patched import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/modeling_opt_patched.py", line 53, in <module>
from ....passes.graph.patching.mase_op_wrapper import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/__init__.py", line 1, in <module>
from . import graph
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/__init__.py", line 1, in <module>
from .analysis import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/__init__.py", line 5, in <module>
from .add_metadata import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/__init__.py", line 1, in <module>
from .add_common_metadata import add_common_metadata_analysis_pass
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/add_common_metadata.py", line 20, in <module>
from chop.ir.graph.mase_metadata import MaseMetadata
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/__init__.py", line 1, in <module>
from .graph.mase_graph import MaseGraph, MaseTracer
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/__init__.py", line 1, in <module>
from .mase_graph import MaseGraph, MaseTracer
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/mase_graph.py", line 15, in <module>
from chop.passes.graph.transforms import utils as utils_passes
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/__init__.py", line 3, in <module>
from .verilog import (
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/__init__.py", line 2, in <module>
from .emit_tb import emit_cocotb_transform_pass
File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/emit_tb.py", line 14, in <module>
from mase_cocotb.testbench import Testbench
ModuleNotFoundError: No module named 'mase_cocotb'
I have synced changes and pulled from the upstream.
I have pulled the latest changes and am getting a different error in Lab3.
./ch search --config configs/examples/jsc_toy_by_type.toml --load ../mase_output/batch_256/jsc-tiny_classification_jsc_2024-01-25/software/training_ckpts/best.ckpt
(mase) laurie2905@LAPTOP-LQSPNHSL:~/mase/machop$ ./ch search --config configs/examples/jsc_toy_by_type.toml --load ../mase_output/batch_256/jsc-tiny_classification_jsc_2024-01-25/software/training_ckpts/best.ckpt
/home/laurie2905/anaconda3/envs/mase/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/home/laurie2905/anaconda3/envs/mase/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/home/laurie2905/anaconda3/envs/mase/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
2024-01-31 13:02:39.182328: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0
.
2024-01-31 13:02:39.221196: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-31 13:02:39.221278: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-31 13:02:39.222569: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-31 13:02:39.229700: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-01-31 13:02:39.903665: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/home/laurie2905/anaconda3/envs/mase/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
/home/laurie2905/mase/machop/chop/actions/simulate.py:3: UserWarning: Python runners and associated APIs are an experimental feature and subject to change.
from cocotb.runner import get_runner, get_results
Seed set to 0
+-------------------------+--------------------------+--------------+--------------------------+--------------------------+
| Name | Default | Config. File | Manual Override | Effective |
+-------------------------+--------------------------+--------------+--------------------------+--------------------------+
| task | classification | cls | | cls |
| load_name | None | | /home/laurie2905/mase/ma | /home/laurie2905/mase/ma |
| | | | se_output/batch_256/jsc- | se_output/batch_256/jsc- |
| | | | tiny_classificationjsc | tiny_classificationjsc |
| | | | 2024-01-25/software/trai | 2024-01-25/software/trai |
| | | | ning_ckpts/best.ckpt | ning_ckpts/best.ckpt |
| load_type | mz | | | mz |
| batch_size | 128 | 512 | | 512 |
| to_debug | False | | | False |
| log_level | info | | | info |
| report_to | tensorboard | | | tensorboard |
| seed | 0 | 42 | | 42 |
| quant_config | None | | | None |
| training_optimizer | adam | | | adam |
| trainer_precision | 16-mixed | | | 16-mixed |
| learning_rate | 1e-05 | 0.01 | | 0.01 |
| weight_decay | 0 | | | 0 |
| max_epochs | 20 | 5 | | 5 |
| max_steps | -1 | | | -1 |
| accumulate_grad_batches | 1 | | | 1 |
| log_every_n_steps | 50 | 5 | | 5 |
| num_workers | 8 | | | 8 |
| num_devices | 1 | | | 1 |
| num_nodes | 1 | | | 1 |
| accelerator | auto | gpu | | gpu |
| strategy | auto | | | auto |
| is_to_auto_requeue | False | | | False |
| github_ci | False | | | False |
| disable_dataset_cache | False | | | False |
| target | xcu250-figd2104-2L-e | | | xcu250-figd2104-2L-e |
| num_targets | 100 | | | 100 |
| is_pretrained | False | | | False |
| max_token_len | 512 | | | 512 |
| project_dir | /home/laurie2905/mase/ma | | | /home/laurie2905/mase/ma |
| | se_output | | | se_output |
| project | None | jsc-tiny | | jsc-tiny |
| model | None | jsc-tiny | | jsc-tiny |
| dataset | None | jsc | | jsc |
+-------------------------+--------------------------+--------------+--------------------------+--------------------------+
INFO Initialising model 'jsc-tiny'...
INFO Initialising dataset 'jsc'...
INFO Project will be created at /home/laurie2905/mase/mase_output/jsc-tiny
Traceback (most recent call last):
File "/home/laurie2905/mase/machop/./ch", line 6, in
@Aaron-Zhao123 The command I used is the following:
!./ch search --config configs/examples/jsc_toy_by_type.toml --load '/model/path' --load-type pl
With the accelerator set to CPU, a CPU runtime and the JSC-Tiny model trained on a CPU.
I have now pulled PR #38 and will try the GPU runtime and see if I no longer run into the error
@Aaron-Zhao123 The command I used is the following:
!./ch search --config configs/examples/jsc_toy_by_type.toml --load '/model/path' --load-type pl
With the accelerator set to CPU, a CPU runtime and the JSC-Tiny model trained on a CPU.
I have now pulled PR #38 and will try the GPU runtime and see if I no longer run into the error
Using this command I no longer get the error
I am also receiving an error when trying to run the search command;
./ch search --config configs/examples/jsc_toy_by_type.toml --load ../mase_output/jsc-tiny_classification_jsc_2024-01-23/software/training_ckpts/best.ckpt
The error can be seen below;
Traceback (most recent call last): File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/./ch", line 2, in <module> from chop.cli import ChopCLI File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/__init__.py", line 1, in <module> from .tools.logger import root_logger File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/__init__.py", line 11, in <module> from .get_input import get_cf_args, get_dummy_input File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/get_input.py", line 3, in <module> from ..models.utils import ModelSource File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/__init__.py", line 6, in <module> from .patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/__init__.py", line 2, in <module> from .opt_patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/__init__.py", line 3, in <module> from .modeling_opt_patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/modeling_opt_patched.py", line 53, in <module> from ....passes.graph.patching.mase_op_wrapper import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/__init__.py", line 1, in <module> from . import graph File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/__init__.py", line 1, in <module> from .analysis import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/__init__.py", line 5, in <module> from .add_metadata import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/__init__.py", line 1, in <module> from .add_common_metadata import add_common_metadata_analysis_pass File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/add_common_metadata.py", line 20, in <module> from chop.ir.graph.mase_metadata import MaseMetadata File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/__init__.py", line 1, in <module> from .graph.mase_graph import MaseGraph, MaseTracer File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/__init__.py", line 1, in <module> from .mase_graph import MaseGraph, MaseTracer File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/mase_graph.py", line 15, in <module> from chop.passes.graph.transforms import utils as utils_passes File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/__init__.py", line 3, in <module> from .verilog import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/__init__.py", line 2, in <module> from .emit_tb import emit_cocotb_transform_pass File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/emit_tb.py", line 14, in <module> from mase_cocotb.testbench import Testbench ModuleNotFoundError: No module named 'mase_cocotb'
I have synced changes and pulled from the upstream.
I solved this issue by moving the mase_cocotb\
and mase_components\
directories into the machop\
directory to be viewed on the path
I am also receiving an error when trying to run the search command;
./ch search --config configs/examples/jsc_toy_by_type.toml --load ../mase_output/jsc-tiny_classification_jsc_2024-01-23/software/training_ckpts/best.ckpt
The error can be seen below;
Traceback (most recent call last): File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/./ch", line 2, in <module> from chop.cli import ChopCLI File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/__init__.py", line 1, in <module> from .tools.logger import root_logger File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/__init__.py", line 11, in <module> from .get_input import get_cf_args, get_dummy_input File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/tools/get_input.py", line 3, in <module> from ..models.utils import ModelSource File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/__init__.py", line 6, in <module> from .patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/__init__.py", line 2, in <module> from .opt_patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/__init__.py", line 3, in <module> from .modeling_opt_patched import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/models/patched/opt_patched/modeling_opt_patched.py", line 53, in <module> from ....passes.graph.patching.mase_op_wrapper import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/__init__.py", line 1, in <module> from . import graph File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/__init__.py", line 1, in <module> from .analysis import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/__init__.py", line 5, in <module> from .add_metadata import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/__init__.py", line 1, in <module> from .add_common_metadata import add_common_metadata_analysis_pass File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/analysis/add_metadata/add_common_metadata.py", line 20, in <module> from chop.ir.graph.mase_metadata import MaseMetadata File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/__init__.py", line 1, in <module> from .graph.mase_graph import MaseGraph, MaseTracer File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/__init__.py", line 1, in <module> from .mase_graph import MaseGraph, MaseTracer File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/ir/graph/mase_graph.py", line 15, in <module> from chop.passes.graph.transforms import utils as utils_passes File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/__init__.py", line 3, in <module> from .verilog import ( File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/__init__.py", line 2, in <module> from .emit_tb import emit_cocotb_transform_pass File "/Users/jared/Documents/Personal/ICL/courses/ADLS/mase/machop/chop/passes/graph/transforms/verilog/emit_tb.py", line 14, in <module> from mase_cocotb.testbench import Testbench ModuleNotFoundError: No module named 'mase_cocotb'
I have synced changes and pulled from the upstream.
This issue was solved by running pip install .
Can you check PR #38 where I just enabled you to pick different --accelerator for that. I can't really reproduce the error you got with pandas though, what command did you use for that?
I solved the error, it seems colab was installing pandas v1.5.3 instead of the most recent 2.2 which has the required .map()
method. It seems the google colab default is 1.5.3 as it gave me a warning saying it is incompatible with v2.2, but the command ran regardless.
I am trying to run the provided code to run the search command in mase for task 3:
I am getting the following error suggesting something is transforming the mase graph incorrectly, I think?
I am not sure to how to proceed - if I have made a mistake somewhere or should debug the codebase
Thank you