Circular imports when installing custom model

[Timber-Ye]

Hi! Thanks for this excellent work! I'm trying to register a custom method in Nerfstudio according to the nerfstudio-method-template repository, but I got the following error when running ns-train or ns-install-cli:

>>> ns-train --help
Traceback (most recent call last):
  File "/home/xxx/miniconda3/envs/nerfstudio/bin/ns-train", line 5, in <module>
    from nerfstudio.scripts.train import entrypoint
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 62, in <module>
    from nerfstudio.configs.method_configs import AnnotatedBaseConfigUnion
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/method_configs.py", line 29, in <module>
    from nerfstudio.configs.external_methods import get_external_methods
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/external_methods.py", line 24, in <module>
    from nerfstudio.engine.trainer import TrainerConfig
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 30, in <module>
    from nerfstudio.configs.experiment_config import ExperimentConfig
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/experiment_config.py", line 35, in <module>
    from nerfstudio.pipelines.base_pipeline import VanillaPipelineConfig
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 43, in <module>
    from nerfstudio.data.datamanagers.base_datamanager import (
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py", line 53, in <module>
    from nerfstudio.configs.dataparser_configs import AnnotatedDataParserUnion
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/dataparser_configs.py", line 57, in <module>
    external_dataparsers, _ = discover_dataparsers()
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/plugins/registry_dataparser.py", line 59, in discover_dataparsers
    spec = discovered_entry_points[name].load()
  File "/home/xxx//miniconda3/envs/nerfstudio/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 208, in load
    module = import_module(match.group('module'))
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/Projects/nerf-augmented-scr/neurep/neurep_config.py", line 4, in <module>
    from neurep.neurep_datamanager import NeuRepDataManagerConfig
  File "/home/Projects/nerf-augmented-scr/neurep/neurep_datamanager.py", line 17, in <module>
    from nerfstudio.data.datamanagers.base_datamanager import (
ImportError: cannot import name 'VanillaDataManagerConfig' from partially initialized module 'nerfstudio.data.datamanagers.base_datamanager' (most likely due to a circular import) (/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py)

[maturk]

Hi @Timber-Ye , can you confirm if you are able to install and train on the nerfstudio-method-template model?

To do this, got to a suitable folder and run the following commands:

conda activate nerfstudio
git clone https://github.com/nerfstudio-project/nerfstudio-method-template.git
cd nerfstudio-method-template/
pip install -e .
ns-install-cli

Then when you run ns-train --help you should see a new method called method-template in the results. Let me know if this works as intended. If it does not work, or you run to some errors, you could consider pulling the latest code from the main nerfstudio and pip installing that again.

PS. Since you already have tried registering a new method, but you are running into errors, please uninstall your custom method first to ensure that it is not messing up with the ns-install-cli commands. To do this run pip uninstall your-custom-method or go to the pyproject.toml file and comment out the line directly under [project.entry-points.'nerfstudio.method_configs'] and run pip install -e . to make sure your method is not trying to register with nerfstudio.

[Timber-Ye]

Hi @maturk , thanks for replying! After removing my-custom-method, everything goes right when I try to install the nerfstudio-method-template model, and I can see the method-template method after running ns-train --help.

Is it possible that I get something wrong when implementing the custom model?

~Meanwhile, I run into the same issue when I try to import any kind of data manager in the interactive console even with my-custom-method uninstalled, like~

Python 3.8.18 (default, Sep 11 2023, 13:40:15) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nerfstudio.data.datamanagers.base_datamanager import (
...    VanillaDataManager,
...    VanillaDataManagerConfig,
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py", line 53, in <module>
    from nerfstudio.configs.dataparser_configs import AnnotatedDataParserUnion
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/dataparser_configs.py", line 57, in <module>
    external_dataparsers, _ = discover_dataparsers()
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/plugins/registry_dataparser.py", line 59, in discover_dataparsers
    spec = discovered_entry_points[name].load()
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 208, in load
    module = import_module(match.group('module'))
  File "/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/xxx/Projects/nerf-augmented-scr/neurep/neurep_config.py", line 4, in <module>
    from neurep.neurep_datamanager import NeuRepDataManagerConfig
  File "/home/xxx/Projects/nerf-augmented-scr/neurep/neurep_datamanager.py", line 17, in <module>
    from nerfstudio.data.datamanagers.base_datamanager import (
ImportError: cannot import name 'VanillaDataManagerConfig' from partially initialized module 'nerfstudio.data.datamanagers.base_datamanager' (most likely due to a circular import) (/home/xxx/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/data/datamanagers/base_datamanager.py)

~It seems unreasonable. I find two of my custom files: neurep_config.py and neurep_datamanager.py still get involved, but I do uninstall my model by running pip uninstall my-custom-method.~ Sorry, it goes back to normal by deleting the egg-info folder.

[maturk]

@Timber-Ye Yea even after pip uninstalling you might have some files left in the cache or anaconda site-packages. Make sure you completely yeet your method from your computer. You are most likely doing something wrong in the super().__init__() part of your datamanager or other custom method files. Carefully check for any differences compared to the template repo. Let me know how it goes.

[Timber-Ye]

Hi @maturk, I just find that the circular imports issue is probably caused by the custom dataparser declaration in the same config file, like

[project]
name = "nerfstudio-method-template"
description = "Template repository for creating and registering new methods in nerfstudio."
version = "0.1.0"

dependencies = ["nerfstudio >= 0.3.0"] 

[tool.setuptools.packages.find]
include = ["method_template*"]

[project.entry-points.'nerfstudio.method_configs']
method-template = 'method_template.template_config:method_template'

[project.entry-points.'nerfstudio.dataparser_configs']
template-dataparser = 'method_template.template_config:template-dataparser'

The problem is solved after moving the dataparser declaration to a new config file.

Thanks a lot!

[shengjie-lin]

I am also running into this exact problem. Would you @Timber-Ye please elaborate on how to "move the dataparser declaration to a new config file"? Is it essentially creating 2 projects for this purpose? Or how can one have 2 pyproject.toml files in the same project? Thanks for your help!

[Timber-Ye]

Hi @shengjie-lin. Sorry for the vagueness. There is still only one pyproject.toml file. What I did was to separate the MethodSpecification and DataParserSpecification into two files.

In detail, I create an additional config file to hold the DataParserSpecification, like:

"""my_method/data/dataparser_config"""

from nerfstudio.plugins.registry_dataparser import DataParserSpecification
from my_method.data.dataparser import MyMethodDataParserConfig

my_method_dataparser = DataParserSpecification(config=MyMethodDataParserConfig())

The pyproject.toml file would be like:

[project]
name = "my_method"
version = "0.1"

dependencies = [
    "nerfstudio" # you may want to consider pinning the version, ie "nerfstudio==0.1.19"
]

[tool.setuptools.packages.find]
include = ["my_method*"]

[project.entry-points.'nerfstudio.method_configs']
my-method = 'my_method.my_config:MyMethod'

[project.entry-points.'nerfstudio.dataparser_configs']
my-method-data = 'my_method.data.dataparser_config:my_method_dataparser'

Hope it works for you!

[SiruiXie]

Hi @Timber-Ye , Thanks for your sharing about this issue. I followed this page and tried to write another config file to specify my dataparser like :

# my_method/dataparser_config.py
from nerfstudio.plugins.registry_dataparser import DataParserSpecification
from zipnerf_ns.kitti360_dataparser import Kitti360DataParserConfig

kitti360DataparserSpec = DataParserSpecification(config=Kitti360DataParserConfig())

But I still got error like this even when running ns-train --help after installing my kitti360DataParser with zipnerf:

ImportError: cannot import name 'VanillaDataManager' from partially initialized module 'nerfstudio.data.datamanagers.base_datamanager' (most likely due to a circular import) (.../envs/jax_dit/zipnerf-xiesr/lib/python3.9/site-packages/nerfstudio/data/datamanagers/base_datamanager.py)

My pyproject.toml looks like this:

[project]
name = "zipnerf"
description = "Zipnerf plugin for creating and registering zipnerf in nerfstudio."
version = "0.1.0"

dependencies = [
    "nerfstudio >= 0.3.0",
    "numpy",
    "torch",
    "absl-py",
    "accelerate",
    "gin-config",
    "imageio",
    "imageio[ffmpeg]",
    "matplotlib",
    "mediapy",
    "opencv-contrib-python",
    "opencv-python",
    "Pillow",
    "trimesh",
    "pymeshlab",
    "xatlas",
    "plyfile",
    "rawpy",
    "ninja",
    "scipy",
    "scikit-image",
    "scikit-learn",
    "tensorboard",
    "tensorboardX",
    "tqdm",
    "gin-config"
]

[tool.setuptools]
include-package-data = true

[tool.setuptools.packages.find]
include = ["zipnerf_ns*","internal*","gridencoder*","extensions*","configs*"]

[tool.setuptools.package-data]
configs = ["*.gin"]

# register the entry point of your new method here:
[project.entry-points.'nerfstudio.method_configs']
zipnerf_ns = 'zipnerf_ns.zipnerf_config:zipnerf_method'

# register kitti360 dataparser here
[project.entry-points.'nerfstudio.dataparser_configs']
zipnerfKitti360-data = 'zipnerf_ns.dataparser_config:zipnerfKitti360Spec'

Have you got any insight on why this import error should happen?

[Kevin-chen-chen]

# my_method/dataparser_config.py
from nerfstudio.plugins.registry_dataparser import DataParserSpecification
from zipnerf_ns.kitti360_dataparser import Kitti360DataParserConfig

kitti360DataparserSpec = DataParserSpecification(config=Kitti360DataParserConfig())

Hey, I have same situation. I solved by this method :

# my_method/dataparser_config.py
from nerfstudio.plugins.registry_dataparser import DataParserSpecification
from zipnerf_ns.kitti360_dataparser import Kitti360DataParserConfig

kitti360DataparserSpec = DataParserSpecification(config=Kitti360DataParserConfig())

Create an additional file with this code, and on your pyproject.toml to call this new file.

try it.