search

facebookresearch / adaptive_teacher

This repo provides the source code for "Cross-Domain Adaptive Teacher for Object Detection".
Other
190 stars 37 forks source link

AttributeError: 'NoneType' object has no attribute 'keys' #30

Open zwang123 opened 2 years ago

zwang123 commented 2 years ago

I was trying to reproduce the training from PASCAL VOC (source) to Clipart1k (target) using

python train_net.py \
      --num-gpus 8 \
      --config configs/faster_rcnn_R101_cross_clipart.yaml\
      OUTPUT_DIR output/exp_clipart

However, I got the following error message:

Traceback (most recent call last):
  File ".../adaptive_teacher/train_net.py", line 73, in <module>
    launch(
  File ".../detectron2-0.3/detectron2/engine/launch.py", line 55, in launch
    mp.spawn(
  File ".../lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File ".../lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File ".../lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File ".../lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File ".../detectron2-0.3/detectron2/engine/launch.py", line 94, in _distributed_worker
    main_func(*args)
  File ".../adaptive_teacher/train_net.py", line 64, in main
    trainer.resume_or_load(resume=args.resume)
  File ".../adaptive_teacher/adapteacher/engine/trainer.py", line 337, in resume_or_load
    checkpoint = self.checkpointer.resume_or_load(
  File ".../lib/python3.10/site-packages/fvcore/common/checkpoint.py", line 229, in resume_or_load
    return self.load(path, checkpointables=[])
  File ".../lib/python3.10/site-packages/fvcore/common/checkpoint.py", line 156, in load
    incompatible = self._load_model(checkpoint)
  File ".../adaptive_teacher/adapteacher/checkpoint/detection_checkpoint.py", line 24, in _load_model
    incompatible = self._load_student_model(checkpoint)
  File ".../adaptive_teacher/adapteacher/checkpoint/detection_checkpoint.py", line 64, in _load_student_model
    self._convert_ndarray_to_tensor(checkpoint_state_dict)
  File ".../lib/python3.10/site-packages/fvcore/common/checkpoint.py", line 368, in _convert_ndarray_to_tensor
    for k in list(state_dict.keys()):
AttributeError: 'NoneType' object has no attribute 'keys'

I have pinpointed that the issue comes from detectron2.checkpoint.c2_model_loading.align_and_update_state_dicts removing all information in checkpoint["model"] because it is totally normal before entering this function but returns None after the function call in Line 17 of file .../adaptive_teacher/adapteacher/checkpoint/detection_checkpoint.py.

Could you please confirm why this function returns None? I really appreciate your help!!

As you might have noticed, I am using the following environment:

yujheli commented 2 years ago

I do not have this error using the current code on my end.

zwang123 commented 2 years ago

@yujheli I have tried a clean clone of the latest version (up to the point of posting the comment) of adaptive teacher and I am still getting the same issue. I strongly suspect if the weights detectron2://ImageNetPretrained/MSRA/R-101.pkl has been updated such that it is not compatible with the old version of Detectron2==0.3.

Could you please reinstall a clean version of Detectron2==0.3 (to prevent any modification of codes) as well as re-download the weights by deleting local cache of R-101.pkl (which, in my case, is located at $HOME/.torch/iopath_cache/detectron2/ImageNetPretrained/MSRA/R-101.pkl) to re-test the codes?

Doris1231 commented 2 years ago

I do not have this error using the current code on my end.

When I use the weight provided by your interal prod_weights "VOC2Clipart.pth" , the code works .But when I use the weights downlode from the detectron2://ImageNetPretrained/MSRA/R-101.pkl , it makes an error " AttributeError: 'NoneType' object has no attribute 'keys' ",

Why does the R-101.pkl not work？ @zwang123 @yujheli

ssunguotu commented 2 years ago

I do not have this error using the current code on my end.

When I use the weight provided by your interal prod_weights "VOC2Clipart.pth" , the code works .But when I use the weights downlode from the detectron2://ImageNetPretrained/MSRA/R-101.pkl , it makes an error " AttributeError: 'NoneType' object has no attribute 'keys' ",

Why does the R-101.pkl not work？ @zwang123 @yujheli

yeah, I have the same question.

ssunguotu commented 2 years ago

I found there has something wrong in adaptive_teacher/adapteacher/checkpoint/detection_checkpoint.py ，and I fixed it.

class DetectionTSCheckpointer(DetectionCheckpointer):
    def _load_model(self, checkpoint):
        if checkpoint.get("__author__", None) == "Caffe2":
            # pretrained model weight: only update student model
            if checkpoint.get("matching_heuristics", False):
                self._convert_ndarray_to_tensor(checkpoint["model"])
                # convert weights by name-matching heuristics
                model_state_dict = self.model.modelStudent.state_dict()
                align_and_update_state_dicts(
                    model_state_dict,
                    checkpoint["model"],
                    c2_conversion=checkpoint.get("__author__", None) == "Caffe2",
                )
                checkpoint["model"] = model_state_dict

            # for non-caffe2 models, use standard ways to load it
            incompatible = self._load_student_model(checkpoint)

            model_buffers = dict(self.model.modelStudent.named_buffers(recurse=False))
            for k in ["pixel_mean", "pixel_std"]:
                # Ignore missing key message about pixel_mean/std.
                # Though they may be missing in old checkpoints, they will be correctly
                # initialized from config anyway.
                if k in model_buffers:
                    try:
                        incompatible.missing_keys.remove(k)
                    except ValueError:
                        pass
            return incompatible

        else:  # whole model
            if checkpoint.get("matching_heuristics", False):
                self._convert_ndarray_to_tensor(checkpoint["model"])
                # convert weights by name-matching heuristics
                model_state_dict = self.model.state_dict()
                align_and_update_state_dicts(
                    model_state_dict,
                    checkpoint["model"],
                    c2_conversion=checkpoint.get("__author__", None) == "Caffe2",
                )
                checkpoint["model"] = model_state_dict
            # for non-caffe2 models, use standard ways to load it
            incompatible = super()._load_model(checkpoint)

            model_buffers = dict(self.model.named_buffers(recurse=False))
            for k in ["pixel_mean", "pixel_std"]:
                # Ignore missing key message about pixel_mean/std.
                # Though they may be missing in old checkpoints, they will be correctly
                # initialized from config anyway.
                if k in model_buffers:
                    try:
                        incompatible.missing_keys.remove(k)
                    except ValueError:
                        pass
            return incompatible
fatherplus commented 1 year ago

I do not have this error using the current code on my end.

When I use the weight provided by your interal prod_weights "VOC2Clipart.pth" , the code works .But when I use the weights downlode from the detectron2://ImageNetPretrained/MSRA/R-101.pkl , it makes an error " AttributeError: 'NoneType' object has no attribute 'keys' ", Why does the R-101.pkl not work？ @zwang123 @yujheli

yeah, I have the same question.

Thank you!It truely work!