SlongLiu
commented
1 year ago Do you evaluate the COCO pre-trained models on your custom dataset? You need to train a new model for your dataset.
Open alrightkami opened 1 year ago
SlongLiu
commented
1 year ago Do you evaluate the COCO pre-trained models on your custom dataset? You need to train a new model for your dataset.
alrightkami
commented
1 year ago @SlongLiu I'm not evaluating, I'm training your pre-trained weights on a custom class with train_net.py
alrightkami
commented
1 year ago @SlongLiu any updates? were you able to reproduce the issue?
SlongLiu
commented
1 year ago As I do not know the custom dataset you used. I cannot reproduce the error.
However, the error is raised by cocoapi. There are some similar problems in other projects, which may be helpful. https://github.com/facebookresearch/detectron2/issues/1570 https://github.com/cocodataset/cocoapi/issues/497
One possible reason: https://github.com/facebookresearch/detectron2/issues/1570#issuecomment-641002199
If your data loading function get_kaist_dicts does not return the same data each time (seems to be the case in your code), the predicted image ids will not match the original image ids.
alrightkami
commented
1 year ago @SlongLiu it does not matter what custom dataset you use as far as it stays custom and not COCO in order to reproduce the error.
Also, I do not understand why are you hinting at somebody else's code, I do not have a function get_kaist_dicts
SlongLiu
commented
1 year ago We did not meet the problem in our other custom datasets.
You seem to have other costume codes, such as the coco_detr_graffiti.py file, which blocks us from reproducing it.
I do not think the problem is caused by our project, as our custom dataset design inherits from detectron2. Hence, I suggest you refer to other similar issues in the project, which may be helpful.
alrightkami
commented
1 year ago @SlongLiu the custom code in coco_detr_graffiti.py is not altering anything, the issue is reproducible with coco_detr.py as well. I just tried it and I'm getting the exact same error
SlongLiu
commented
1 year ago Thanks for the question, I will check it out soon.
Could you provide more details about the commands you used? As well as other modified codes if there are.
alrightkami
commented
1 year ago @SlongLiu sure, the command I used is
!cd /home/jovyan/data/kamila/detrex && python tools/train_net_satellit_graffiti.py --config-file projects/dino/configs/dino_swin_large_384_4scale_12ep_graffiti_satellit.py
The config is the same as above, except for the dataloader:
dataloader = get_config("common/data/coco_detr").dataloader
However, as I mentioned before, both custom and standard dataloader lead to the same issue.
And the train_net_satellit_graffiti.py is a copy of train_net.py with a single change: I'm registering my datasets within detectron2 in the main method:
def register_data()
"""
Registers all of the graffiti data in Detectron's DatasetCatalog.
"""
dirs = ['train', 'val', 'test']
dataset_size = []
for i in range(len(dirs)):
folder = '/home/jovyan/data/kamila/data/{}/'.format(dirs[i])
annotations = folder + 'annotations/instances_default.json'
images = folder + 'images/'
dataset_name = 'graffiti_{}'.format(dirs[i])
register_coco_instances(
dataset_name,
{},
annotations,
images)
dataset_size.append(len(glob.glob(images + '*.jpg')))
print('[{}/{}] Following dataset was sucesfullly registred: {} \npath: {} \ntotal_images: {}'.format(i, len(dirs), dataset_name, folder, sum(dataset_size)))
def main(args):
register_data()
cfg = LazyConfig.load(args.config_file)
cfg = LazyConfig.apply_overrides(cfg, args.opts)
default_setup(cfg, args)
...Thanks for the details.
I tested the model with the modification you provided. I find you still infer models on your custom dataset in your latest experiments. Although changing the line dataloader = get_config("common/data/coco_detr.py").dataloader, lines 57-59 in the config file specify the used dataset to your custom dataset.
The lines 57-59:
dataloader.train.dataset.names = 'graffiti_train'
dataloader.test.dataset.names = 'graffiti_test'
dataloader.evaluator.dataset_name = 'graffiti_val'When I direct ran the shell you provided, it raised an error:
Traceback (most recent call last):
File "tools/train_net_satellit_graffiti.py", line 262, in <module>
args=(args,),
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/engine/launch.py", line 82, in launch
main_func(*args)
File "tools/train_net_satellit_graffiti.py", line 249, in main
print(do_test(cfg, model))
File "tools/train_net_satellit_graffiti.py", line 136, in do_test
model, instantiate(cfg.dataloader.test), instantiate(cfg.dataloader.evaluator)
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 67, in instantiate
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 67, in <dictcomp>
cfg = {k: instantiate(v) for k, v in cfg.items()}
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/config/instantiate.py", line 83, in instantiate
return cls(**cfg)
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/build.py", line 241, in get_detection_dataset_dicts
dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/build.py", line 241, in <listcomp>
dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/catalog.py", line 58, in get
return f()
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/datasets/coco.py", line 500, in <lambda>
DatasetCatalog.register(name, lambda: load_coco_json(json_file, image_root, name))
File "/home/liushilong/code/maskdino_raw/detectron2/detectron2/data/datasets/coco.py", line 69, in load_coco_json
coco_api = COCO(json_file)
File "/home/liushilong/anaconda3/envs/q2x/lib/python3.7/site-packages/pycocotools/coco.py", line 84, in __init__
with open(annotation_file, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/data/kamila/data/test/annotations/instances_default.json'It means the code still runs on your custom dataset. (The dataset registration process ran successfully.)
When annotating the lines 57-59, the inference runs successfully on COCO without any error.
I suggest stepping into the error lines, i.e.,
File "/opt/conda/lib/python3.8/site-packages/pycocotools/coco.py", line 327, in loadRes
assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \to compare the difference between the ground truth ids and model-predicted ids.
alrightkami
commented
1 year ago @SlongLiu It is pretty obvious my code crushes if you run it, because you don't have the data I have. This is what error FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/data/kamila/data/test/annotations/instances_default.json' means, it is because this file is located on my computer and not yours.
Let me now repeat myself.
What I want to do: I want to use your pre-trained weights to fine-tune them on my custom data with one class 'graffiti'. I do not want to evaluate a trained model, and I do not want to run inference on COCO. I want to see the evaluation of the model during the training every n iterations as in train.eval_period in config file i.e. while the model is training I want to see every n iterations how/if is it improving.
What I did: As stated in the tutorial, I added init_checkpoint of the pre-trained model, custom data and n_classes = 1 to the config file and I run train_net.py.
What happened: Apparently, instead of evaluating the performance of the model on my custom class I'm currently training the model on, it is being evaluated on COCO classes, which I do not need because when I'm fine-tuning on my custom class and custom data, I need to see performance of my custom class.
Can you tell me now what am I doing wrong? Is it not the way to achieve what I want?
@rentainhe it would be really nice if you could help out here because I opened the issue a month ago and there's still no improvement whatsoever.
SlongLiu
commented
1 year ago @alrightkami
Hello, I want to explain what my experiments show:
@SlongLiu the custom code in coco_detr_graffiti.py is not altering anything, the issue is reproducible with
coco_detr.pyas well. I just tried it and I'm getting the exact same error
You said the model was crushed on COCO as well in your experiments. The reason is that you still inferred the model on your custom dataset. The model can work well on COCO.
Hence, the reason for your error is your custom dataset.
alrightkami
commented
1 year ago @SlongLiu Of course, the model can work well on COCO, but this is not what I need. While training on a custom class 'graffiti', I want the model's performance on this custom class 'graffiti' and not on people, cars, or birds. So here I am, asking what am I doing wrong and how can I achieve that. But all you tell me is that your model works fine on COCO.
alrightkami
commented
1 year ago are there any updates? @rentainhe
When fine-tuning with following config:
I'm getting an exception: