search

shanice-l / gdrnpp_bop2022

PyTorch Implementation of GDRNPP, winner (most of the awards) of the BOP Challenge 2022 at ECCV'22
Apache License 2.0
246 stars 52 forks source link

[egl OffscreenContext:] Bindless Texture Not supported #15

Open FabianSchuetze opened 2 years ago

FabianSchuetze commented 2 years ago

Thanks for continuing the work on GDR. I enjoyed working with the initial version a year ago and look forward to trying the revised version.

When I start the training, I receive the following error message from egl_offscreen_context.py:227:

RuntimeError: Bindless Textures not supported

The entire log of the program is:

./root@n156egjsrm:/notebooks/gdrnpp_bop2022./core/gdrn_modeling/train_gdrn.sh /notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py 0
++ dirname ./core/gdrn_modeling/train_gdrn.sh
+ this_dir=./core/gdrn_modeling
+ CFG=/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py
+ CUDA_VISIBLE_DEVICES=0
+ IFS=,
+ read -ra GPUS
+ NGPU=1
+ echo 'use gpu ids: 0 num gpus: 1'
use gpu ids: 0 num gpus: 1
+ NCCL_DEBUG=INFO
+ OMP_NUM_THREADS=1
+ MKL_NUM_THREADS=1
+ PYTHONPATH=./core/gdrn_modeling/../..:
+ CUDA_VISIBLE_DEVICES=0
+ python ./core/gdrn_modeling/main_gdrn.py --config-file /notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py --num-gpus 1
/usr/local/lib/python3.9/dist-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
You requested to import horovod which is missing or not supported for your OS.
[1126_222408@main_gdrn:216] soft limit:  500000 hard limit:  1048576
[1126_222408@main_gdrn:227] Command Line Args: Namespace(config_file='/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py', resume=False, eval_only=False, launcher='none', local_rank=0, fp16_allreduce=False, use_adasum=False, num_gpus=1, num_machines=1, machine_rank=0, dist_url='tcp://127.0.0.1:49152', opts=None, strategy=None)
[1126_222408@main_gdrn:101] optimizer_cfg: {'type': 'Ranger', 'lr': 0.0008, 'weight_decay': 0.01}
[1126_222409@itodd_pbr:358] DBG register dataset: itodd_train_pbr
[1126_222409@itodd_bop_test:379] DBG register dataset: itodd_bop_test
20221126_142410|core.utils.default_args_setup@123: Rank of current process: 0. World size: 1
20221126_142410|core.utils.default_args_setup@124: Environment info:
----------------------  ----------------------------------------------------------------
sys.platform            linux
Python                  3.9.13 (main, May 23 2022, 22:01:06) [GCC 9.4.0]
numpy                   1.23.1
detectron2              0.6 @/usr/local/lib/python3.9/dist-packages/detectron2
Compiler                GCC 9.4
CUDA compiler           CUDA 11.2
detectron2 arch flags   6.1
DETECTRON2_ENV_MODULE   <not set>
PyTorch                 1.12.0+cu116 @/usr/local/lib/python3.9/dist-packages/torch
PyTorch debug build     False
GPU available           Yes
GPU 0                   Quadro P5000 (arch=6.1)
Driver version          510.73.05
CUDA_HOME               /usr/local/cuda
Pillow                  9.3.0
torchvision             0.13.0+cu116 @/usr/local/lib/python3.9/dist-packages/torchvision
torchvision arch flags  3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
fvcore                  0.1.5.post20221122
iopath                  0.1.9
cv2                     4.6.0
----------------------  ----------------------------------------------------------------
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.6
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.3.2  (built against CUDA 11.5)
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.12.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=OFF, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

20221126_142410|core.utils.default_args_setup@126: Command line arguments: Namespace(config_file='/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py', resume=False, eval_only=False, launcher='none', local_rank=0, fp16_allreduce=False, use_adasum=False, num_gpus=1, num_machines=1, machine_rank=0, dist_url='tcp://127.0.0.1:49152', opts=None, strategy=None)
20221126_142410|core.utils.default_args_setup@128: Contents of args.config_file=/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py:
_base_ = ["../../_base_/gdrn_base.py"]

OUTPUT_DIR = "output/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd"
INPUT = dict(
    DZI_PAD_SCALE=1.5,
    TRUNCATE_FG=False,
    CHANGE_BG_PROB=0.5,
    COLOR_AUG_PROB=0.8,
    MIN_SIZE_TRAIN=960,
    MAX_SIZE_TRAIN=1280,
    MIN_SIZE_TEST=960,
    MAX_SIZE_TEST=1280,
    COLOR_AUG_TYPE="code",
    COLOR_AUG_CODE=(
        "Sequential(["
        # Sometimes(0.5, PerspectiveTransform(0.05)),
        # Sometimes(0.5, CropAndPad(percent=(-0.05, 0.1))),
        # Sometimes(0.5, Affine(scale=(1.0, 1.2))),
        "Sometimes(0.5, CoarseDropout( p=0.2, size_percent=0.05) ),"
        "Sometimes(0.4, GaussianBlur((0., 3.))),"
        "Sometimes(0.3, pillike.EnhanceSharpness(factor=(0., 50.))),"
        "Sometimes(0.3, pillike.EnhanceContrast(factor=(0.2, 50.))),"
        "Sometimes(0.5, pillike.EnhanceBrightness(factor=(0.1, 6.))),"
        "Sometimes(0.3, pillike.EnhanceColor(factor=(0., 20.))),"
        "Sometimes(0.5, Add((-25, 25), per_channel=0.3)),"
        "Sometimes(0.3, Invert(0.2, per_channel=True)),"
        "Sometimes(0.5, Multiply((0.6, 1.4), per_channel=0.5)),"
        "Sometimes(0.5, Multiply((0.6, 1.4))),"
        "Sometimes(0.1, AdditiveGaussianNoise(scale=10, per_channel=True)),"
        "Sometimes(0.5, iaa.contrast.LinearContrast((0.5, 2.2), per_channel=0.3)),"
        "Sometimes(0.5, Grayscale(alpha=(0.0, 1.0))),"  # maybe remove for det
        "], random_order=True)"
        # cosy+aae
    ),
)

SOLVER = dict(
    IMS_PER_BATCH=48,
    TOTAL_EPOCHS=40,  # 25
    LR_SCHEDULER_NAME="flat_and_anneal",
    ANNEAL_METHOD="cosine",  # "cosine"
    ANNEAL_POINT=0.72,
    OPTIMIZER_CFG=dict(_delete_=True, type="Ranger", lr=8e-4, weight_decay=0.01),
    WEIGHT_DECAY=0.0,
    WARMUP_FACTOR=0.001,
    WARMUP_ITERS=1000,
)

DATASETS = dict(
    TRAIN=("itodd_train_pbr",),
    TEST=("itodd_bop_test",),
    DET_FILES_TEST=("datasets/BOP_DATASETS/itodd/test/test_bboxes/yolox_x_640_itodd_pbr_itodd_bop_test.json",),
    DET_TOPK_PER_OBJ=100,
    DET_THR=0.05,
)

DATALOADER = dict(
    # Number of data loading threads
    NUM_WORKERS=8,
    FILTER_VISIB_THR=0.3,
)

MODEL = dict(
    LOAD_DETS_TEST=True,
    PIXEL_MEAN=[0.0, 0.0, 0.0],
    PIXEL_STD=[255.0, 255.0, 255.0],
    BBOX_TYPE="AMODAL_CLIP",  # VISIB or AMODAL
    POSE_NET=dict(
        NAME="GDRN_double_mask",
        XYZ_ONLINE=True,
        NUM_CLASSES=28,
        BACKBONE=dict(
            FREEZE=False,
            PRETRAINED="timm",
            INIT_CFG=dict(
                type="timm/convnext_base",
                pretrained=True,
                in_chans=3,
                features_only=True,
                out_indices=(3,),
            ),
        ),
        ## geo head: Mask, XYZ, Region
        GEO_HEAD=dict(
            FREEZE=False,
            INIT_CFG=dict(
                type="TopDownDoubleMaskXyzRegionHead",
                in_dim=1024,  # this is num out channels of backbone conv feature
            ),
            NUM_REGIONS=64,
            XYZ_CLASS_AWARE=True,
            MASK_CLASS_AWARE=True,
            REGION_CLASS_AWARE=True,
        ),
        PNP_NET=dict(
            INIT_CFG=dict(norm="GN", act="gelu"),
            REGION_ATTENTION=True,
            WITH_2D_COORD=True,
            ROT_TYPE="allo_rot6d",
            TRANS_TYPE="centroid_z",
        ),
        LOSS_CFG=dict(
            # xyz loss ----------------------------
            XYZ_LOSS_TYPE="L1",  # L1 | CE_coor
            XYZ_LOSS_MASK_GT="visib",  # trunc | visib | obj
            XYZ_LW=1.0,
            # mask loss ---------------------------
            MASK_LOSS_TYPE="L1",  # L1 | BCE | CE
            MASK_LOSS_GT="trunc",  # trunc | visib | gt
            MASK_LW=1.0,
            # full mask loss ---------------------------
            FULL_MASK_LOSS_TYPE="L1",  # L1 | BCE | CE
            FULL_MASK_LW=1.0,
            # region loss -------------------------
            REGION_LOSS_TYPE="CE",  # CE
            REGION_LOSS_MASK_GT="visib",  # trunc | visib | obj
            REGION_LW=1.0,
            # pm loss --------------
            PM_LOSS_SYM=True,  # NOTE: sym loss
            PM_R_ONLY=True,  # only do R loss in PM
            PM_LW=1.0,
            # centroid loss -------
            CENTROID_LOSS_TYPE="L1",
            CENTROID_LW=1.0,
            # z loss -----------
            Z_LOSS_TYPE="L1",
            Z_LW=1.0,
        ),
    ),
)

VAL = dict(
    DATASET_NAME="itodd",
    SCRIPT_PATH="lib/pysixd/scripts/eval_pose_results_more.py",
    TARGETS_FILENAME="test_targets_bop19.json",
    ERROR_TYPES="mspd,mssd,vsd,ad,reS,teS",
    RENDERER_TYPE="cpp",  # cpp, python, egl
    SPLIT="test",
    SPLIT_TYPE="",
    N_TOP=-1,  # SISO: 1, VIVO: -1 (for LINEMOD, 1/-1 are the same)
    EVAL_CACHED=False,  # if the predicted poses have been saved
    SCORE_ONLY=False,  # if the errors have been calculated
    EVAL_PRINT_ONLY=False,  # if the scores/recalls have been saved
    EVAL_PRECISION=False,  # use precision or recall
    USE_BOP=True,  # whether to use bop toolkit
    SAVE_BOP_CSV_ONLY=True,
)

TEST = dict(EVAL_PERIOD=0, VIS=False, TEST_BBOX_TYPE="est")  # gt | est

20221126_142410|core.utils.default_args_setup@135: Running with full config:
Config (path: /notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py): {'OUTPUT_ROOT': 'output', 'OUTPUT_DIR': 'output/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd', 'EXP_NAME': '', 'DEBUG': False, 'SEED': -1, 'CUDNN_BENCHMARK': True, 'IM_BACKEND': 'cv2', 'VIS_PERIOD': 0, 'INPUT': {'FORMAT': 'BGR', 'MIN_SIZE_TRAIN': 960, 'MAX_SIZE_TRAIN': 1280, 'MIN_SIZE_TRAIN_SAMPLING': 'choice', 'MIN_SIZE_TEST': 960, 'MAX_SIZE_TEST': 1280, 'WITH_DEPTH': False, 'BP_DEPTH': False, 'AUG_DEPTH': False, 'NORM_DEPTH': False, 'DROP_DEPTH_RATIO': 0.2, 'DROP_DEPTH_PROB': 0.5, 'ADD_NOISE_DEPTH_LEVEL': 0.01, 'ADD_NOISE_DEPTH_PROB': 0.9, 'COLOR_AUG_PROB': 0.8, 'COLOR_AUG_TYPE': 'code', 'COLOR_AUG_CODE': 'Sequential([Sometimes(0.5, CoarseDropout( p=0.2, size_percent=0.05) ),Sometimes(0.4, GaussianBlur((0., 3.))),Sometimes(0.3, pillike.EnhanceSharpness(factor=(0., 50.))),Sometimes(0.3, pillike.EnhanceContrast(factor=(0.2, 50.))),Sometimes(0.5, pillike.EnhanceBrightness(factor=(0.1, 6.))),Sometimes(0.3, pillike.EnhanceColor(factor=(0., 20.))),Sometimes(0.5, Add((-25, 25), per_channel=0.3)),Sometimes(0.3, Invert(0.2, per_channel=True)),Sometimes(0.5, Multiply((0.6, 1.4), per_channel=0.5)),Sometimes(0.5, Multiply((0.6, 1.4))),Sometimes(0.1, AdditiveGaussianNoise(scale=10, per_channel=True)),Sometimes(0.5, iaa.contrast.LinearContrast((0.5, 2.2), per_channel=0.3)),Sometimes(0.5, Grayscale(alpha=(0.0, 1.0))),], random_order=True)', 'COLOR_AUG_SYN_ONLY': False, 'RANDOM_FLIP': 'none', 'WITH_BG_DEPTH': False, 'BG_DEPTH_FACTOR': 10000.0, 'BG_TYPE': 'VOC_table', 'BG_IMGS_ROOT': 'datasets/VOCdevkit/VOC2012/', 'NUM_BG_IMGS': 10000, 'CHANGE_BG_PROB': 0.5, 'TRUNCATE_FG': False, 'BG_KEEP_ASPECT_RATIO': True, 'DZI_TYPE': 'uniform', 'DZI_PAD_SCALE': 1.5, 'DZI_SCALE_RATIO': 0.25, 'DZI_SHIFT_RATIO': 0.25, 'SMOOTH_XYZ': False}, 'DATASETS': {'TRAIN': ('itodd_train_pbr',), 'TRAIN2': (), 'TRAIN2_RATIO': 0.0, 'DATA_LEN_WITH_TRAIN2': True, 'PROPOSAL_FILES_TRAIN': (), 'PRECOMPUTED_PROPOSAL_TOPK_TRAIN': 2000, 'TEST': ('itodd_bop_test',), 'PROPOSAL_FILES_TEST': (), 'PRECOMPUTED_PROPOSAL_TOPK_TEST': 1000, 'DET_FILES_TRAIN': (), 'DET_TOPK_PER_OBJ_TRAIN': 1, 'DET_TOPK_PER_IM_TRAIN': 30, 'DET_THR_TRAIN': 0.0, 'DET_FILES_TEST': ('datasets/BOP_DATASETS/itodd/test/test_bboxes/yolox_x_640_itodd_pbr_itodd_bop_test.json',), 'DET_TOPK_PER_OBJ': 100, 'DET_TOPK_PER_IM': 30, 'DET_THR': 0.05, 'INIT_POSE_FILES_TEST': (), 'INIT_POSE_TOPK_PER_OBJ': 1, 'INIT_POSE_TOPK_PER_IM': 30, 'INIT_POSE_THR': 0.0, 'SYM_OBJS': ['bowl', 'cup', 'eggbox', 'glue'], 'EVAL_SCENE_IDS': None}, 'DATALOADER': {'NUM_WORKERS': 8, 'PERSISTENT_WORKERS': False, 'MAX_OBJS_TRAIN': 120, 'ASPECT_RATIO_GROUPING': False, 'SAMPLER_TRAIN': 'TrainingSampler', 'REPEAT_THRESHOLD': 0.0, 'FILTER_EMPTY_ANNOTATIONS': True, 'FILTER_EMPTY_DETS': True, 'FILTER_VISIB_THR': 0.3, 'REMOVE_ANNO_KEYS': []}, 'SOLVER': {'IMS_PER_BATCH': 48, 'REFERENCE_BS': 48, 'TOTAL_EPOCHS': 40, 'OPTIMIZER_CFG': {'type': 'Ranger', 'lr': 0.0008, 'weight_decay': 0.01}, 'GAMMA': 0.1, 'BIAS_LR_FACTOR': 1.0, 'LR_SCHEDULER_NAME': 'flat_and_anneal', 'WARMUP_METHOD': 'linear', 'WARMUP_FACTOR': 0.001, 'WARMUP_ITERS': 1000, 'ANNEAL_METHOD': 'cosine', 'ANNEAL_POINT': 0.72, 'POLY_POWER': 0.9, 'REL_STEPS': (0.5, 0.75), 'CHECKPOINT_PERIOD': 5, 'CHECKPOINT_BY_EPOCH': True, 'MAX_TO_KEEP': 5, 'CLIP_GRADIENTS': {'ENABLED': False, 'CLIP_TYPE': 'value', 'CLIP_VALUE': 1.0, 'NORM_TYPE': 2.0}, 'SET_NAN_GRAD_TO_ZERO': False, 'AMP': {'ENABLED': False}, 'WEIGHT_DECAY': 0.01, 'OPTIMIZER_NAME': 'Ranger', 'BASE_LR': 0.0008, 'MOMENTUM': 0.9}, 'TRAIN': {'PRINT_FREQ': 100, 'VERBOSE': False, 'VIS': False, 'VIS_IMG': False}, 'VAL': {'DATASET_NAME': 'itodd', 'SCRIPT_PATH': 'lib/pysixd/scripts/eval_pose_results_more.py', 'RESULTS_PATH': '', 'TARGETS_FILENAME': 'test_targets_bop19.json', 'ERROR_TYPES': 'mspd,mssd,vsd,ad,reS,teS', 'RENDERER_TYPE': 'cpp', 'SPLIT': 'test', 'SPLIT_TYPE': '', 'N_TOP': -1, 'EVAL_CACHED': False, 'SCORE_ONLY': False, 'EVAL_PRINT_ONLY': False, 'EVAL_PRECISION': False, 'USE_BOP': True, 'SAVE_BOP_CSV_ONLY': True}, 'TEST': {'EVAL_PERIOD': 0, 'VIS': False, 'TEST_BBOX_TYPE': 'est', 'PRECISE_BN': {'ENABLED': False, 'NUM_ITER': 200}, 'AMP_TEST': False, 'COLOR_AUG': False, 'USE_PNP': False, 'SAVE_RESULTS_ONLY': False, 'PNP_TYPE': 'ransac_pnp', 'USE_DEPTH_REFINE': False, 'DEPTH_REFINE_ITER': 2, 'DEPTH_REFINE_THRESHOLD': 0.8, 'USE_COOR_Z_REFINE': False}, 'DIST_PARAMS': {'backend': 'nccl'}, 'MODEL': {'DEVICE': 'cuda', 'WEIGHTS': '', 'PIXEL_MEAN': [0.0, 0.0, 0.0], 'PIXEL_STD': [255.0, 255.0, 255.0], 'LOAD_DETS_TEST': True, 'BBOX_CROP_REAL': False, 'BBOX_CROP_SYN': False, 'BBOX_TYPE': 'AMODAL_CLIP', 'EMA': {'ENABLED': False, 'INIT_CFG': {'decay': 0.9999, 'updates': 0}}, 'POSE_NET': {'NAME': 'GDRN_double_mask', 'XYZ_ONLINE': True, 'XYZ_BP': True, 'NUM_CLASSES': 28, 'USE_MTL': False, 'INPUT_RES': 256, 'OUTPUT_RES': 64, 'BACKBONE': {'FREEZE': False, 'PRETRAINED': 'timm', 'INIT_CFG': {'type': 'timm/convnext_base', 'in_chans': 3, 'features_only': True, 'pretrained': True, 'out_indices': (3,)}}, 'DEPTH_BACKBONE': {'ENABLED': False, 'FREEZE': False, 'PRETRAINED': 'timm', 'INIT_CFG': {'type': 'timm/resnet18', 'in_chans': 1, 'features_only': True, 'pretrained': True, 'out_indices': (4,)}}, 'FUSE_RGBD_TYPE': 'cat', 'NECK': {'ENABLED': False, 'FREEZE': False, 'LR_MULT': 1.0, 'INIT_CFG': {'type': 'FPN', 'in_channels': [256, 512, 1024, 2048], 'out_channels': 256, 'num_outs': 4}}, 'GEO_HEAD': {'FREEZE': False, 'LR_MULT': 1.0, 'INIT_CFG': {'type': 'TopDownDoubleMaskXyzRegionHead', 'in_dim': 1024, 'up_types': ('deconv', 'bilinear', 'bilinear'), 'deconv_kernel_size': 3, 'num_conv_per_block': 2, 'feat_dim': 256, 'feat_kernel_size': 3, 'norm': 'GN', 'num_gn_groups': 32, 'act': 'GELU', 'out_kernel_size': 1, 'out_layer_shared': True}, 'XYZ_BIN': 64, 'XYZ_CLASS_AWARE': True, 'MASK_CLASS_AWARE': True, 'REGION_CLASS_AWARE': True, 'MASK_THR_TEST': 0.5, 'NUM_REGIONS': 64}, 'PNP_NET': {'FREEZE': False, 'LR_MULT': 1.0, 'INIT_CFG': {'type': 'ConvPnPNet', 'norm': 'GN', 'act': 'gelu', 'num_gn_groups': 32, 'drop_prob': 0.0, 'denormalize_by_extent': True}, 'WITH_2D_COORD': True, 'COORD_2D_TYPE': 'abs', 'REGION_ATTENTION': True, 'MASK_ATTENTION': 'none', 'ROT_TYPE': 'allo_rot6d', 'TRANS_TYPE': 'centroid_z', 'Z_TYPE': 'REL'}, 'LOSS_CFG': {'XYZ_LOSS_TYPE': 'L1', 'XYZ_LOSS_MASK_GT': 'visib', 'XYZ_LW': 1.0, 'FULL_MASK_LOSS_TYPE': 'L1', 'FULL_MASK_LW': 1.0, 'MASK_LOSS_TYPE': 'L1', 'MASK_LOSS_GT': 'trunc', 'MASK_LW': 1.0, 'REGION_LOSS_TYPE': 'CE', 'REGION_LOSS_MASK_GT': 'visib', 'REGION_LW': 1.0, 'NUM_PM_POINTS': 3000, 'PM_LOSS_TYPE': 'L1', 'PM_SMOOTH_L1_BETA': 1.0, 'PM_LOSS_SYM': True, 'PM_NORM_BY_EXTENT': False, 'PM_R_ONLY': True, 'PM_DISENTANGLE_T': False, 'PM_DISENTANGLE_Z': False, 'PM_T_USE_POINTS': True, 'PM_LW': 1.0, 'ROT_LOSS_TYPE': 'angular', 'ROT_LW': 0.0, 'CENTROID_LOSS_TYPE': 'L1', 'CENTROID_LW': 1.0, 'Z_LOSS_TYPE': 'L1', 'Z_LW': 1.0, 'TRANS_LOSS_TYPE': 'L1', 'TRANS_LOSS_DISENTANGLE': True, 'TRANS_LW': 0.0, 'BIND_LOSS_TYPE': 'L1', 'BIND_LW': 0.0}}, 'KEYPOINT_ON': False, 'LOAD_PROPOSALS': False}, 'EXP_ID': 'convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd', 'RESUME': False}
Global seed set to 11295325
20221126_142411|core.utils.default_args_setup@144: Full config saved to output/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_itodd.py
20221126_142411|d2.utils.env@41: Using a generated random seed 11295325
20221126_142411|core.utils.default_args_setup@162: Used mmcv backend: cv2
20221126_142411|DBG|OpenGL.platform.ctypesloader@65: Loaded libEGL.so => libEGL.so.1 <CDLL 'libEGL.so.1', handle 8630300 at 0x7f537f987820>
WARN
20221126_142411|ERR|__main__@233: An error has been caught in function '<module>', process 'MainProcess' (5946), thread 'MainThread' (139998642771776):
Traceback (most recent call last):

> File "/notebooks/gdrnpp_bop2022/./core/gdrn_modeling/main_gdrn.py", line 233, in <module>
    main(args)
    │    └ Namespace(config_file='/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClip...
    └ <function main at 0x7f530caa1b80>

  File "/notebooks/gdrnpp_bop2022/./core/gdrn_modeling/main_gdrn.py", line 199, in main
    Lite(
    └ <class '__main__.Lite'>

  File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/lite/lite.py", line 408, in _run_impl
    return run_method(*args, **kwargs)
           │           │       └ {}
           │           └ (Namespace(config_file='/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalCli...
           └ functools.partial(<bound method LightningLite._run_with_strategy_setup of <__main__.Lite object at 0x7f53f92ee310>>, <bound m...

  File "/usr/local/lib/python3.9/dist-packages/pytorch_lightning/lite/lite.py", line 413, in _run_with_strategy_setup
    return run_method(*args, **kwargs)
           │           │       └ {}
           │           └ (Namespace(config_file='/notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalCli...
           └ <bound method Lite.run of <__main__.Lite object at 0x7f53f92ee310>>

  File "/notebooks/gdrnpp_bop2022/./core/gdrn_modeling/main_gdrn.py", line 155, in run
    renderer = get_renderer(cfg, data_ref, obj_names=train_obj_names, gpu_id=render_gpu_id)
               │            │    │                   │                       └ 0
               │            │    │                   └ ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '...
               │            │    └ <module 'ref.itodd' from '/notebooks/gdrnpp_bop2022/core/gdrn_modeling/../../ref/itodd.py'>
               │            └ Config (path: /notebooks/gdrnpp_bop2022/configs/gdrn/itodd_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_class...
               └ <function get_renderer at 0x7f530cfde820>

  File "/notebooks/gdrnpp_bop2022/core/gdrn_modeling/../../core/gdrn_modeling/engine/engine_utils.py", line 280, in get_renderer
    ren = EGLRenderer(
          └ <class 'lib.egl_renderer.egl_renderer_v3.EGLRenderer'>

  File "/notebooks/gdrnpp_bop2022/core/gdrn_modeling/../../lib/egl_renderer/egl_renderer_v3.py", line 81, in __init__
    self._context = OffscreenContext(gpu_id=cuda_device_idx)
    │               │                       └ 0
    │               └ <class 'lib.egl_renderer.glutils.egl_offscreen_context.OffscreenContext'>
    └ <lib.egl_renderer.egl_renderer_v3.EGLRenderer object at 0x7f537f987100>

  File "/notebooks/gdrnpp_bop2022/core/gdrn_modeling/../../lib/egl_renderer/glutils/egl_offscreen_context.py", line 157, in __init__
    self.init_context()
    │    └ <function OffscreenContext.init_context at 0x7f530e98e940>
    └ <lib.egl_renderer.glutils.egl_offscreen_context.OffscreenContext object at 0x7f537f987370>

  File "/notebooks/gdrnpp_bop2022/core/gdrn_modeling/../../lib/egl_renderer/glutils/egl_offscreen_context.py", line 227, in init_context
    raise RuntimeError("Bindless Textures not supported")

RuntimeError: Bindless Textures not supported

I could install all the required pacakges in scripts/install_deps.sh and also compiled the egl_renderer via sh ./lib/egl_renderer/compile_cpp_egl_renderer.sh. After the compilation of the egl_renderer, the query devices identifies the gpu:

root@n156egjsrm:/notebooks/gdrnpp_bop2022/lib/egl_renderer# ./build/query_devices
query devices:
num devices: 1

However, the example program egl_renderer_v3 fails with the same errors as the training program:

root@n156egjsrm:/notebooks/gdrnpp_bop2022# python -m lib.egl_renderer.egl_renderer_v3
/usr/local/lib/python3.9/dist-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
WARN
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/notebooks/gdrnpp_bop2022/lib/egl_renderer/egl_renderer_v3.py", line 1415, in <module>
    renderer = EGLRenderer(
  File "/notebooks/gdrnpp_bop2022/lib/egl_renderer/egl_renderer_v3.py", line 81, in __init__
    self._context = OffscreenContext(gpu_id=cuda_device_idx)
  File "/notebooks/gdrnpp_bop2022/lib/egl_renderer/glutils/egl_offscreen_context.py", line 157, in __init__
    self.init_context()
  File "/notebooks/gdrnpp_bop2022/lib/egl_renderer/glutils/egl_offscreen_context.py", line 227, in init_context
    raise RuntimeError("Bindless Textures not supported")
RuntimeError: Bindless Textures not supported

The program in run in an google-colab type environment.

Do you have any idea how to fix this issue?

FabianSchuetze commented 1 year ago

What is the egl_renderer actually needed for? I think its just imported in 

pysixd/scripts/eval_calc_errors.py
meshrenderer/meshrenderer_texture_color_tensor.py

I guess I could calculate the errors somewhere else and the meshrenderer calcualtes the NOCs, I guess? Do the NOCs need to be available on disk, as they were in GDR-Net, so that I could pre-calculate them on a different machine and the use them on a different server? Could I also use the meshrenderer_texture_color.py for example?

wangg12 commented 1 year ago

Maybe your cuda driver is not the opengl cuda driver. Reinstalling the driver from https://developer.nvidia.com/opengl-driver might solve your problem.

Egl renderer is needed for training with XYZ_ONLINE, where the object coordinates are rendered online.

Training with pre-generated coordinates is also possible by disabling the option, but it might not be well supported since we have stopped maintaining this feature for a long time. Saving the coordinates on the disk also needs a huge amount of storage space.

FabianSchuetze commented 1 year ago

Thanks for the recommendations so far.

I had no luck working with the egl renderer so far though. I will try with a dedicate docker nvidia opengl environment soon.

However, I had some luck working with vispy renderer. Would it be possible to use this to generate the Nocs for me? If so, what (roughly) would i have to change to use this?

wangg12 commented 1 year ago

FYI: https://github.com/THU-DA-6D-Pose-Group/GDR-Net/blob/main/tools/lm/lm_pbr_1_gen_xyz_crop.py#L185

z7r7y7 commented 1 year ago

Hello！I also encountered a runtime error "RuntimeError: Bindless Textures not supported" while running the gdrnpp model code. raise RuntimeError("Bindless Textures not supported") The following is the entire log of the program:

----------------------------------------------------------------------------------------------------

20230725_073142|core.utils.default_args_setup@123: Rank of current process: 0. World size: 2
20230725_073143|core.utils.default_args_setup@124: Environment info:
-------------------------------  --------------------------------------------------------------------------
sys.platform                     linux
Python                           3.8.10 (default, May 26 2023, 14:05:08) [GCC 9.4.0]
numpy                            1.24.4
detectron2                       0.6 @/home/appuser/detectron2_repo/detectron2
Compiler                         GCC 9.4
CUDA compiler                    CUDA 11.7
detectron2 arch flags            3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 7.0, 7.5
DETECTRON2_ENV_MODULE            <not set>
PyTorch                          2.0.1+cu117 @/home/appuser/.local/lib/python3.8/site-packages/torch
PyTorch debug build              False
torch._C._GLIBCXX_USE_CXX11_ABI  False
GPU available                    Yes
GPU 0,1                          NVIDIA GeForce RTX 3090 (arch=8.6)
Driver version                   535.86.05
CUDA_HOME                        /usr/local/cuda
TORCH_CUDA_ARCH_LIST             Kepler;Kepler+Tesla;Maxwell;Maxwell+Tegra;Pascal;Volta;Turing
Pillow                           9.0.0.post1
torchvision                      0.15.2+cu117 @/home/appuser/.local/lib/python3.8/site-packages/torchvision
torchvision arch flags           3.5, 5.0, 6.0, 7.0, 7.5, 8.0, 8.6
fvcore                           0.1.5.post20221221
iopath                           0.1.9
cv2                              4.8.0
-------------------------------  --------------------------------------------------------------------------
PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.7.3 (Git Hash 6dbeffbae1f23cbbeae17adb7b5b13f1f37c080e)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: NO AVX
  - CUDA Runtime 11.7
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
  - CuDNN 8.5
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

20230725_073143|core.utils.default_args_setup@126: Command line arguments: Namespace(config_file='./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py', dist_url='tcp://127.0.0.1:50152', eval_only=False, fp16_allreduce=False, launcher='none', local_rank=0, machine_rank=0, num_gpus=2, num_machines=1, opts=None, resume=False, strategy='ddp', use_adasum=False)
20230725_073143|core.utils.default_args_setup@128: Contents of args.config_file=./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py:
_base_ = ["../../_base_/gdrn_base.py"]

OUTPUT_DIR = "output/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo"
INPUT = dict(
    DZI_PAD_SCALE=1.5,
    TRUNCATE_FG=False,
    CHANGE_BG_PROB=0.5,
    COLOR_AUG_PROB=0.8,
    COLOR_AUG_TYPE="code",
    COLOR_AUG_CODE=(
        "Sequential(["
        # Sometimes(0.5, PerspectiveTransform(0.05)),
        # Sometimes(0.5, CropAndPad(percent=(-0.05, 0.1))),
        # Sometimes(0.5, Affine(scale=(1.0, 1.2))),
        "Sometimes(0.5, CoarseDropout( p=0.2, size_percent=0.05) ),"
        "Sometimes(0.4, GaussianBlur((0., 3.))),"
        "Sometimes(0.3, pillike.EnhanceSharpness(factor=(0., 50.))),"
        "Sometimes(0.3, pillike.EnhanceContrast(factor=(0.2, 50.))),"
        "Sometimes(0.5, pillike.EnhanceBrightness(factor=(0.1, 6.))),"
        "Sometimes(0.3, pillike.EnhanceColor(factor=(0., 20.))),"
        "Sometimes(0.5, Add((-25, 25), per_channel=0.3)),"
        "Sometimes(0.3, Invert(0.2, per_channel=True)),"
        "Sometimes(0.5, Multiply((0.6, 1.4), per_channel=0.5)),"
        "Sometimes(0.5, Multiply((0.6, 1.4))),"
        "Sometimes(0.1, AdditiveGaussianNoise(scale=10, per_channel=True)),"
        "Sometimes(0.5, iaa.contrast.LinearContrast((0.5, 2.2), per_channel=0.3)),"
        "Sometimes(0.5, Grayscale(alpha=(0.0, 1.0))),"  # maybe remove for det
        "], random_order=True)"
        # cosy+aae
    ),
)

SOLVER = dict(
    IMS_PER_BATCH=48,
    TOTAL_EPOCHS=40,  # 30
    LR_SCHEDULER_NAME="flat_and_anneal",
    ANNEAL_METHOD="cosine",  # "cosine"
    ANNEAL_POINT=0.72,
    OPTIMIZER_CFG=dict(_delete_=True, type="Ranger", lr=8e-4, weight_decay=0.01),
    WEIGHT_DECAY=0.0,
    WARMUP_FACTOR=0.001,
    WARMUP_ITERS=1000,
)

DATASETS = dict(
    TRAIN=("lmo_pbr_train",),
    TEST=("lmo_bop_test",),
    DET_FILES_TEST=("datasets/BOP_DATASETS/lmo/test/test_bboxes/yolox_x_640_lmo_pbr_lmo_bop_test.json",),
)

DATALOADER = dict(
    # Number of data loading threads
    NUM_WORKERS=2,
    FILTER_VISIB_THR=0.3,
)

MODEL = dict(
    LOAD_DETS_TEST=True,
    PIXEL_MEAN=[0.0, 0.0, 0.0],
    PIXEL_STD=[255.0, 255.0, 255.0],
    BBOX_TYPE="AMODAL_CLIP",  # VISIB or AMODAL
    POSE_NET=dict(
        NAME="GDRN_double_mask",
        XYZ_ONLINE=True,
        NUM_CLASSES=8,
        BACKBONE=dict(
            FREEZE=False,
            PRETRAINED="timm",
            INIT_CFG=dict(
                type="timm/convnext_base",
                pretrained=True,
                in_chans=3,
                features_only=True,
                out_indices=(3,),
            ),
        ),
        ## geo head: Mask, XYZ, Region
        GEO_HEAD=dict(
            FREEZE=False,
            INIT_CFG=dict(
                type="TopDownDoubleMaskXyzRegionHead",
                in_dim=1024,  # this is num out channels of backbone conv feature
            ),
            NUM_REGIONS=64,
            XYZ_CLASS_AWARE=True,
            MASK_CLASS_AWARE=True,
            REGION_CLASS_AWARE=True,
        ),
        PNP_NET=dict(
            INIT_CFG=dict(norm="GN", act="gelu"),
            REGION_ATTENTION=True,
            WITH_2D_COORD=True,
            ROT_TYPE="allo_rot6d",
            TRANS_TYPE="centroid_z",
        ),
        LOSS_CFG=dict(
            # xyz loss ----------------------------
            XYZ_LOSS_TYPE="L1",  # L1 | CE_coor
            XYZ_LOSS_MASK_GT="visib",  # trunc | visib | obj
            XYZ_LW=1.0,
            # mask loss ---------------------------
            MASK_LOSS_TYPE="L1",  # L1 | BCE | CE
            MASK_LOSS_GT="trunc",  # trunc | visib | gt
            MASK_LW=1.0,
            # full mask loss ---------------------------
            FULL_MASK_LOSS_TYPE="L1",  # L1 | BCE | CE
            FULL_MASK_LW=1.0,
            # region loss -------------------------
            REGION_LOSS_TYPE="CE",  # CE
            REGION_LOSS_MASK_GT="visib",  # trunc | visib | obj
            REGION_LW=1.0,
            # pm loss --------------
            PM_LOSS_SYM=True,  # NOTE: sym loss
            PM_R_ONLY=True,  # only do R loss in PM
            PM_LW=1.0,
            # centroid loss -------
            CENTROID_LOSS_TYPE="L1",
            CENTROID_LW=1.0,
            # z loss -----------
            Z_LOSS_TYPE="L1",
            Z_LW=1.0,
        ),
    ),
)

VAL = dict(
    DATASET_NAME="lmo",
    SCRIPT_PATH="lib/pysixd/scripts/eval_pose_results_more.py",
    TARGETS_FILENAME="test_targets_bop19.json",
    ERROR_TYPES="mspd,mssd,vsd,ad,reS,teS",
    RENDERER_TYPE="cpp",  # cpp, python, egl
    SPLIT="test",
    SPLIT_TYPE="",
    N_TOP=1,  # SISO: 1, VIVO: -1 (for LINEMOD, 1/-1 are the same)
    EVAL_CACHED=False,  # if the predicted poses have been saved
    SCORE_ONLY=False,  # if the errors have been calculated
    EVAL_PRINT_ONLY=False,  # if the scores/recalls have been saved
    EVAL_PRECISION=False,  # use precision or recall
    USE_BOP=True,  # whether to use bop toolkit
)

TEST = dict(EVAL_PERIOD=0, VIS=False, TEST_BBOX_TYPE="est")  # gt | est`

And details of the error:

20230725_073144|core.utils.default_args_setup@144: Full config saved to output/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py
Global seed set to 44706856
20230725_073144|d2.utils.env@41: Using a generated random seed 44706856
20230725_073144|core.utils.default_args_setup@162: Used mmcv backend: cv2
20230725_073144|DBG|OpenGL.platform.ctypesloader@65: Loaded libEGL.so => libEGL.so <CDLL 'libEGL.so', handle a2a9b10 at 0x7f66aeeb8b80>
20230725_073145|ERR|__main__@233: An error has been caught in function '<module>', process 'MainProcess' (36), thread 'MainThread' (140082418370368):
Traceback (most recent call last):

> File "./core/gdrn_modeling/main_gdrn.py", line 233, in <module>
    main(args)
    │    └ Namespace(config_file='./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py', di...
    └ <function main at 0x7f66af0c8040>

  File "./core/gdrn_modeling/main_gdrn.py", line 199, in main
    Lite(
    └ <class '__main__.Lite'>

  File "/home/appuser/.local/lib/python3.8/site-packages/pytorch_lightning/lite/lite.py", line 406, in _run_impl
    return self._strategy.launcher.launch(run_method, *args, **kwargs)
           │    │         │               │            │       └ {}
           │    │         │               │            └ (Namespace(config_file='./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py', d...
           │    │         │               └ functools.partial(<bound method LightningLite._run_with_strategy_setup of <__main__.Lite object at 0x7f6649eb3c10>>, <bound m...
           │    │         └ <property object at 0x7f66e8411360>
           │    └ <pytorch_lightning.strategies.ddp.DDPStrategy object at 0x7f66af029730>
           └ <__main__.Lite object at 0x7f6649eb3c10>
  File "/home/appuser/.local/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
           │         │       └ {}
           │         └ (Namespace(config_file='./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py', d...
           └ functools.partial(<bound method LightningLite._run_with_strategy_setup of <__main__.Lite object at 0x7f6649eb3c10>>, <bound m...
  File "/home/appuser/.local/lib/python3.8/site-packages/pytorch_lightning/lite/lite.py", line 413, in _run_with_strategy_setup
    return run_method(*args, **kwargs)
           │           │       └ {}
           │           └ (Namespace(config_file='./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py', d...
           └ <bound method Lite.run of <__main__.Lite object at 0x7f6649eb3c10>>

  File "./core/gdrn_modeling/main_gdrn.py", line 155, in run
    renderer = get_renderer(cfg, data_ref, obj_names=train_obj_names, gpu_id=render_gpu_id)
               │            │    │                   │                       └ 0
               │            │    │                   └ ['ape', 'can', 'cat', 'driller', 'duck', 'eggbox', 'glue', 'holepuncher']
               │            │    └ <module 'ref.lmo_full' from '/home/appuser/gdrnpp-docker/core/gdrn_modeling/../../ref/lmo_full.py'>
               │            └ Config (path: ./configs/gdrn/lmo_pbr/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_lmo.py): {'OUTPUT_RO...
               └ <function get_renderer at 0x7f664a14c5e0>

  File "/home/appuser/gdrnpp-docker/core/gdrn_modeling/../../core/gdrn_modeling/engine/engine_utils.py", line 280, in get_renderer
    ren = EGLRenderer(
          └ <class 'lib.egl_renderer.egl_renderer_v3.EGLRenderer'>

  File "/home/appuser/gdrnpp-docker/core/gdrn_modeling/../../lib/egl_renderer/egl_renderer_v3.py", line 81, in __init__
    self._context = OffscreenContext(gpu_id=cuda_device_idx)
    │               │                       └ 0
    │               └ <class 'lib.egl_renderer.glutils.egl_offscreen_context.OffscreenContext'>
    └ <lib.egl_renderer.egl_renderer_v3.EGLRenderer object at 0x7f66aeeb8fa0>

  File "/home/appuser/gdrnpp-docker/core/gdrn_modeling/../../lib/egl_renderer/glutils/egl_offscreen_context.py", line 157, in __init__
    self.init_context()
    │    └ <function OffscreenContext.init_context at 0x7f664a137a60>
    └ <lib.egl_renderer.glutils.egl_offscreen_context.OffscreenContext object at 0x7f66aeeb8f40>

  File "/home/appuser/gdrnpp-docker/core/gdrn_modeling/../../lib/egl_renderer/glutils/egl_offscreen_context.py", line 226, in init_context
    raise RuntimeError("Bindless Textures not supported")

RuntimeError: Bindless Textures not supported
20230725_073145|DBG|filelock._api@212: Attempting to acquire lock 140077390192112 on /home/appuser/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@215: Lock 140077390192112 acquired on /home/appuser/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@244: Attempting to release lock 140077390192112 on /home/appuser/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@247: Lock 140077390192112 released on /home/appuser/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@212: Attempting to acquire lock 140077390192112 on /home/appuser/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@215: Lock 140077390192112 acquired on /home/appuser/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@244: Attempting to release lock 140077390192112 on /home/appuser/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
20230725_073145|DBG|filelock._api@247: Lock 140077390192112 released on /home/appuser/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock`

I tried the solution mentioned by you to update the NVIDIA driver to version 535.86.05, but the problem was not resolved. I can't find detailed information and system requirements about the Bindless Texture feature. If you have any experience with it, could you please share the relevant details？

Or, if possible, could you provide the system configuration information that allowed you to successfully run gdrnpp?（Such as the system version, graphics card driver, OpenGL version, etc.）

Looking forward to your response. Thank you!

wangg12 commented 1 year ago

Please first make sure your environment can successfully run other egl programs such as https://github.com/vispy/vispy/blob/main/examples/offscreen/simple_egl.py , https://github.com/DLR-RM/AugmentedAutoencoder#headless-rendering , https://github.com/thodan/bop_toolkit#vispy-renderer-default .

z7r7y7 commented 1 year ago

Thank you very much for your help！ I followed your advice and ran the programs https://github.com/vispy/vispy/blob/main/examples/offscreen/simple_egl.py and https://github.com/thodan/bop_toolkit#vispy-renderer-default, both of which ran successfully. It seems that the EGL renderer in the environment is working fine. However, the issue with Bindless Texture still persists. Do you have any other suggestions to resolve this problem? Any information would be greatly appreciated. Thank you!

z7r7y7 commented 1 year ago

Hello！I wanted to inform you that the issue mentioned earlier has been resolved. However, I cannot pinpoint the exact step that proved crucial, because I took multiple measures to address the problem.Firstly, I updated the NVIDIA driver to version 535.86.05. Next, I upgraded OpenGL to version 3.1. During the reinstallation of OpenGL, I also made modifications to the versions of some related resource libraries. Fortunately, after following these steps, I haven't encountered the previous error anymore. I am delighted by this outcome and appreciate your assistance and guidance in resolving the issue.

Thank you once again for your patient support.

utsavrai commented 2 months ago

I am getting the same error in a conda environment in a remote gpu server. However the exact same configuration of conda environment works fine in my local computer system. Both the driver, cuda and other dependency versions are same. I was able to run https://github.com/vispy/vispy/blob/main/examples/offscreen/simple_egl.py as suggested. But no luck in resolving this issue.

wangg12 commented 2 months ago

Are you using the opengl cuda driver?