search

IGNF / myria3d

Myria3D: Aerial Lidar HD Semantic Segmentation with Deep Learning
https://ignf.github.io/myria3d/
BSD 3-Clause "New" or "Revised" License
169 stars 23 forks source link

Running predict on raw IGN Lidar HD dataset returns error #55

Closed phardy-egis closed 1 year ago

phardy-egis commented 1 year ago

I ran a predict treatment with the following configuration:

The command ran is the following:

python -m myria3d.predict --config-path /model/ --config-name proto151_V2.0_epoch_100_Myria3DV3.1.0_predict_config_V3.2.0.yaml predict.src_las=/lidar/input/Semis_2021_0772_6279_LA93_IGN69.laz predict.output_dir=/lidar/output predict.ckpt_path=/model/proto151_V2.0_epoch_100_Myria3DV3.1.0.ckpt predict.gpus=0 datamodule.batch_size=10 hydra.run.dir=/log/hydra

The predict treatment end up with an error: 

File "/app/myria3d/pctl/points_pre_transform/lidar_hd.py", line 36, in lidar_hd_pre_transform
assert points[color].max() <= COLORS_NORMALIZATION_MAX_VALUE
ValueError: no field of name Red

How can this error be fixed ?

Complete log of the predict command ``` (base) root@c2e8b4ce7daa:/app# python -m myria3d.predict --config-path /model/ --config-name proto151_V2.0_epoch_100_Myria3DV3.1.0_predict_config_V3.2.0.yaml predict.src_las=/lidar/input/Semis_2021_0772_6279_LA93_IGN69.laz predict.output_dir=/lidar/output predict.ckpt_path=/model/proto151_V2.0_epoch_100_Myria3DV3.1.0.ckpt predict.gpus=0 datamodule.batch_size=10 hydra.run.dir=/log/hydra CONFIG ├── task │ └── task_name: predict │ ├── seed │ └── 12345 ├── logger │ └── comet: │ _target_: pytorch_lightning.loggers.comet.CometLogger │ api_key: ${oc.env:COMET_API_TOKEN} │ workspace: ${oc.env:COMET_WORKSPACE} │ project_name: ${oc.env:COMET_PROJECT_NAME} │ experiment_name: '[V3.0.2-BS10xMAX40000pts] RandLaNet_base_run_FR_pyg_randla_net_NoRS' │ auto_log_co2: false │ disabled: false │ ├── trainer │ └── _target_: pytorch_lightning.Trainer │ gpus: │ - 0 │ - 1 │ min_epochs: 100 │ max_epochs: 150 │ log_every_n_steps: 1 │ weights_summary: null │ progress_bar_refresh_rate: 1 │ auto_lr_find: false │ num_sanity_val_steps: 2 │ accumulate_grad_batches: 3 │ strategy: ddp │ accelerator: gpu │ num_processes: 2 │ ├── model │ └── optimizer: │ _target_: functools.partial │ _args_: │ - ${get_method:torch.optim.Adam} │ lr: ${model.lr} │ lr_scheduler: │ _target_: functools.partial │ _args_: │ - ${get_method:torch.optim.lr_scheduler.ReduceLROnPlateau} │ mode: min │ factor: 0.5 │ patience: 20 │ cooldown: 5 │ verbose: true │ criterion: │ _target_: torch.nn.CrossEntropyLoss │ label_smoothing: 0.0 │ _target_: myria3d.models.model.Model │ d_in: ${dataset_description.d_in} │ num_classes: ${dataset_description.num_classes} │ ckpt_path: null │ neural_net_class_name: PyGRandLANet │ neural_net_hparams: │ num_features: ${model.d_in} │ num_classes: ${model.num_classes} │ num_neighbors: 16 │ decimation: 4 │ return_logits: true │ interpolation_k: ${predict.interpolator.interpolation_k} │ num_workers: 4 │ iou: │ _target_: functools.partial │ _args_: │ - ${get_method:torchmetrics.JaccardIndex} │ - ${model.num_classes} │ absent_score: 1.0 │ momentum: 0.9 │ monitor: val/loss_epoch │ lr: 0.003933709606504788 │ ├── datamodule │ └── transforms: │ preparations: │ train: │ TargetTransform: │ _target_: myria3d.pctl.transforms.transforms.TargetTransform │ _args_: │ - ${dataset_description.classification_preprocessing_dict} │ - ${dataset_description.classification_dict} │ GridSampling: │ _target_: torch_geometric.transforms.GridSampling │ _args_: │ - 0.25 │ MinimumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MinimumNumNodes │ _args_: │ - 300 │ MaximumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MaximumNumNodes │ _args_: │ - 40000 │ Center: │ _target_: torch_geometric.transforms.Center │ eval: │ TargetTransform: │ _target_: myria3d.pctl.transforms.transforms.TargetTransform │ _args_: │ - ${dataset_description.classification_preprocessing_dict} │ - ${dataset_description.classification_dict} │ CopyFullPreparedTargets: │ _target_: myria3d.pctl.transforms.transforms.CopyFullPreparedTargets │ CopyFullPos: │ _target_: myria3d.pctl.transforms.transforms.CopyFullPos │ GridSampling: │ _target_: torch_geometric.transforms.GridSampling │ _args_: │ - 0.25 │ MinimumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MinimumNumNodes │ _args_: │ - 300 │ MaximumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MaximumNumNodes │ _args_: │ - 40000 │ CopySampledPos: │ _target_: myria3d.pctl.transforms.transforms.CopySampledPos │ Center: │ _target_: torch_geometric.transforms.Center │ predict: │ CopyFullPos: │ _target_: myria3d.pctl.transforms.transforms.CopyFullPos │ GridSampling: │ _target_: torch_geometric.transforms.GridSampling │ _args_: │ - 0.25 │ MinimumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MinimumNumNodes │ _args_: │ - 300 │ MaximumNumNodes: │ _target_: myria3d.pctl.transforms.transforms.MaximumNumNodes │ _args_: │ - 40000 │ CopySampledPos: │ _target_: myria3d.pctl.transforms.transforms.CopySampledPos │ Center: │ _target_: torch_geometric.transforms.Center │ augmentations: │ x_flip: │ _target_: torch_geometric.transforms.RandomFlip │ _args_: │ - 0 │ p: 0.5 │ y_flip: │ _target_: torch_geometric.transforms.RandomFlip │ _args_: │ - 1 │ p: 0.5 │ normalizations: │ NullifyLowestZ: │ _target_: myria3d.pctl.transforms.transforms.NullifyLowestZ │ NormalizePos: │ _target_: myria3d.pctl.transforms.transforms.NormalizePos │ subtile_width: ${datamodule.subtile_width} │ StandardizeRGBAndIntensity: │ _target_: myria3d.pctl.transforms.transforms.StandardizeRGBAndIntensity │ augmentations_list: '${oc.dict.values: datamodule.transforms.augmentations}' │ preparations_train_list: '${oc.dict.values: datamodule.transforms.preparations.train}' │ preparations_eval_list: '${oc.dict.values: datamodule.transforms.preparations.eval}' │ preparations_predict_list: '${oc.dict.values: datamodule.transforms.preparations.predict}' │ normalizations_list: '${oc.dict.values: datamodule.transforms.normalizations}' │ _target_: myria3d.pctl.datamodule.hdf5.HDF5LidarDataModule │ data_dir: null │ split_csv_path: null │ hdf5_file_path: /var/data/cgaydon/data/20220607_151_dalles_proto.myria3d-V2.4.hdf5 │ points_pre_transform: │ _target_: functools.partial │ _args_: │ - ${get_method:myria3d.pctl.points_pre_transform.lidar_hd.lidar_hd_pre_transform} │ pre_filter: │ _target_: functools.partial │ _args_: │ - ${get_method:myria3d.pctl.dataset.utils.pre_filter_below_n_points} │ min_num_nodes: 50 │ tile_width: 1000 │ subtile_width: 50 │ subtile_shape: square │ subtile_overlap_train: 0 │ subtile_overlap_predict: ${predict.subtile_overlap} │ batch_size: 10 │ num_workers: 3 │ prefetch_factor: 3 │ ├── dataset_description │ └── _convert_: all │ classification_preprocessing_dict: │ 3: 5 │ 4: 5 │ 160: 64 │ 161: 64 │ 162: 64 │ 0: 1 │ 7: 1 │ 46: 1 │ 47: 1 │ 48: 1 │ 49: 1 │ 50: 1 │ 51: 1 │ 52: 1 │ 53: 1 │ 54: 1 │ 55: 1 │ 56: 1 │ 57: 1 │ 58: 1 │ 64: 1 │ 65: 1 │ 66: 1 │ 67: 1 │ 77: 1 │ 155: 1 │ 204: 1 │ classification_dict: │ 1: unclassified │ 2: ground │ 5: vegetation │ 6: building │ 9: water │ 17: bridge │ 64: lasting_above │ class_weights: │ - 0.25 │ - 0.1 │ - 0.1 │ - 0.5 │ - 2.0 │ - 2.0 │ - 2.0 │ d_in: 9 │ num_classes: 7 │ ├── callbacks │ └── log_code: │ _target_: myria3d.callbacks.comet_callbacks.LogCode │ code_dir: ${work_dir}/myria3d │ log_logs_dir: │ _target_: myria3d.callbacks.comet_callbacks.LogLogsPath │ lr_monitor: │ _target_: pytorch_lightning.callbacks.LearningRateMonitor │ logging_interval: step │ log_momentum: true │ log_iou_by_class: │ _target_: myria3d.callbacks.logging_callbacks.LogIoUByClass │ classification_dict: ${dataset_description.classification_dict} │ model_checkpoint: │ _target_: pytorch_lightning.callbacks.ModelCheckpoint │ monitor: val/loss_epoch │ mode: min │ save_top_k: 1 │ save_last: true │ verbose: true │ dirpath: checkpoints/ │ filename: epoch_{epoch:03d} │ auto_insert_metric_name: false │ early_stopping: │ _target_: pytorch_lightning.callbacks.EarlyStopping │ monitor: val/loss_epoch │ mode: min │ patience: 6 │ min_delta: 0 │ └── predict └── src_las: /lidar/input/Semis_2021_0772_6279_LA93_IGN69.laz output_dir: /lidar/output ckpt_path: /model/proto151_V2.0_epoch_100_Myria3DV3.1.0.ckpt subtile_overlap: 0 gpus: 0 interpolator: _target_: myria3d.models.interpolation.Interpolator interpolation_k: 10 classification_dict: ${dataset_description.classification_dict} probas_to_save: - building - ground - vegetation - unclassified predicted_classification_channel: confidence entropy_channel: entropy 0%| | 0/1 [00:00 main() File "/app/myria3d/utils/utils.py", line 159, in timed result = method(*args, **kwargs) File "/miniconda/lib/python3.9/site-packages/hydra/main.py", line 48, in decorated_main _run_hydra( File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra run_and_report( File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 214, in run_and_report raise ex File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 211, in run_and_report return func() File "/miniconda/lib/python3.9/site-packages/hydra/_internal/utils.py", line 378, in lambda: hydra.run( File "/miniconda/lib/python3.9/site-packages/hydra/_internal/hydra.py", line 111, in run _ = ret.return_value File "/miniconda/lib/python3.9/site-packages/hydra/core/utils.py", line 233, in return_value raise self._return_value File "/miniconda/lib/python3.9/site-packages/hydra/core/utils.py", line 160, in run_job ret.return_value = task_function(task_cfg) File "/app/myria3d/predict.py", line 100, in main predict(config) File "/app/myria3d/utils/utils.py", line 159, in timed result = method(*args, **kwargs) File "/app/myria3d/predict.py", line 57, in predict for batch in tqdm(datamodule.predict_dataloader()): File "/miniconda/lib/python3.9/site-packages/tqdm/std.py", line 1185, in __iter__ for obj in iterable: File "/miniconda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 530, in __next__ data = self._next_data() File "/miniconda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1224, in _next_data return self._process_data(data) File "/miniconda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1250, in _process_data data.reraise() File "/miniconda/lib/python3.9/site-packages/torch/_utils.py", line 457, in reraise raise exception ValueError: Caught ValueError in DataLoader worker process 0. Original Traceback (most recent call last): File "/miniconda/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/miniconda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 32, in fetch data.append(next(self.dataset_iter)) File "/app/myria3d/pctl/dataset/iterable.py", line 58, in get_iterator sample_data = self.points_pre_transform(sample_points) File "/app/myria3d/pctl/points_pre_transform/lidar_hd.py", line 36, in lidar_hd_pre_transform assert points[color].max() <= COLORS_NORMALIZATION_MAX_VALUE ValueError: no field of name Red ```
phardy-egis commented 1 year ago

It looks like the code is looking for "Red", "Green", "Blue", "Infrared" (check file here) data fields inside the .LAS file. However, the LIDAR HD I downloaded on IGN website does not includes these fields.

IGN do not seems to provide RVB and Infrarred values as part of the dataset (refer to the metadata description here). Does it means that these values should be retrieved by a third-party data (for instance LANDSAT/SENTINEL ?) and added to the existing dataset prior running a prediction ?

jistiak commented 1 year ago

Hello @phardy-egis, as you already found out that Myria3D needs these "Red", "Green", "Blue", "Infrared" values for each point in the point cloud and you need to colorize your data. Myria3D converts pdal points into torch-geometric Data objects at first. So it needs to have these data to perform the pre-transform.

IGN provides some data that are colorized, for example this one. You can find preprocessed .las files that you can use directly for predictions or fine-tuning. But unfortunately the .laz files you got are not colorized. You need to create a pipeline of your own using PDAL or some other library of preference to colorize the point clouds in bulk in order to use in Myria3D directly. You can find some ideas here.

You can also refer to @CharlesGaydon's reply here.

CharlesGaydon commented 1 year ago

IGN do not seems to provide RVB and Infrarred values as part of the dataset (refer to the metadata description here). Does it means that these values should be retrieved by a third-party data (for instance LANDSAT/SENTINEL ?) and added to the existing dataset prior running a prediction ?

@phardy-egis The LAS should indeed be colorized with additional data : we are using IGN Orthoimages data when training the models. Therefore, it would be optimal to use the same source of colors during inference. I was not the one to perform this task, so I cannot give you more specific advices/scripts on this, sorry!

phardy-egis commented 1 year ago

Thank you very much for these clarifications @CharlesGaydon and @jistiak I will try to colorize the IGN LIDAR HD from the IGN BD ORTHO data and share a piece of code (here in this issue, or maybe a pull request) if I succeed.

CharlesGaydon commented 1 year ago

@phardy-egis I just remembered that I have a repo that does colorization for Swiss Lidar data : Colorize-SwissSURFACE3D-Lidar. Hope it can inspire you. Maybe something similar could be done for French lidar HD, in order to keep the Myria3D coherent (i.e. deep learning only).

CharlesGaydon commented 1 year ago

Closing this.

CharlesGaydon commented 1 year ago

@phardy-egis @jistiak Quick notification to mention that there is in fact an open pip package for RGBNIR colorization of IGN lidar 🥳. It could be better documented, and the code is not open (yet - would probably happen at some point), but I tried it out and it works quite well

I point to it in https://github.com/IGNF/myria3d/issues/75.