search

centreborelli / satnerf

Satellite Neural Radiance Fields
https://centreborelli.github.io/satnerf/
BSD 3-Clause "New" or "Revised" License
122 stars 23 forks source link

Sat-NeRF for improving DEM accuracy. #32

Closed Aventine12 closed 2 months ago

Aventine12 commented 2 months ago

I am working to improve the accuracy of the SRTM DEM in an area in Southeast Asia. The region has dense vegetation, causing the elevation values to be overestimated since the radar cannot penetrate vegetation to reach the ground. This leads to inaccuracies in flood simulations for the area. I have tried to use Sat-NeRF on windows to achieve this. I used WSL/Ubuntu to install some of the dependencies and have updated some dependencies to their newest version.

I have downloaded sentinel-2 MSI products which contains sun angle data and have attempted to create the root_dir from scratch.

This is what I run:

python satnerf\main.py --model sat-nerf --exp_name JAX_068_ds1_sat-nerf --root_dir C:\Users\JohannesBach\Desktop\Sentinel2MSIproducts\S2A_MSIL2A_20230824T033541_N0509_R061_T47NPJ_20230824T085901.SAFE\root_dir --img_dir C:\Users\JohannesBach\Desktop\Sentinel2MSIproducts\S2A_MSIL2A_20230824T033541_N0509_R061_T47NPJ_20230824T085901.SAFE\GRANULE\L2A_T47NPJ_A042672_20230824T035441\IMG_DATA\R10m\img_dir --gt_dir C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\dataset\DFC2019\Track3-Truth --logs_dir C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\dataset\logs --ckpts_dir C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\dataset\ckpts --gpu_id 0

I get this error:

Traceback (most recent call last): File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\main.py", line 263, in main() File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\main.py", line 259, in main trainer.fit(system, ckpt_path=args.ckpt_path if args.ckpt_path else None) File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 543, in fit call._call_and_handle_interrupt( File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\call.py", line 44, in _call_and_handle_interrupt
return trainer_fn(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 579, in _fit_impl self._run(model, ckpt_path=ckpt_path) File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 946, in _run self._data_connector.prepare_data() File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\connectors\data_connector.py", line 100, in prepare_data call._call_lightning_module_hook(trainer, "prepare_data") File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\pytorch_lightning\trainer\call.py", line 159, in _call_lightning_module_hook output = fn(args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\main.py", line 78, in prepare_data self.train_dataset = [] + load_dataset(self.args, split="train") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\datasets__init__.py", line 9, in load_dataset d1 = SatelliteDataset(root_dir=args.root_dir, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\datasets\satellite.py", line 113, in init self.load_train_split() File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\datasets\satellite.py", line 121, in load_train_split self.all_rays, self.all_rgbs, self.all_ids = self.load_data(self.json_files, verbose=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\datasets\satellite.py", line 193, in load_data rays = get_rays(cols.flatten(), rows.flatten(), rpc, min_alt, max_alt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\satnerf\datasets\satellite.py", line 41, in get_rays lons, lats = rpc.localization(cols, rows, max_alts) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\rpcm\rpc_model.py", line 129, in localization lon, lat = self.localization_iterative(ncol, nrow, nalt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\JohannesBach\anaconda3\envs\satnerf3\Lib\site-packages\rpcm\rpc_model.py", line 165, in localization_iterative raise MaxLocalizationIterationsError("Max localization iterations (100) exceeded") rpcm.rpc_model.MaxLocalizationIterationsError: Max localization iterations (100) exceeded.

My guess is that that I need RPC data for the Sentinel-2 imagery, but I don't know.

Any suggestions?

Kind regards, Johannes Bach

tehret commented 2 months ago

The method was developed for high-resolution imagery and in particular acquisitions with different point of views (compared to the spatial resolution of the image). As such, it cannot work for Sentinel-2. Moreover, Sentinel-2 data provided by the Copernicus program are ortho-photos (i.e resampled to compensate the possible difference in the acquisition point of view).

To answer your specific question, yes RPCs are necessary. They define the camera model (i.e how the satellite is looking at the scene). It is equivalent to the pinhole camera model fit with COLMAP for non-remote sensing based NeRF paper.