Open meeotch opened 2 months ago
Hi @meeotch, thanks for using our code!
Your questions are excellent and indeed these two extra requirements defeat the purpose of a non-development install.
The purpose of a non-devel install is for users to run evc-gui
, which will open up a rendering GUI and support all methods that are only dependent on pytorch.
However, our provided example that uses instant-ngp and gaussian splatting as baselines/backbones will require cuda compilation as you've suggested.
In the meantime, the provided ENeRFi examples shouldn't require extra dependencies, could you try running that and check whether there are any issues?
We will update the examples to include more pytorch-only baselines (ENeRFi etc.) and make it more clear on the extra dependencies of instant-ngp and gaussians.
Are full development dependencies required to run the examples?
At least the answer to this question is no. Aside from the minimal requirements, other dependencies are model-specific. For example, running instant-ngp based methods requires tinycudann, and running gaussian splatting based methods requires compiling their (or our modified) cuda rasterizer. And running NeRF+t or ENeRF shouldn't require extra dependencies.
What shell are people successfully using under Windows? The stock cmd.exe is a nightmare - I much prefer a linux-like shell.
The provided examples are all tested on powershell on Windows. I couldn't agree more that linux shells are much easier to use but I'm unsure whether you can start native a window in MINGW64 shells like git bash. Could you try with the enerfi example and check whether there's any warning on the screen about cuda-gl interop?
On another note, we've also supported WSL on windows. It's performance might be slower than running natively, but the shell experience sure gets much better : ]
Thanks for the info. (I was wondering about performance under WSL, in fact. So knowing that it's slower saves me from having to test it.)
I got both the zju3dv and actor1_4_subseq examples working. (Though I did run out of VRAM during the 3DGS training step. I'm running on a 1080Ti that only has 11GB.) For the benefit of others, here are the dependencies I had to install, beyond the non-development ones:
pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
pip install git+https://github.com/facebookresearch/pytorch3d
pip install lpips
pip install open3d
pip install plyfile
pip install pymcubes
pip install git+https://gitlab.inria.fr/bkerbl/simple-knn
pip install git+https://github.com/dendenxu/diff-gaussian-rasterization
pip install pytorch_msssim
Also, I had to add import mcubes
to the voxel_reconstruction() function in fusion_utils.py, as it was failing to find it at line 38.
Some additional random questions:
Thanks for the help!
Thank you for your insightful observation. I'll update the documentation & example to reflect these hidden requirements.
As for the other questions:
runner_cfg.save_latest_ep
and runner_cfg.save_ep
, which indicates how many epochs to save the latest model and a permanent copy of the model to the disk respectively. To resume training, just rerun the previous training command and it will automatically pick up those checkpoints.PlenOctree
and instant-ngp's. The main GUI logic is written in volumetric_video_viewer.py
where there's a VolumetricVideoViewer
class. All methods starting with draw_
is the actual GUI part. And the keyboard shortcuts are defined in the glfw_key_callback
function. I've always wanted to include a thorough doc about this (or better yet a screen overlay) but haven't found time yet.As for the VRAM issue, you could first try disabling evaluation with lpips
by passing runner_cfg.evaluator_cfg.compute_metrics='PSNR,SSIM'
to the training command.
If the problem persists (occurs during training), we can always lower the rendering resolution size with dataloader_cfg.dataset_cfg.ratio=xxx
.
A more advanced solution would be to tune the initial number of points by controlling the volume_fusion
part.
Thanks for the continued support. Here are a few other tweaks I've discovered, to get the examples working under Windows Git Bash:
os.makedirs(os.path.dirname(db), exist_ok=True)
Pretty sure this is specific to git bash, which uses linux-like forward slashes, but I didn't test it extensively.If I get free time at some point, I'll clean up my notes an post a comprehensive guide for Windows users here.
Some more noob questions... Sorry for packing so many topics in one issue, but it's probably cleaner than starting a new issue for each one:
Minor update: I'm trying to run the zju3dv configuration on a single frame extracted from the actor1_4 data. (So, static frame, 18 cameras.) The colmap step seemed to produce reasonable results. But the l3mhet step seems to run 5 epochs, then goes nuts - been waiting ~2 hours for the second 5 epochs, and it's still running...
I admit that I don't fully understand the difference between the zju3dv exp config and the actor1_4 exp config. Is there an exp config yaml for actor1_4 that is known to work on a single (static) frame of input from that dataset?
Sorry for the n00b questions... I'm trying to play around with EVC on Windows 10, using miniconda and git bash. I've successfully installed CUDA-enabled pytorch manually, and the non-development dependencies with
pip install -e .
evc-gui seems to run.However, when I get to the Examples section, and run
evc -c configs/exps/l3mhet/l3mhet_actor1_4_subseq.yaml
, it fails after loading the images, because tinycudann isn't installed:As far as I can tell, tinycudann is part of the development dependencies, requiring CUDA Toolkit, which requires Visual Studio, etc. (I have read install.md)
Similarly,
python scripts/fusion/volume_fusion.py -- -c configs/exps/l3mhet/l3mhet_actor1_4_subseq.yaml val_dataloader_cfg.dataset_cfg.ratio=0.15
fails to find open3d, which is another development dependency.Is this correct? Are full development dependencies required to run the examples? (And if so, what is the purpose of the non-development install?)
Unrelated question: both of the above also fail at the end with:
NoConsoleScreenBufferError: Found xterm, while expecting a Windows console. Maybe try to run this program using "winpty" or run it in cmd.exe instead. Or otherwise, in case of Cygwin, use the Python executable that is compiled for Cygwin.
I understand that this is a limitation of MINGW64 shells, like git bash. (Though evc-gui does run from git bash.) What shell are people successfully using under Windows? The stock cmd.exe is a nightmare - I much prefer a linux-like shell.