search

lfranke / TRIPS

https://lfranke.github.io/trips/
MIT License
495 stars 28 forks source link

viewer.exe not working #22

Closed lh-dm closed 4 months ago

lh-dm commented 4 months ago

I can't get viewer to work. I have downloaded "tt_scenes.zip" and unzipped it in the scene folder. I downloaded "tt_experiments.zip" and unzipped it in the experiments folder.

This is what it currently "experiments" looks like for me: grafik

But it should look so (???):

    experiments/
        checkpoint_train
        checkpoint_playground
        ...

I am already confused - here are no checkpoints.

I tried it anyway:

D:\TRIPS>D:\TRIPS\build\bin\RelWithDebInfo\viewer.exe --scene_dir D:\Trips\scenes\tt_train
register neural render info
register TnnInfo
Git ref: 2c2a5e1822791a0f53296c8e5c9fa2058a03c4a9

And then it stops... no error... but i found in %AppData% huge crash dump files - each over 1GB:

grafik

Then I started to train... That didn't work first and stopped at Epoch 1.

complete log: log.txt

Then I changed the crop-size and this solved: --TrainParams.train_crop_size 256

I managed to train up to epoch 10. Epoch 10 was my goal - it's OK for the 1st run.

This is my generated data:

grafik

grafik

The question is now... how can I view it? Nothing seems to work...

My GFX-Card has 8 GB and is during training fully loaded (>7 GB).

iFimo commented 4 months ago

Answer to: https://github.com/lfranke/TRIPS/issues/23#issuecomment-1944274015

Hi, @dm-de, I will try to answer your questions here to keep these two different topics separate.

Can you also give me some hints?

  1. Can you confirm, that "tt_experiments.zip" works? Here are no checkpoint's in zip. See here my post: viewer.exe not working #22

I think the folders themselves that containing the training epochs are the checkpoints. Yes, they are not called checkpoints. That surprised me too, but I think the description in the readme file is just a structure tree. Maybe the same as mine “dense_point_cloud.ply”. ?

  1. What is your GPU memory usage with viewer.exe & playground scene I have a issue with viewer and I'm not sure, if this happen, because I have only 8GB VRAM or because my exe ist compiled wrong. I think this has something to do with saiga & cuda compute capability. My special problem is, that I compile on one machine and copy release to another machine (where I have no admin rights). For this I need to set cuda version manually... It is hard to find out what is wrong.

From what I currently know, you don't need the experiments folder at all to use the viewer. I'll test this again tomorrow (up until now I had only moved the checkpoints/experiments to a subfolder for testing). I'll throw out the experiments folder completely tomorrow, then I'll try to start the viewer with just the scenes folder present and I'll pay attention to the VRAM. I'll report back tomorrow.

I have a 3090 with 24GB VRAM.

I also suspect that something has been compiled badly for you or is a question of admin rights.

Have you tried to recompile it?

lh-dm commented 4 months ago

I am disappointed. I have compiled again. For some reason cmake has set different cuda versions. I have set everything to cuda 7.5 only... But it makes no difference.

My 8GB VRAM is probably not enough. :-( viewer crashes without error message. It would be great if there was a very slimmed-down viewer.

train on the other hand works (but only with train_crop_size param) VRAM is hard at the limit 7.8 of 8GB used!

iFimo commented 4 months ago

Hey, here are my test results.

VRAM usage: If you have limited VRAM, don't try the viewer with the "train" or "playground", instead use the "horse" for testing.

Here is my VRAM usage in various scenes (1.1 GB, is already occupied on my system without the viewer, so that's where it comes from):

Train           =1,1 GB + 5,2 GB = 6,3 GB
Playground      =1,1 GB + 5,1 GB = 6,2 GB
Horse           =1,1 GB + 3,5 GB = 4,6 GB

Each scene should work with 8GB of VRAM, but if you have problems try the smaller scenes rather than the largest ones.

Result: If the experiments folder has been removed:

To open a scene, the scenes folder AND the experiments folder must exist. With removed experiments folder it dosnt work. I also tried putting the experiments themselves in a subfolder and also renaming them to checkpoint_playground. Result: Both use cases seem to work fine, as long as the experiments are somewhere in the experiments folder it should work.

But I have it like in your first screenshot and that works fine.

I have compiled again.

You need:

CUDA 11.8 (make sure to at least include Nsight NVTX, Development/ , Runtime/Libraries/ and the Visual Studio Integration)

These are the things I paid attention to when compiling for the twentieth time, maybe it will help.

Otherwise I don't know what to do next.

lfranke commented 4 months ago

Hi,

yes, the folders in the tt_experiments.zip and tt_scenes.zip are named the same. Sorry for the oversight, I will update this in the supplemental soon. In the meantime, you can rename the folders in the experiments however you want.

Thanks @iFimo, yes the viewer fails if the experiments folder is completely missing, otherwise if the folder exists but no experiments are present just the Model View of the point cloud will start and you will be unable to switch to Neural View.

I just retried on Windows with a RTX2070 and my VRAM measurements were similar to @iFimo.

Maybe try to start the viewer with --render_scale 0.5, this forces the resolution to be lower and should require less VRAM.

dm-de commented 4 months ago

I (dm-de = lh-dm) got viewer working without crash. I copied TRIPS root directory with all files to other computer. No other tool installation was requred (conda, cuda etc.). I don't know what was missing or wrong before.

lfranke commented 4 months ago

Hi, that is indeed odd! But I'm happy that it is working for you :)