search

NeRF-or-Nothing / vidtonerf

Web server and full video synthesis pipeline based on NeRF. RCOS
MIT License
23 stars 19 forks source link

Rabbitmq timing out after repeated use #109

Open rougejaw opened 8 months ago

rougejaw commented 8 months ago

During end-to-end testing, nerf_worker is unable to connect to rabbitmq, timing out & causing the process to halt. Another notable issue in the screenshot is multiple progress bars being printed, when only one should appear.

The following log was from a run the only change to the TensoRF config file being "n_iters = 1000". Note that the test completes as expected when "n_iters = 1" image

SimonDaKappa commented 7 months ago

Possible cause is in TensoRF/main.py:nerf_worker(). The pika connection and training occur on the same thread, and the pika conneciton sends a hearbeat every 300s. If training takes >300(s) the pika connection cannot send a im good and rabbitmq terminates the connection.