search

alicevision / Meshroom

3D Reconstruction Software
http://alicevision.org
Other
11.05k stars 1.08k forks source link

[Enhancement] Network processing client #357

Open natowi opened 5 years ago

natowi commented 5 years ago

A simple Meshroom client for network processing to distribute the workload on multiple machines would be useful.

The idea is to utilize the computers you have at home/in the office/school/university/... This could be done by directly connecting running Meshroom instances and set one as master and all the others as computing clients or by installing MR computing clients and connect them to MR.

The installation and setup should be as simple as possible.

(Not everyone has access to a renderfarm/workstation)

fabiencastan commented 5 years ago

That was the idea of PR #81, without re-inventing the wheel. Fireworks allows to launch a master and slaves on other machines easily. But I never managed to finalize it...

fabiencastan commented 5 years ago

If you can test fireworks, I would love to have other feedback to decide if it's the right way to go! From my test, it was easy to install and setup (I have only tested on linux for now). The UI for monitoring is limited, but as we have progression feedback within meshroom, that's not a too big issue.

natowi commented 5 years ago

@fabiencastan Thank you. I installed fireworks and added the fireworks submitter to Meshroom. However Meshroom does not find the fireworks installation. Do I have to add/edit a settings file somewhere to point Meshroom to fireworks? I'll give it another try in a few weeks when I have more time.

ShalokShalom commented 5 years ago

Interested in this

natowi commented 5 years ago

Where do I have to install fireworks to allow this submitter to find the fireworks module and the mongodb yaml config file? I get the error WARNING:root:== The following plugins could not be loaded ==

I think I need to set an environment variable...?

-- fireworks 1.8.7 mongo 3.7.2 python 3.6.2

thegallivanter commented 5 years ago

Natowi did you ever get distributed processing with Fireworks set up and working? Would be a great speed up for me as I have several comparable workstations and a single workstation takes weeks processing my projects. Could cut down my processing time to a single week.

natowi commented 5 years ago

@thegallivanter No, I did not figure out how to link Meshroom and fireworks (yet), but I will continue looking into this, as is on the to-do list for the documentation. This should not be too difficult, but without an (working) example it is time consuming to figure out.

natowi commented 5 years ago

Reference https://github.com/alicevision/meshroom/pull/213

natowi commented 4 years ago

@sergiy-nazarenko I read you successfully use meshroom on a render farm. Can you describe your render farm setup? The documentation (wiki) on this feature is quite thin and could use some input.

sergiy-nazarenko commented 4 years ago

Hej, natowi Sure, I can describe. I actually modified the simpleFarmSubmitter.py , here is the link of our slurmFarmSubmitter : https://raw.githubusercontent.com/sergiy-nazarenko/hafarm/0.3.3/scripts/meshroom/submitters/slurmFarmSubmitter.py We use slurm on render farm. Slurm needs bash script to load jobs with specific comments such #SBATCH (https://support.ceci-hpc.be/doc/_contents/QuickStart/SubmittingJobs/SlurmTutorial.html) . slurmFarmSubmitter converts the meshroom nodes to those scripts through python + jinja template https://github.com/sergiy-nazarenko/hafarm/blob/master/slurm_job.schema

Cheers, Sergiy

spreck commented 3 years ago

@natowi @sergiy-nazarenko Is there any more info on how to implement this?

I looked through the linked documents, but frankly they are beyond my level of expertise. If this could be made more easily accessible that would be great. Any tips on how to go about using multiple computers for meshroom reconstructions?

natowi commented 3 years ago

@spreck https://github.com/alicevision/meshroom/wiki/Renderfarm-submitters