search

nerfstudio-project / nerfstudio

A collaboration friendly studio for NeRFs
https://docs.nerf.studio
Apache License 2.0
9.09k stars 1.21k forks source link

Reality Capture using .xmp files #2383

Open s1ddok opened 11 months ago

s1ddok commented 11 months ago

Hey!

I've got some awesome data I would love to train some nerfs on, but the data is in .xmp file format and looks like this:

<x:xmpmeta xmlns:x="adobe:ns:meta/">
  <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
    <rdf:Description xcr:Version="3" xcr:PosePrior="locked" xcr:Coordinates="absolute"
       xcr:DistortionModel="brown3t2" xcr:FocalLength35mm="65.478964140309"
       xcr:Skew="0" xcr:AspectRatio="1" xcr:PrincipalPointU="-0.012871369016343"
       xcr:PrincipalPointV="0.0017358637506797" xcr:CalibrationPrior="exact"
       xcr:CalibrationGroup="-1" xcr:DistortionGroup="-1" xcr:latitude="179.998188655585693N"
       xcr:longitude="37.983578116223768E" xcr:version="2.2.0.0" xcr:altitude="643099440/10000"
       xmlns:xcr="http://www.capturingreality.com/ns/xcr/1.1#">
      <xcr:Rotation>-0.628072249977553 0.778153220693978 -0.00167747778664398 0.0382608693246996 0.0287283186031107 -0.998854738982983 -0.777213841028742 -0.627417125072173 -0.0478162783946945</xcr:Rotation>
      <xcr:Position>3.92101192706972 3.06162084997137 1.34968371224218</xcr:Position>
      <xcr:DistortionCoeficients>-0.062191034720569 2.3214570830388 -26.2121457576208 0 0.000332769342338838 0.000451019835287317</xcr:DistortionCoeficients>
    </rdf:Description>
  </rdf:RDF>
</x:xmpmeta>

I went through the code of reality capture converter and unfortunately it only supports CSV files. At first I thought that it is going to have the same data just reorganized, but from I saw the expected data in .csv file is different. I would also love to undistort images using that brown3t2 model that I don't really familiar with. Do you think it is possible for me to use .xmp files to train data in nerfstudio?

tancik commented 11 months ago

If you can find a way to convert it to the format described here you should be all set. Note that we only support openCV camera models.

rumblecoder commented 11 months ago

@s1ddok You can use https://github.com/Fraunhofer-IIS/camorph to convert from XMP files to NeRF JSON files. I would also recommend to export your data from RealityCapture undistorted as FBX and then convert to JSON with Camorph.