Compatibility with OpenMVG

danylaksono commented 9 years ago

Hi, Just a quick question: with the new output format in OpenMVG (i.e. using a single SfM_data.json rather than a SfM_Output directory), how can we convert SfM format from OpenMVG to MVE, while makescene expects SfM_output dir?

Thanks

pmoulon commented 9 years ago

Hi, OpenMVG v0.8 introduce a new generic SfM data container. MVE was supporting openMVG v0.7 format that was made from a series of files exported in a SfM_Output directory. We should see with @simonfuhrmann how maintain an importer/exporter:

we could write a sfm_data.json converter to the old SfM_Ouput format in OpenMVG.
we could rewrite the makescene to support the new sfm_data format of OpenMVG in MVE.

simonfuhrmann commented 9 years ago

If the currently supported format is deprecated, we should rewrite the makescene importer. Can you point me to documentation for the new format?

danylaksono commented 9 years ago

This is the documentation that I could find so far: SfM_Data format. I think the best way is to refer at the SfM_data.json itself, which somewhat looks like this:

         //project path and views  
{
    "sfm_data_version": "0.1",
    "root_path": "/home/dany/workdir/cloudsfm/uploaded/dany/theprambanan/images",
    "views": [
        {
            "key": 0,
            "value": {
                "ptr_wrapper": {
                    "id": 2147483649,
                    "data": {
                        "local_path": "/",
                        "filename": "3f8142c7d5e37cf4c8c000755a3902af.jpg",
                        "width": 1920,
                        "height": 1080,
                        "id_view": 0,
                        "id_intrinsic": 0,
                        "id_pose": 0
                    }
                }
            }
        },
        {
            "key": 1,
            "value": {
                "ptr_wrapper": {
                    "id": 2147483650,
                    "data": {
                        "local_path": "/",
                        "filename": "419190128baeb9b80ecbdf84d627167e.jpg",
                        "width": 1920,
                        "height": 1080,
                        "id_view": 1,
                        "id_intrinsic": 0,
                        "id_pose": 1
                    }
                }
            }
        },
       {
            "key": 2, 
        ... // instrinsic parameters

],
    "intrinsics": [
        {
            "key": 0,
            "value": {
                "polymorphic_id": 2147483649,
                "polymorphic_name": "pinhole_radial_k3",
                "ptr_wrapper": {
                    "id": 2147483666,
                    "data": {
                        "width": 1920,
                        "height": 1080,
                        "focal_length": 2356.3642758961146,
                        "principal_point": [
                            1057.9389463937234,
                            551.79986482906179
                        ],
                        "disto_k3": [
                            0.24891878391848027,
                            -0.13397421225623776,
                            1.0019358427506504
                        ]
                    }
                }
            }
        }
    ],

      // extrinsic parameters   

    "extrinsics": [
        {
            "key": 0,
            "value": {
                "rotation": [
                    [
                        0.99959283988299119,
                        -0.0032754160171489031,
                        0.028344772085366655
                    ],
                    [
                        0.0033428094288908456,
                        0.99999169699379487,
                        -0.0023305725889873541
                    ],
                    [
                        -0.028336903143761048,
                        0.002424374844166053,
                        0.99959548934898468
                    ]
                ],
                "center": [
                    0.051451443934849024,
                    0.0032790146204521127,
                    -0.011287905013075042
                ]
            }
        },
        {
            "key": 1,
            "value": {
                "rotation": [
                    [
                        0.99618260859259722,
                        -0.0071818319542005481,
                        -0.086997882890503642
                    ],
                    [
                        0.0075766213257551143,
                        0.99996244069241669,
                        0.0042085643337399659
                    ],
                    [
                        0.086964390108447698,
                        -0.004851648611218588,
                        0.99619960668473406
                    ]
                ],
                "center": [
                    -0.78216890236071102,
                    -0.029742625235454394,
                    -0.042334232557200327
                ]
            }
        },

      ...  // next is recovered structure

 "structure": [
        {
            "key": 0,
            "value": {
                "X": [
                    -0.42598182641319271,
                    0.10051549152421303,
                    1.1878502049756672
                ],
                "observations": [
                    {
                        "key": 0,
                        "value": {
                            "id_feat": 2511,
                            "x": [
                                167.83200073242188,
                                739.20697021484375
                            ]
                        }
                    },
    ....

of the two options suggested by @pmoulon, I think the second one would be a more sustainable(for future releases) and efficient solution. However, any solutions is good, as long as OpenMVG and MVE goes well together.

Thanks for your help

pmoulon commented 9 years ago

Hi, To @simonfuhrmann

option 1: limit the change of code in MVE and force to maintain an exporter in OpenMVG option 2: limit the code change in OpenMVG and force to maintain an importer in MVE

We can make a MVE sfm_data.json importer, but it will require to add a JSON library (rapidjson) as dependencies to MVE. Does it sounds as something feasible?

Note that the sfm_data container can contain abstract view & camera type. But we can limit the supported format to a given list in order to ease the process.

The importer must follow this rules:

json parsing of the Views
json parsing of the intrinsics
json parsing of the extrinsics
json parsing of the structure (if required)

convert those data to valid MVE data.

simonfuhrmann commented 9 years ago

No, adding a JSON library is not an option. What is an "abstract view & camera type"? I will see what I can do to implement the new format. But I suppose this is not happening before the end of June.

pmoulon commented 9 years ago

I mean that the sfm_data container can support various view and camera type. For example the camera can be (by now):

pinhole,
pinhole with a 1 coeff radial distortion,
pinhole with a 3 coeff radial distortion.

No problem regarding the deadline.

@danylaksono , if you need a fast solution you could write a new openMVG2MVE application that will export a sfm_data file to the old SfM_Output data format.

Meekohi commented 9 years ago

@simonfuhrmann, why is there resistance to adding a JSON library to MVE? It'd be great if everyone (MVE, Theia, openMVG, etc) could agree on a standard JSON format and stick with it when possible instead of re-inventing a new custom-parsed format every time.

simonfuhrmann commented 9 years ago

Because there is a resistance to adding any library. It's my personal preference and doesn't need to be discussed here. But in principle I agree with you that standards are a good thing.

danylaksono commented 9 years ago

I don't have any time constraint either. I'll get my hands on a Python implementation as suggested by @pmoulon, but don't expect it to be completed in anytime soon. Any suggestion or ideas are greatly appreciated.

simonfuhrmann commented 9 years ago

I have given this some thought. I think an importer on MVE's side of things would be much harder to realize then an exporter from OpenMVG. In a couple of weeks, we are introducing a new view format, where a view is not a single .mve file anymore, but a directory. It contains an INI-like file for camera parameters and stuff, and regular image files for the embeddings. Something like this can trivially be exported from OpenMVG I guess.

@pmoulon: What about I go ahead and document the new file system format for MVE in the wiki, and we can both try to write an exporter in OpenMVG once I am finished?

pmoulon commented 9 years ago

@simonfuhrmann Let's do that. As exporter for PMVS, CMPMVS already exists, it will be a pleasure to add a new one for MVE ;-)

I don't know which camera model you will support. But we can do like the last time use undistorted images and model the camera as a classic Pinhole camera model at export time.

simonfuhrmann commented 9 years ago

Hi Pierre,

it took a while, but the new MVE file format is documented here: https://github.com/simonfuhrmann/mve/wiki/The-MVE-File-Format

Note that, right now, UMVE does not work in MVE2 and testing the exporter output might be tricky. It's probably a good idea to try reconstructing with the dmrecon app. If that works, chances are high that everything is correct.

pmoulon commented 9 years ago

Thanks @simonfuhrmann

simonfuhrmann commented 9 years ago

Just an update on this. The "mve2" branch is now fully functional, and the OpenMVG exporter to the new format can be tested by loading the scene up in UMVE. The branch requires more testing, though.

Again, the new format is documented here. https://github.com/simonfuhrmann/mve/wiki/The-MVE-File-Format

KevinCain commented 9 years ago

I've switched to Simon's mve2 branch and would like to convert my OpenMVG 'reconstruction_global' test scene to MVE with './makescene /path_to_OpenMVG_v0.8_scene /path_to_MVE2_scene'.

I believe there is no converter yet from OpenMVG v0.8 to the format MVE2 expects. Or am I overlooking something?

pmoulon commented 9 years ago

We plan to work on it soon. If you want to help your welcome ;-) @rhiestand have also made a OpenMVG0.8 to OpenMVG0.7 data converter that MVE that should be able to load... but it is not released yet, since the MVE2 format export will be the best solution for the future.

simonfuhrmann commented 9 years ago

We decided to put an exporter to MVE2 into OpenMVG code base. I believe, however, Pierre hasn't started the implementation yet.

KevinCain commented 9 years ago

I'm happy to write a OpenMVG0.8 to MVE2 scene converter. Last week I saw Simon's clear documentation for the MVE2 format, cited earlier in this thread. However, I'm quite new to OpenMVG. Is there a compact description of the 0.8 scene format that I could use as a basis? Thanks again.

pmoulon commented 9 years ago

Thanks @KevinCain

So basically the SfM Data are stored in one unique structure (the SfM_Data container). It is compound of the following elements:

a map of Views (mapping between the image and the link to intrinsic and pose defined with IDs)
a map of Intrinsics data
a map of Pose data
a map of Landmarks (the 3D points with view visibility and 2d feature ids and positions)

The container is defined here https://github.com/openMVG/openMVG/blob/master/src/openMVG/sfm/sfm_data.hpp#L33

When you want to create a new exporter you have to parse the SfM_Data container and convert the data to the desired output format. => Here it's done for

So you can start from on of those and modify it, please consider to start from the develop branch in your fork. Let's consider main_openMVG2MVE2.cpp as a good starting name ;-)

Don't hesitate to ask more question or ask collaboration into your fork if you need help to code and test the converter on various scenes later on.

KevinCain commented 9 years ago

Thanks for the clear introduction -- I'll have a look and check back in once I have a chance to parse this. It also occurs to me that it could be valuable to have a standalone openMVG2MVE2 binary as both scene containers are clay on the wheel.

pmoulon commented 9 years ago

Hi @KevinCain do you have start something about the exporter in which I could help?

KevinCain commented 9 years ago

Hello, Pierre,

I was temporarily blocked last week in compiling the most recent 'devel' branch commits for MVE2. I have compiled MVE2 on another machine, so I figured that recent commits broke the build. Of course, it is just as likely that the problem is entirely mine.

I will return to this in the coming days and present my status, in case you have time to advise or jump in.

Thanks again,

-Kevin

On Sun, Jul 19, 2015 at 12:53 PM, Pierre Moulon notifications@github.com wrote:

Hi @KevinCain https://github.com/KevinCain do you have start something about the exporter in which I could help?

— Reply to this email directly or view it on GitHub https://github.com/simonfuhrmann/mve/issues/148#issuecomment-122698038.

simonfuhrmann commented 9 years ago

If there are issues with compiling MVE2, just post the error.

KevinCain commented 9 years ago

Hello, Simon,

I'll rebuild and run the MVE2 branch and log an entry on github if I face the same trouble.

Separately, is MVE2 support anticipated for MVS texturing? (I don't see an MVE2 branch on https://github.com/nmoehrle/mvs-texturing). Thanks,

-Kevin

On Mon, Jul 20, 2015 at 12:36 AM, Simon Fuhrmann notifications@github.com wrote:

If there are issues with compiling MVE2, just post the error.

— Reply to this email directly or view it on GitHub https://github.com/simonfuhrmann/mve/issues/148#issuecomment-122786701.

nmoehrle commented 9 years ago

I will implement the support for the new MVE file format within the next week, I have opened an issue for it (https://github.com/nmoehrle/mvs-texturing/issues/17).

Cheers, Nils

KevinCain commented 9 years ago

Many thanks,

-Kevin

On Monday, July 20, 2015, Nils Moehrle notifications@github.com wrote:

I will implement the support for the new MVE file format within the next week, I have opened an issue for it (nmoehrle/mvs-texturing#17 https://github.com/nmoehrle/mvs-texturing/issues/17).

Cheers, Nils

— Reply to this email directly or view it on GitHub https://github.com/simonfuhrmann/mve/issues/148#issuecomment-122808096.

simonfuhrmann commented 9 years ago

Funded research opportunities should not be discussed here. If you are interested in collaboration, please send a mail to the MVE staff (mail address in the below link) with a few more details what you have in mind. http://www.gris.informatik.tu-darmstadt.de/projects/multiview-environment/

simonfuhrmann commented 9 years ago

Support for importing the old OpenMVG format has been removed in MVE2. Since the new exporter should be ready soon, this keeps the code clean of deprecated features.

simonfuhrmann commented 9 years ago

Instead of having an OpenMVG importer in MVE, an MVE exporter has been implemented to OpenMVG. I am considering this issues as resolved.

simonfuhrmann / mve

Compatibility with OpenMVG #148