Compilation failure with Visual Studio 2008 SP1 in x64 mode only. Serialization might be corrupt. (FLANN 1.8.4)

[sromberg]

Dear Marius,

I found the following bug in FLANN 1.8.4: When compiling my code with Visual Studio 2008 SP1 in x86 mode compilation succeeds. However, when I compile my code in x64 mode then the following compilation error appears:

"1>f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(17) : error C2228: left of '.serialize' must have class/struct/union"

(The full error output is appended at the end of this message)

It is easy to fix by adding

#ifdef _MSC_VER
BASIC_TYPE_SERIALIZER(unsigned __int64);
#endif

in serialization.h.

However, the critical questions I have in mind are:

• What does it mean for the deserialization of already saved indices?
• I suppose that the difference results from using size_t as type somewhere in the serialization code. Does the written format therefore depend on the architecture (i.e. x86 vs. x64) then?
• Can I still load with the current version 1.8.4 my pre-computed indices that have been written to disk with Flann 1.7.1?

Thanks, Stefan

Full error message:

1>f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(17) : error C2228: left of '.serialize' must have class/struct/union
1>        type is 'unsigned __int64'
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(25) : see reference to function template instantiation 'void flann::serialization::access::serialize<Archive,T>(Archive &,T &)' being compiled
1>        with
1>        [
1>            Archive=flann::serialization::SaveArchive,
1>            T=unsigned __int64
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(39) : see reference to function template instantiation 'void flann::serialization::serialize<OutputArchive,T>(Archive &,T &)' being compiled
1>        with
1>        [
1>            OutputArchive=flann::serialization::SaveArchive,
1>            T=unsigned __int64,
1>            Archive=flann::serialization::SaveArchive
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(263) : see reference to function template instantiation 'void flann::serialization::Serializer<T>::save<Archive>(OutputArchive &,const T &)' being compiled
1>        with
1>        [
1>            T=unsigned __int64,
1>            Archive=flann::serialization::SaveArchive,
1>            OutputArchive=flann::serialization::SaveArchive
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(112) : see reference to function template instantiation 'Archive &flann::serialization::OutputArchive<Archive>::operator &<unsigned __int64>(const T &)' being compiled
1>        with
1>        [
1>            Archive=flann::serialization::SaveArchive,
1>            T=unsigned __int64
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(263) : see reference to function template instantiation 'void flann::serialization::Serializer<T>::save<Archive>(OutputArchive &,const std::vector<_Ty> &)' being compiled
1>        with
1>        [
1>            T=std::vector<flann::KDTreeSingleIndex<flann::DummyDistance>::Interval>,
1>            Archive=flann::serialization::SaveArchive,
1>            OutputArchive=flann::serialization::SaveArchive,
1>            _Ty=flann::KDTreeSingleIndex<flann::DummyDistance>::Interval
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/kdtree_single_index.h(192) : see reference to function template instantiation 'Archive &flann::serialization::OutputArchive<Archive>::operator &<std::vector<_Ty>>(const T &)' being compiled
1>        with
1>        [
1>            Archive=flann::serialization::SaveArchive,
1>            _Ty=flann::KDTreeSingleIndex<flann::DummyDistance>::Interval,
1>            T=std::vector<flann::KDTreeSingleIndex<flann::DummyDistance>::Interval>
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(17) : see reference to function template instantiation 'void flann::KDTreeSingleIndex<Distance>::serialize<Archive>(Archive &)' being compiled
1>        with
1>        [
1>            Distance=flann::DummyDistance,
1>            Archive=flann::serialization::SaveArchive
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(25) : see reference to function template instantiation 'void flann::serialization::access::serialize<Archive,T>(Archive &,T &)' being compiled
1>        with
1>        [
1>            Archive=flann::serialization::SaveArchive,
1>            T=flann::KDTreeSingleIndex<flann::DummyDistance>
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(39) : see reference to function template instantiation 'void flann::serialization::serialize<OutputArchive,T>(Archive &,T &)' being compiled
1>        with
1>        [
1>            OutputArchive=flann::serialization::SaveArchive,
1>            T=flann::KDTreeSingleIndex<flann::DummyDistance>,
1>            Archive=flann::serialization::SaveArchive
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/util/serialization.h(263) : see reference to function template instantiation 'void flann::serialization::Serializer<T>::save<Archive>(OutputArchive &,const T &)' being compiled
1>        with
1>        [
1>            T=flann::KDTreeSingleIndex<flann::DummyDistance>,
1>            Archive=flann::serialization::SaveArchive,
1>            OutputArchive=flann::serialization::SaveArchive
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/kdtree_single_index.h(216) : see reference to function template instantiation 'Archive &flann::serialization::OutputArchive<Archive>::operator &<flann::KDTreeSingleIndex<Distance>>(const T &)' being compiled
1>        with
1>        [
1>            Archive=flann::serialization::SaveArchive,
1>            Distance=flann::DummyDistance,
1>            T=flann::KDTreeSingleIndex<flann::DummyDistance>
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/kdtree_single_index.h(214) : while compiling class template member function 'void flann::KDTreeSingleIndex<Distance>::saveIndex(FILE *)'
1>        with
1>        [
1>            Distance=flann::DummyDistance
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/all_indices.h(89) : see reference to class template instantiation 'flann::KDTreeSingleIndex<Distance>' being compiled
1>        with
1>        [
1>            Distance=flann::DummyDistance
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/all_indices.h(119) : see reference to class template instantiation 'flann::needs_kdtree_distance<T>' being compiled
1>        with
1>        [
1>            T=flann::KDTreeSingleIndex<flann::DummyDistance>
1>        ]
1>        f:\dev\projects\code\clusterlib_v3\include\flann/algorithms/all_indices.h(157) : see reference to class template instantiation 'flann::valid_combination<Index,Distance,ElemType>' being compiled
1>        with
1>        [
1>            Index=flann::KDTreeSingleIndex,
1>            Distance=flann::L2<float>,
1>            ElemType=ElementType
1>        ]

[mariusmuja]

Yes, serialized indexes are not portable to different architectures currently. This is something I'm planing to fix in a future version.

Also, indexes computed with FLANN 1.7.1 would not work in 1.8.4 due to some significant changes in the library that were made when support for incremental insertion/removal wad added.

[sromberg]

Uff, that is bad news. I have tons of different visual vocabularies stored on my disk. I cannot simply move to a new on-disk format. Also this fact should be clearly documented.

Despite that I have written several different serialization routines before and I found that it is actually quite portable as long as you stick to types with predefined sizes such as int32_t``or int64_t. In other words, never use int and long and size_t in the structures being stored. You can simple convert to these types when loading/saving.

[mariusmuja]

Unfortunately the changes from version 1.7 were significant enough that it was too hard to maintain on-disk compatibility. You are correct, this should be better documented, I'll make sure to add this to the documentation.

Using fixed size primitive types such as int32_t and int64_t sounds like the right thing to do. I'll be adding this to a future version.