yahoojapan / NGT

Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data
Apache License 2.0
1.26k stars 115 forks source link

How to reduce grp / objpo / trei / trel files size? #131

Closed netdur closed 1 year ago

netdur commented 1 year ago

Currently each file is ~11 GB in size, my data set is around 100 records, my configuration as following

NGT::Property property;
property.dimension = 512;
property.objectType = NGT::Index::Property::ObjectType::Float;
property.distanceType = NGT::Index::Property::DistanceType::DistanceTypeCosine;
property.indexType = NGT::Index::Property::GraphAndTree;

NGT::Index::create(indexName, property);

thank you

masajiro commented 1 year ago

You can change the sizes of the files like this. The parameters limit the maximum number of stored vectors. You can store up to about 50M vectors for every 512. However, the maximum size also depends on the number of dimensions.

netdur commented 1 year ago

thank you sir