search

ot / partitioned_elias_fano

Code used for the experiments in the paper "Partitioned Elias-Fano Indexes"
Other
37 stars 11 forks source link

Can't write the compressed collection into disk file #1

Closed Sparklexs closed 8 years ago

Sparklexs commented 8 years ago

After creating the compressed collection, the final work is to write down it into output file using "succinct::mapper::freeze(coll, output_filename);". However, the program just halts there and does not respond any more. Function 'freeze' seems do not work with 'coll' of huge size, after looking into the code I believe it is the 'm_data' of 'mappable_vector' which is too large to fit in 'ofstream::write' in only one invoke. And I split 'm_data' into smaller size and write them recursively, then the whole 'coll' are writen into output file successfully. There must be an limitation of bytes to be written for 'ofstream::write', and compacted bitvectors from corpora like GOV2 and WebClue09 definitely exceeds the limitaion.

Sparklexs commented 8 years ago

It turns out to be bugs inside the platform(cygwin).