ddiakopoulos / tinyply

:earth_africa: C++11 ply 3d mesh format importer & exporter
598 stars 118 forks source link

Reading Error: All zeros when reading huge ply file (>2GB on Windows, >2.48GB on Ubuntu) #67

Open zyqhnu opened 9 months ago

zyqhnu commented 9 months ago

All the points are (0, 0, 0) when I read a huge .ply file, both on Windows and Linux. I seem to have figured out why: I pre-load the entire file upfront (preload_into_memory = true), just like in example.cpp and example-utils.hpp. (It seems to be prepared for files smaller than 1GB. It's already stated in the notes.) In my experiment, this occurs when the .ply file > 2.01GB (2160000275 bytes) on Windows, and > 2.48GB (2685605907 bytes) on Ubuntu.

But the memoryStream capacity is automatically expanded according to the reference

[@ SaiKishor-MSFT] The maximum size of a MemoryStream object that can be handled by the System.IO.MemoryStream class is determined by the amount of available memory on the system. The maximum size of a MemoryStream object is 2 gigabytes (GB) by default.

My questions are:

  1. Do others get an all-zero error when reading a large file? (Does the capacity of MemoryStream automatically expand?)

  2. If all-zero errors are common, do you need to add a judgment: If the file size is too large, preload is not allowed? like this (in example.cpp, line 64):

    // For most files < 1gb, pre-loading the entire file upfront and wrapping it into a 
    // stream is a net win for parsing speed, about 40% faster. 
    std::ifstream file_helper(filepath, std::ios::binary);
    file_helper.seekg(0, std::ios::end);
    std::streamsize byte_size = file_helper.tellg(); // get the byte_size of the file to be read
    file_helper.close();
    if (preload_into_memory && byte_size < 1.9e9)
    {
    byte_buffer = read_file_binary(filepath);
    file_stream.reset(new memory_stream((char*)byte_buffer.data(), byte_buffer.size()));
    }
    else
    {
    file_stream.reset(new std::ifstream(filepath, std::ios::binary));
    }
zyqhnu commented 9 months ago

oh, BTW, tinyply is cool. I love it