Importing large files

[michpaulatto]

Hi, I am trying to import a file with more than 1e8 triangles, which is set as the limit in stl.py. What is the reason for the hard limit and is there a workaround or an alternative? I would be happy for example to import a subregion of the file for inspection.

Thanks Michele

[wolph]

You can easily override the limit, but you would rarely want to. Here's a similar recent issue: https://github.com/WoLpH/numpy-stl/issues/143

A single row is 50 bytes which means that the 1e8 triangles will consume 5GB of memory already. It's not impossible to have a STL file that large, but in most cases it indicates that your file is damaged instead.

One of the largest test files commonly used by 3D libraries, the Thai Statue, is only 1e7 triangles: http://graphics.stanford.edu/data/3Dscanrep/

So having much more is very unlikely

[michpaulatto]

Thanks, I tried increasing max_count but it doesn't help. I get an overflow error. My model is from a CT scan of a rocks sample and comprises ~4.5 million faces so it should not be a problem. I think I may have an issue with data encoding.

[michpaulatto]

I have done some more testing and it seems to be a problem with endianness. The binary stl specification assumes that the data are little endian but my file is big endian. Since there are multiple data types with different byte length in the stl file I don't think I can do a simple byte swap on the whole file. I could try to add a byte swap option stl.py.

[michpaulatto]

Here is my crude byte swapping script. This solved my problem.

import numpy as np
import os

filenamein="file.stl"
filenameout="file_swapped.stl"

f1 = open(filenamein, "rb")
f2 = open(filenameout, "w")

# Read the header
header=f1.read(80)
# Replace if needed
header='My new header '.ljust(80)

# Skip header and read number of triangles
f1.seek(80, os.SEEK_SET)
idtype='>u4' # Big Endian uint32

nt = np.fromfile(f1, dtype=idtype,count=1)[0]

# Read triangles one by one. Use big endian.
fdtype='>f4' # Big Endian float32

normals=np.zeros((nt,3),dtype='<f4')
v1=np.zeros((nt,3),dtype='<f4')
v2=np.zeros((nt,3),dtype='<f4')
v3=np.zeros((nt,3),dtype='<f4')
attr=np.zeros(nt,dtype='<u2')
for i in range(nt):
    normals[i,:] = np.fromfile(f1, dtype=fdtype,count=3)
    v1[i,:] = np.fromfile(f1, dtype=fdtype,count=3)
    v2[i,:] = np.fromfile(f1, dtype=fdtype,count=3)
    v3[i,:] = np.fromfile(f1, dtype=fdtype,count=3)
    attr[i] = np.fromfile(f1, dtype='>u2',count=1)
f1.close()

f2.write(header)
np.asarray(nt,dtype='<u4').tofile(f2)
for i in range(nt):
    normals[i,:].tofile(f2)
    v1[i,:].tofile(f2)
    v2[i,:].tofile(f2)
    v3[i,:].tofile(f2)
    attr[i].tofile(f2)
f2.close()

[wolph]

How I loathe endianness issues... glad to see you've got it working :)

I do wonder what your source system is that it uses different endianness, that's becoming exceedingly rare in my experience.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.