Got Error When Generating 90K Sample Data with generate_train_data.py

[nixczhou]

Hi I got this error when generating more 90K Sample training data, but it just work with 10K, any solutions? Please help me :(

dict_keys(['__header__', '__version__', '__globals__', 'points', 'line_segment_index', 'grid_points'])
Traceback (most recent call last):
  File "generate_train_data.py", line 27, in <module>
    sio.savemat(save_file, {'pivot_images':pivot_images,
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio.py", line 296, in savemat
    MW.put_variables(mdict)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 893, in put_variables
    self._matrix_writer.write_top(var, asbytes(name), is_global)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 634, in write_top
    self.write(arr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 673, in write
    self.write_numeric(narr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 699, in write_numeric
    self.write_element(arr)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 540, in write_element
    self.write_regular_element(arr, mdtype, byte_count)
  File "C:\Users\nico\anaconda3\lib\site-packages\scipy\io\matlab\mio5.py", line 554, in write_regular_element
    tag['byte_count'] = byte_count
OverflowError: Python int too large to convert to C long

[lood339]

Can you provide details of your running? For example, the data and how your command line.

[nixczhou]

Used data:

1. worldcup_sampled_cameras.mat
2. worldcup2014.mat

I only change the n=10000 to n=90000 and then run it using _python generate_traindataa.py It works for n=10000, but breaks when n=90000

And the I got error message like the one I posted earlier

OverflowError: Python int too large to convert to C long

Any Suggestions?

[lood339]

Thanks for the details. It turns out that the size of saved .mat file can not be larger than 2G. https://stackoverflow.com/questions/33565414/why-does-saving-mat-files-with-scipy-result-in-larger-file-size-than-with-matlab

When n = 10000, the file size is about 1.15GB. As a result, the file size will be about 10GB when n = 90000. One way to walk around that: saving training data to separated files, for example, 'train_data_90k_1.mat, train_data_90k_2.mat, ...' Each file has 10000 training examples.

[nixczhou]

Thanks for the reply, I found out that numpy .npz format can handle the file size larger than 2G :)

Here is how I modify the code to save .npz file