Closed nixczhou closed 3 years ago
Can you provide details of your running? For example, the data and how your command line.
Used data:
I only change the n=10000 to n=90000 and then run it using _python generate_traindataa.py It works for n=10000, but breaks when n=90000
And the I got error message like the one I posted earlier
OverflowError: Python int too large to convert to C long
Any Suggestions?
Thanks for the details. It turns out that the size of saved .mat file can not be larger than 2G. https://stackoverflow.com/questions/33565414/why-does-saving-mat-files-with-scipy-result-in-larger-file-size-than-with-matlab
When n = 10000, the file size is about 1.15GB. As a result, the file size will be about 10GB when n = 90000. One way to walk around that: saving training data to separated files, for example, 'train_data_90k_1.mat, train_data_90k_2.mat, ...' Each file has 10000 training examples.
Thanks for the reply, I found out that numpy .npz format can handle the file size larger than 2G :)
Here is how I modify the code to save .npz file
Hi I got this error when generating more 90K Sample training data, but it just work with 10K, any solutions? Please help me :(